DeBERTaV3

Keras

Model Details

DeBERTaV3 encoder networks are a set of transformer encoder models published by Microsoft. DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder.

Weights are released under the MIT License. Keras model code is released under the Apache 2 License.

Links


  1. DeBERTaV3 Quickstart Notebook
  2. DeBERTaV3 API Documentation
  3. DeBERTaV3 Model Paper
  4. KerasNLP Beginner Guide
  5. KerasNLP Model Publishing Guide

Installation


Keras and KerasNLP can be installed with:

pip install -U -q keras-nlp
pip install -U -q keras>=3

Jax, TensorFlow, and Torch come preinstalled in Notebooks. For instruction on installing them in another environment see the Keras Getting Started page.


Prompts

DeBERTa's main use as a classifier takes in raw text that is labelled by the class it belongs to. In practice this can look like this:

features = ["The quick brown fox jumped.", "I forgot my homework."]
labels = [0, 3]


Example Use

import keras
import keras_nlp
import numpy as np


Raw string data.

features = ["The quick brown fox jumped.", "I forgot my homework."]
labels = [0, 3]

# Pretrained classifier.
classifier = keras_nlp.models.DebertaV3Classifier.from_preset(
"deberta_v3_base_en",
num_classes=4,
)
classifier.fit(x=features, y=labels, batch_size=2)
classifier.predict(x=features, batch_size=2)

# Re-compile (e.g., with a new learning rate).
classifier.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=keras.optimizers.Adam(5e-5),
jit_compile=True,
)
# Access backbone programmatically (e.g., to change `trainable`).
classifier.backbone.trainable = False
# Fit again.
classifier.fit(x=features, y=labels, batch_size=2)


Preprocessed integer data.

features = {
"token_ids": np.ones(shape=(2, 12), dtype="int32"),
"padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]] * 2),
}
labels = [0, 3]

# Pretrained classifier without preprocessing.
classifier = keras_nlp.models.DebertaV3Classifier.from_preset(
"deberta_v3_base_en",
num_classes=4,
preprocessor=None,
)
classifier.fit(x=features, y=labels, batch_size=2)


Model Comments

1 comments
avatar
Jinendra
Jul 25, 2024 03:02 pm

Nice EDA! Great work right there! Any idea how to get a better score with XGB, CBC or LightGBM?