DistilBERT

Keras

Model Details

DistilBert is a set of language models published by HuggingFace. They are efficient, distilled version of BERT, and are intended for classification and embedding of text, not for text-generation. See the model card below for benchmarks, data sources, and intended use cases.

Weights and Keras model code are released under the Apache 2 License.

Links

  1. DistilBert Quickstart Notebook
  2. DistilBert API Documentation
  3. DistilBert Model Card
  4. KerasNLP Beginner Guide
  5. KerasNLP Model Publishing Guide

Installation

Keras and KerasNLP can be installed with:

pip install -U -q keras-nlp
pip install -U -q keras>=3

Jax, TensorFlow, and Torch come preinstalled in Kaggle Notebooks. For instruction on installing them in another environment see the Keras Getting Started page.


Example Use

import keras
import keras_nlp
import numpy as np

Raw string data.

features = ["The quick brown fox jumped.", "I forgot my homework."]
labels = [0, 3]

# Use a shorter sequence length.
preprocessor = keras_nlp.models.DistilBertPreprocessor.from_preset(
"distil_bert_base_en",
sequence_length=128,
)
# Pretrained classifier.
classifier = keras_nlp.models.DistilBertClassifier.from_preset(
"distil_bert_base_en",
num_classes=4,
preprocessor=preprocessor,
)
classifier.fit(x=features, y=labels, batch_size=2)

# Re-compile (e.g., with a new learning rate)
classifier.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=keras.optimizers.Adam(5e-5),
jit_compile=True,
)
# Access backbone programmatically (e.g., to change `trainable`).
classifier.backbone.trainable = False
# Fit again.
classifier.fit(x=features, y=labels, batch_size=2)

Preprocessed integer data.

features = {
"token_ids": np.ones(shape=(2, 12), dtype="int32"),
"padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]] * 2)
}
labels = [0, 3]

# Pretrained classifier without preprocessing.
classifier = keras_nlp.models.DistilBertClassifier.from_preset(
"distil_bert_base_en",
num_classes=4,
preprocessor=None,
)
classifier.fit(x=features, y=labels, batch_size=2)



Model Comments

0 comments

No comments yet.