GPT2 Model performance dismal with enable_lora

[ashep29]

The following code trains fine - loss decreases, accuracy improves:

chosen_preset = "gpt2_base_en"

preprocessor = keras_nlp.models.GPT2CausalLMPreprocessor.from_preset(
chosen_preset,
sequence_length=128,
)

gpt2_causal_lm = keras_nlp.models.GPT2CausalLM.from_preset(
chosen_preset, preprocessor=preprocessor
)

learning_rate = keras.optimizers.schedules.PolynomialDecay(
5e-5,
decay_steps=train_ds.cardinality() * num_epochs,
end_learning_rate=0.0,
)
loss = keras.losses.SparseCategoricalCrossentropy(from_logits=True)
optimiser = keras.optimizers.Adam(learning_rate)
metrics = [keras.metrics.SparseCategoricalAccuracy()]

gpt2_causal_lm.compile(
optimizer=optimiser,
loss=loss,
weighted_metrics=metrics,
)
gpt2_causal_lm.fit(train_ds, epochs=num_epochs)

---

Unfortunately, when lora is enabled, the model training fails dismally - all settings remain the same, other than .quantize and enable_lora, but the loss increases and accuracy is extremely low (around 1% as opposed to 46% without lora).

gpt2_causal_lm_with_qlora = keras_nlp.models.GPT2CausalLM.from_preset(
chosen_preset, preprocessor=preprocessor
)

gpt2_causal_lm_with_qlora.quantize("int8")
gpt2_causal_lm_with_qlora.backbone.enable_lora(rank=4)

The model produces junk output, e.g.;

This

12

L

12

This

12

[26262626

12

[12.

[12.

[[.

[[. the r)

[2

Is there anything wrong with how I'm using lora or is it simply not implemented or available for use with GPT2?

[ashep29]

Note: the same model without qlora works nicely, e.g.

Epoch 1/3
ptxas spam warnings
2/960 ━━━━━━━━━━━━━━━━━━━━ 1:15 79ms/step - loss: 0.6675 - sparse_categorical_accuracy: 0.1776
ptxas spam warnings
959/960 ━━━━━━━━━━━━━━━━━━━━ 0s 60ms/step - loss: 0.6554 - sparse_categorical_accuracy: 0.3852
ptxas spam warnings
960/960 ━━━━━━━━━━━━━━━━━━━━ 172s 107ms/step - loss: 0.6554 - sparse_categorical_accuracy: 0.3853
Epoch 2/3
960/960 ━━━━━━━━━━━━━━━━━━━━ 60s 62ms/step - loss: 0.5610 - sparse_categorical_accuracy: 0.4341
Epoch 3/3
960/960 ━━━━━━━━━━━━━━━━━━━━ 61s 62ms/step - loss: 0.5279 - sparse_categorical_accuracy: 0.4554

<keras.src.callbacks.history.History at 0x7ff928290100>

output_finetuned = gpt2_causal_lm.generate(sentence_starters,max_length=max_length)

for sentence in output_finetuned:
print(f"--{sentence}\n")

--I bought a pair of crocs for my daughter, and they were very good.

--Crocs are so comfortable and convenient, and I have worn them to work, and I am very happy with them.

--Received a pair of crocs in white, and I love them! I'm a fan since I started wearing them.

[james77777778]

Hi @ashep29, would you mind sharing the outputs only after quantize?

Dynamic int8 quantization might fail if the value range of the weights is too large. Imagine trying to narrow an large range (such as -1e5 to 1e5) into an 8-bit capacity.

Most modern models should be okay and have similar performance after quantize, such as the Gemma and Llama models, but GPT2 is untested during my implementation.
Ref:

• https://keras.io/examples/keras_recipes/parameter_efficient_finetuning_of_gemma_with_lora_and_qlora/
• https://www.kaggle.com/models/keras/llama2/keras/llama2_instruct_7b_en_int8