This example: GPT-2 movie-review sentiment
GPT-2 small (124M parameters, frozen) fine-tuned for movie-review sentiment (SST-2). The circuit generates the
weight changes, so training adjusts only 1,692 circuit angles (plus a 1,536-number classification head).
LoRA rank 4 would train 147,456 numbers for the same layers.
Table | |
|---|
| Accuracy (SST-2 validation, 872 reviews) | 86.7% (756 / 872) |
| Trainable numbers | 3,228 (1,692 circuit angles + 1,536 head) |
| Adapter file size | 15 KB |
| Circuit per layer | 14 qubits, 10 rounds of RY rotations + a CNOT ring |
Use this adapter
pip install quantum-adapterhf download aishwaryawambule/gpt2-sst2-quantum-adapter --local-dir sst2-adapterquantum-adapter predict --adapter sst2-adapter "A beautiful, moving film."# positive 100% A beautiful, moving film.
Or from Python:
from transformers import AutoModelForSequenceClassification, AutoTokenizerfrom quantum_adapter import QuantumModel tok = AutoTokenizer.from_pretrained("gpt2")tok.pad_token = tok.eos_tokenbase = AutoModelForSequenceClassification.from_pretrained("gpt2", num_labels=2)base.config.pad_token_id = tok.pad_token_idmodel = QuantumModel.from_pretrained(base, "aishwaryawambule/gpt2-sst2-quantum-adapter") logits = model(**tok(["A beautiful, moving film."], return_tensors="pt")).logitsprint(["negative", "positive"][logits.argmax(-1).item()])
How it was trained
With the library's own command, on 4,000 SST-2 training sentences (10% held out for picking the best epoch):
quantum-adapter train --csv sst2_train.csv --out sst2-adapter --model gpt2 --lr 3e-3 --seed 0
The training output (on an Apple M5 Pro, about 8 minutes):
4000 rows, 2 labels ['negative', 'positive']: training on 3601, validating on 399.trainable params: 3,228 || all params: 124,443,036 || trainable%: 0.0026epoch 1/6: train loss 0.6641 validation accuracy 75.4%epoch 2/6: train loss 0.4193 validation accuracy 82.2%epoch 3/6: train loss 0.3827 validation accuracy 84.7%epoch 4/6: train loss 0.3530 validation accuracy 83.2%epoch 5/6: train loss 0.3509 validation accuracy 83.2%epoch 6/6: train loss 0.3313 validation accuracy 81.2%kept epoch 3 (best validation accuracy 84.7%)Saved the adapter to sst2-adapter/. Try it: quantum-adapter predict --adapter sst2-adapter "some text"
6 epochs, batch size 32, circuit learning rate 3e-3, head learning rate 1e-3, no weight decay. Validation
accuracy on the 399 held-out rows peaked at epoch 3, so that epoch is the saved adapter. It was then scored on
the 872 SST-2 validation reviews it never saw: 86.7% (756 / 872).
Two predictions from the saved adapter:
$ quantum-adapter predict --adapter sst2-adapter "A beautiful, moving film." "Two hours of my life I will never get back."positive 100% A beautiful, moving film.negative 84% Two hours of my life I will never get back.
Honest notes
- The circuit is simulated on an ordinary CPU or GPU. No quantum hardware is needed to train or use it, and
the trained adapter can be merged into GPT-2's weights for zero extra cost at inference.
- In the library's benchmark (2,000 examples, 3 seeds), this method came within 0.4 points of LoRA at 6 epochs
and beat an equal-size classical generator by 0.8 points in every seed. That is one task so far; see the
benchmarks.
- GPT-2 is a 2019 model; this adapter is a demo of the method, not a state-of-the-art sentiment model.
Train your own
See the box at the top, and the README for every option.