Winner (speed)
Unsloth under valid protocol (lora_dropout=0, SDPA, shared text-field SFT path).
Validity rule
Unsloth results only count when the train log shows non-zero QKV/O/MLP patches, e.g.:
patched 36 layers with 36 QKV layers, 36 O layers and 36 MLP layers
lora_dropout=0.05 disables Unsloth LoRA fast-patching → invalid comparison.
Benchmark summary
HF–Unsloth Fine-Tuning Benchmark
Table with columns: Exp, Stage, Backend, Model, Len, BS, r, 4bit, GC, Attn, Time (s), Peak VRAM (GB), Tokens/s, GPU util., Train loss, Eval loss, Task acc.| Exp | Stage | Backend | Model | Len | BS | r | 4bit | GC | Attn | Time (s) | Peak VRAM (GB) | Tokens/s | GPU util. | Train loss | Eval loss | Task acc. |
|---|
| baseline-v2 | standard | hf | Qwen/Qwen3-4B-Instruct-2507 | 2048 | 1 | 16 | True | True | sdpa | 614.957 | 4.110 | 271.117 | 42.680% | 2.532 | 2.792 | n/a |
| baseline-v2 | standard | unsloth | Qwen/Qwen3-4B-Instruct-2507 | 2048 | 1 |
Each run directory contains its resolved config, metrics, saved adapter, and task evaluation when applicable.
Lessons
AI Engineer Lessons — HF vs Unsloth QLoRA (L4)
Living document. Updated after each valid experiment cell.
Core decision so far
Under valid protocol (lora_dropout=0, SDPA, shared text SFT path), Unsloth wins on throughput (~+27% tokens/s, −19% wall time) with slightly higher peak VRAM (+0.3 GB) on Qwen3-4B QLoRA / L4.
Invalid runs with dropout=0.05 showed the opposite — those are not Unsloth measurements.
Lesson 1 — Framework comparisons need validity gates
Table with columns: Mistake, Symptom, Fix| Mistake | Symptom | Fix |
|---|
lora_dropout=0.05 “because PEFT defaults” | Unsloth logs 0 QKV layers patched | Force lora_dropout=0 for both backends when measuring Unsloth |
Trusting install of xformers | ABI mismatch (torch/cuda/py) | Uninstall broken wheel; pin attn_implementation=sdpa (or install matching FA2) |
| Mixing chat-messages vs text-field paths | Different loss scales | Shared dataset_text_field="text" for both |
Engineer rule: Before reading the table, grep the Unsloth log for patched … QKV layers. If zeros → discard the cell.
Lesson 2 — What Unsloth actually optimizes
Unsloth is not “a different Trainer API for free speed.” It:
- Patches attention / MLP / LoRA kernels (requires dropout=0 for LoRA fast path)
- Uses custom gradient checkpointing (
use_gradient_checkpointing="unsloth")
- Often uses cut-cross-entropy (loss numbers ≠ HF)
- Still sits on TRL
SFTTrainer in our harness
So the fair A/B is: same data, seed, LoRA geometry, seq, batch, optim, attn backend — with Unsloth’s required constraints respected.
Lesson 3 — Method knobs > random hyperparam spray
Prefer experiments that change stack decisions:
Table with columns: Method, Why it matters, Use-case signal| Method | Why it matters | Use-case signal |
|---|
optim=adamw_torch vs adamw_8bit | Optimizer state VRAM vs step speed | Long SFT on 24GB; 8bit often frees VRAM for larger batch/context |
| Packing on/off | Throughput vs sample boundary noise | Short instruction data → packing helps; tool-call JSON → often pack-off safer |
| Target modules attention vs all-linear | Capacity vs VRAM/time | Format learning / tool schemas often need MLP too |
| GC on/off | Activation memory vs compute | Off only if VRAM headroom; else OOM |
| 4-bit QLoRA vs 16-bit LoRA |
Hyperparam rank/batch sweeps are secondary once the method is chosen.
Lesson 4 — Production path after the winner
Once Unsloth is the speed winner:
- Re-run method matrix on Unsloth only (optim / packing / targets)
- Domain SFT (appointment native tools) with Unsloth
- Eval with task metrics (tool-call / format / args) — not train loss
- Export adapter → optional merge → GGUF/vLLM → FastAPI
- Publish adapters + experiment log to Hub for CV / reproducibility
Open follow-ups
Security note
Never commit Hugging Face tokens. Use HF_TOKEN env var; rotate any token pasted into chat.
Reproduce
pip install -r requirements-benchmark.txt
python scripts/run_l4_benchmark.py --stage standard --lora-dropout 0 --attn-implementation sdpa
Repo card for Hikmet58/qwen3-4b-unsloth-qlora-l4-bench.
