Evaluation
Same protocol for every model in the portfolio: 60 stratified validation prompts
(6 per difficulty level, seed 42), temperature 0.6 / top-p 0.95 / top-k 20,
max 16,384 new tokens. Metrics: complete HTML document returned; pinned Three.js
0.160.0 module import present; extracted module script passes node --check.
Table with columns: Model, Complete HTML, Pinned import, JS syntax| Model | Complete HTML | Pinned import | JS syntax |
|---|
| Gold (dataset targets) | 60/60 | 59/60 | 59/60 |
| Qwen3.8-27B QLoRA (this model) | 59/60 | 59/60 | 58/60 |
| LFM3-1.7B full SFT | 16/60 | 16/60 | 3/60 |
| LFM3-1.7B-Base (zero-shot) | 40/60 | 7/60 | 16/60 |
Failure analysis of this model's outputs: 59/60 generations finish with a complete
HTML document, and 0 of the 59 checkable scripts fail with a duplicate-declaration
error — the failure mode that dominates the 1.7B model's fine-tune (11 of its 13
finished-but-invalid outputs) does not appear at this scale. Raw generations and
scores are in eval/.
Training
- Method: Unsloth QLoRA — 4-bit NF4 base, LoRA r=64 / alpha=128 / dropout 0 on the
language model (attention + MLP); vision tower frozen. Prompt-completion format
(loss on completions only), NEFTune alpha=5, effective batch 16, bf16,
gradient checkpointing, EOS forced to
<|im_end|>.
- Data: HTML-only targets (reasoning traces dropped). At max_len 12,288 the Qwen
tokenizer keeps 1,886 of 3,933 training scenes; the rest exceed the cap and
were excluded. This is a real limitation of the run, disclosed deliberately.
- Compute: single RTX PRO 6000 (96 GB) on molab, ~$0. Tracked on Weights & Biases
(project
fde-3dcoder-portfolio).
- Interruption history (honest record): training ran in two phases because the
ephemeral compute sessions terminated mid-run three times. Phase 1 reached step
160 of 236 (lr 1e-4 cosine) before its session died; phase 2 continued from the
step-160 adapter — which had been pushed to this repo by the every-20-steps
checkpointing — for one further epoch of 118 steps at lr 3e-5 cosine. No optimizer
state survived, so phase 2 is a continuation from the adapter weights, not an
exact resume. Total optimisation: 278 steps over the kept subset.
Intended use and limits
Research/portfolio artifact for Three.js scene generation in the dataset's format.
Outputs are single-file HTML scenes; generated code should be reviewed before use
in production. The model inherits the base model's general capabilities and limits
outside this task; it was not evaluated on general benchmarks.
Licence
Apache-2.0, following the base model. Dataset: MIT (ccore/ThreeJS-3D-Scenes).