Training
- Base revision:
1cfa9a7208912126459214e8b04321603b3df60c.
- LoRA rank 16, alpha 32; attention q/k/v/o and MLP gate/up/down projections.
- 52,706,304 trainable parameters: 33,030,144 LoRA plus 19,676,160 sampler.
- AdamW, learning rate 1e-4, betas (0.9, 0.999), no weight decay; gradient norm capped at 1.
- BF16 decoder computation, FP32 trainable parameter storage, activation checkpointing.
LoRA is exported in BF16; sampler weights and frozen mask vectors are exported in FP32.
- Global batch 16, microbatch 4; one epoch, 907 optimizer steps numbered 0 through 906.
- Dolly rows 0–14,499 train; rows 14,500–15,010 validate. Assistant-answer loss masking,
Qwen chat template, maximum 512 tokens, shuffled training, seed 1111.
- Objective: base CE plus conditioned-sampler CE on clean and future mask positions,
with detached latent consistency matching. CE is normalized per sample over its
supervised clean and mask positions; consistency balances matches per teacher position.
The original run's complete YAML was unavailable. The base model, adapter size,
mask count, data split, and epoch count were recovered from its metadata. Batch
size 16 was inferred from its step count; learning rate, sequence length, chat
formatting, and shuffle settings were reconstructed. This is a new run with the
corrected objective and saved sampler. Its exact configuration is in
training_config.yaml, with runtime versions in environment.json and source
files in automodel_source.tar.gz. The source files overlay the base commit
recorded in source_manifest.json.
Validation
Table with columns: Checkpoint step, Frozen-base CE, Teacher-forced acceptance diagnostic| Checkpoint step | Frozen-base CE | Teacher-forced acceptance diagnostic |
|---|
| 299 | 4.0624 | 1.4695 |
| 599 | 4.0624 | 1.6357 |
| 899 | 4.0624 | 1.7284 |
The last full held-out validation is at step 899. The final step-906 adapter
was checked separately for export and generation correctness.
The acceptance diagnostic compares sequential sampler drafts against frozen-base
greedy predictions on held-out teacher-forced prefixes and excludes ignored
anchors. It is not a decoding speed measurement or an autoregressive benchmark.
The final export is checked for finite tensors, exact sampler-state reload via
HF PEFT, unchanged logits with ordinary-token gating, and matching 16-token greedy
outputs on three prompts using draft verification. See adapter_validation.json
on main for those final-export checks.
Generation checks and measured throughput
These tests used the final step-906 weights at immutable revision
cd6f84de00dd1f5b9bc20e4ac585876290860f4f
on one NVIDIA H100 80GB, using greedy decoding with thinking disabled.
Two longer requests each generated 512 new tokens: an exercise guide and an
explanation of sky color and sunsets. Both outputs were coherent on manual
inspection, with connected sections and no repetitive collapse through the token
limit. Both matched the uncached base-model greedy output token-for-token in
these tests. Samples: exercise guide and
sky explanation.
After a full 512-token warm-up of each path, three measurements were taken per
prompt and method, alternating execution order. Values below are median output
tokens per second, including prefill and generation time:
Table with columns: Prompt, Cached base, Current MTP helper, MTP / base throughput| Prompt | Cached base | Current MTP helper | MTP / base throughput |
|---|
| Exercise guide, 512 tokens | 54.65 | 28.84 | 0.53x |
| Sky explanation, 512 tokens | 54.58 | 28.46 | 0.52x |
The baseline is the same PEFT-wrapped model with the adapter disabled, using
Hugging Face generate(use_cache=True). The MTP helper uses eight mask depths and
recomputes the prefix without a KV cache. The current helper is about 1.9x
slower in these warmed measurements; it does not demonstrate an inference
speedup. An earlier cold measurement suggested a speedup and did not hold up
under repeated warmed timing. These are small reference-implementation tests,
not a comparison with an optimized serving engine.
MTP was active: the long runs accepted an average of 0.38 and 0.36 extra MTP draft
tokens per verifier round, respectively. This count excludes the ordinary base
next token and the verifier's correction/bonus token. Many proposed deeper tokens
were rejected. Faster inference needs a KV-cached draft/verify implementation and
sufficient accepted drafts to offset drafting and verification overhead.
BF16 numerical behavior
In an eight-prompt short-generation test, seven outputs matched uncached base
greedy generation exactly. One sky-explanation continuation diverged at a close
BF16 logit decision: the ordinary prefix forward scored appears at 39.5 and
is at 39.25; batched verification rounded both to 39.5, changing the argmax
choice. Ordinary-token gating still preserved the base logits exactly when the
forward layout was identical. Cached and uncached base generation also differed
on the longer prompts. Greedy text is therefore not guaranteed to be bitwise
identical across BF16 batching and cache layouts. This limitation should be
considered separately from sampler loading, token gating, and text coherence.
Raw results and benchmark scripts are included in benchmarks/:
short_generation_results.json, long_generation_results.json,
repeated_timing.json, and bf16_diagnosis.json. The decoder is a single-prompt,
uncached greedy correctness reference, with no stochastic speculative sampler.
Usage
Download inference.py and gated_mtp_sampler.py from this revision alongside
the adapter, then use the included loader:
from inference import load_adapter, greedy_generate model, sampler, tokenizer = load_adapter("kashif/Qwen3-4B-Gated-LoRA-MTP")text = tokenizer.apply_chat_template( [{"role": "user", "content": "Name three benefits of regular exercise."}], tokenize=False, add_generation_prompt=True, enable_thinking=False,)inputs = tokenizer(text, return_tensors="pt").input_ids.to("cuda")outputs = greedy_generate( model, sampler, inputs, max_new_tokens=32, eos_token_id=tokenizer.eos_token_id,)print(tokenizer.decode(outputs[0, inputs.shape[1]:], skip_special_tokens=True))
To load an intermediate checkpoint, pass revision="step-299", "step-599", or
"step-899" to load_adapter. Ordinary generation must disable the adapter:
with model.disable_adapter(): outputs = model.generate(inputs, max_new_tokens=32, do_sample=False)
Standard PEFT loading requires attaching the sampler module to the base model
first. Plain PeftModel.from_pretrained does not create the sampler architecture,
token gating, or speculative decoding. The included helpers provide a single
unpadded prompt, uncached greedy reference implementation; they do not implement
KV-cache acceleration or stochastic speculative sampling. The examples were
checked with PyTorch 2.11.0, Transformers 5.18.0, and PEFT 0.20.0.
Earlier weights remain available through the archive-2026-11-14-* tags.
License
The adapter uses Apache 2.0. The training dataset is licensed CC BY-SA 3.0.