JamieBradfield

qwen3.8-9b-hermes-fc-clean-restraint

README

License: apache-2.0

Addendum 2026-09-03 — evaluation-methodology correction

Read this before you use the numbers below.

After this checkpoint was published, we tested the base model (Empero/Qwen3.8-9B-Distill, no fine-tuning) through the live Hermes interface: the OpenAI-native tools parameter path that Hermes actually uses. The base model calls tools correctly on its own.

The harness that certified this checkpoint did not use that interface. It baked tool schemas into the system text, never passed tools, and required raw <tool_call> XML in the response. That prompt shape matches neither the base model's own chat template nor the live Hermes request path. Every gate score in the v26-to-v29 lineage measured this synthetic interface.

The statement in this README that "the base distill contributes zero tool-calling" is incorrect. It measured the synthetic interface.

Corrected comparison (native tools path, same 4-tool battery, greedy):

Table with columns: model, tier-1 fired, tier-1 name=todo, tier-2 todo-first, tier-3 fired (want 0), tier-4 fired (want 0)
model	tier-1 fired	tier-1 name=todo	tier-2 todo-first	tier-3 fired (want 0)	tier-4 fired (want 0)
base (no fine-tune)	18/20	17/20	3/10	5/10	0/10
v28 ckpt-175 (this model)	17/20	10/20	8/10	9/10	4/10
armB2-175 (clean retrain)	10/20	8/20	10/10	4/10	0/10

What the fine-tune lineage actually changed: todo-first planning on multi-step tasks (tier-2 name=todo 3/10 base to 8/10 v28 to 10/10 armB2). That is real, and it is the one capability the lineage added. The costs: restraint regressions (tier-3 over-trigger 5/10 to 9/10; tier-4 tool-drift on trivia 0/10 to 4/10) and, in armB2, over-suppression (tier-1 10/20).

Status: the fine-tune program is frozen as of 2026-09-03. The base model is the author's runtime default. This checkpoint stays published as the strongest measured demonstration of SFT-taught todo-first planning in the lineage, and as the artifact that exposed the harness error. The Evaluation section below describes behavior on the synthetic interface; do not read it as native-interface competence.

Reproduction: scripts/native_battery.py + scripts/todo_probes.jsonl

scripts/no_drift_probes.jsonl produce the corrected table (any GGUF, CTX=65536 to match the serve config).

Base model: Empero/Qwen3.8-9B (Apache-2.0)
Warm start: v27 merged weights (qwen3.8-9b-hermes-fc-todo), QLoRA adapter on top (envelope + tool tokens already baked in)
Training data: 218 trajectories (ShareGPT format, 551 todo calls), generated by GLM-5.3-Flash via the HuggingFace router on 112 synthetic tasks mined from the author's agent corpus — not the author's own traces:
- 190 teacher trajectories (85% verified by sandbox execution; failures with recovery injected for 15.8% of rows)
- 61 rows carry distilled short-chain-of-thought; 159 envelope-only
- 12 synthetic web-search tasks (first web signal)
- Full dataset published separately (see hermes-fc-v28-data); probes were kept virgin (zero overlap with generation or training)
Method: QLoRA 4-bit, rank 16 / alpha 16, dropout 0, batch 1 x grad-accum 8 (effective 8), lr 2e-4, warmup 0.1, MAX_SEQ 6144, 8 epochs = 214 steps (final loss 0.02085, avg train loss 0.09405)
248077 → 248079 (2 added tool tokens: , )

Evaluation (50-probe battery, greedy, @ Q4 64k ctx)

Tiers: T1 = 20 real DJLougen todo traces held out of ALL training generations (GT = first todo call). T2 = 10 synthetic multi-step tasks → autonomous todo reach-for. T3 = 10 trivial asks (want 0/10) → over-trigger guard. T4 = 10 off-topic chat asks (want 0 fired, 0 topic drift into fine-tune vocabulary like rocm/quant/llama-swap).

Table with columns: tier, v28-175 (control, warm), armB2-175 (this model)
tier	v28-175 (control, warm)	armB2-175 (this model)
T1 fired	17/20	6/20
T2 fired / fmt-exact	8/10 / 4/10	10/10 / 10/10
T3 fired (want 0)	5/10	1/10
T4 fired / drift (want 0/0)	2/10 / 0	0/10 / 0

What this model proves (the experiment's result):

Contamination is real and this arm fixes it. v28-175 fires on 2 of 10 off-topic chat probes (learned tool reflex from ~1900 rocm-mention rows in its warm lineage); armB2 fires 0/10 with zero topic drift. Clean base
- filtered corpus = no tool reflex on chat.
Restraint works. T3 1/10 vs v28's 5/10 — the 87 no-call rows teach when NOT to reach for a tool.
When it fires, it is clean. T2 10/10 with 10/10 format-exact (bare envelopes, no prose wrap) — the best format score in the lineage.
Perplexity on a held-out probe ruler: Q4 1.148 vs Q8 1.145 (the published Q4 quant is effectively lossless).

Known limitation (why this is experimental — read this)

Tier-1 collapse on assistant-final transcripts. When a conversation ends with an assistant narration that in the real trace continued into a todo call, this model emits nothing (<|im_end|> immediately) instead of continuing. 13 of 20 T1 probes end that way; all 13 fail. Root cause (measured): 37% of its training rows end on bare narration (the restraint rows teach "answer in text, then stop"), and 12 of those 20 probes also carry the harness cap message ("maximum iterations… without calling any more tools") — this model obeys it, where v28-175 ignores it and fires anyway.

Live impact: on fresh turns (user ask or tool result → model's next turn) it calls correctly — T2's 10/10 is that shape. The stall only appears when the model is asked to continue from its own prior narration (real traces cut mid-flow, or a truncated continuation).

Fix in training: v29 armB2-r4 adds fused narration+call rows and cap-message rows ending in a proper text summary. When it lands it will be published as the follow-up; this checkpoint exists so the restraint / contamination findings are testable now.

Do not use this as a drop-in replacement for the current default in long agentic sessions without the continuation fix. For general tool use today, prefer qwen3.8-9b-hermes-fc-tooluse (v28). Use this model to test restraint, contamination-free chat, and format cleanliness.

Intended use

Research artifact for experimenters working on tool-call behavior in 9B-class models — not a product. The tool_call XML envelope and tool schemas match the Hermes agent runtime this was trained on. v29 is an ongoing A/B (arm A = v28-175 control, arm B = this clean-base treatment, arm B2 = this checkpoint's retrain lineage).

What this repo contains

The BF16 merged weights (this repo)
All scripts that produced the model and dataset (scripts/)

Reproducibility

All scripts that produced this model are in scripts/:

Table with columns: file, purpose
file	purpose
`build_v26.py`	shared helpers (build_system, pad_to_bucket, conv_hash, constants)
`filter_legacy_meta.py`	meta-keyword filtering of legacy corpora (META_KEYWORDS on first human prompt)
`trim_long_rows.py`	row token ceiling (drop >5100 tok rows for the OOM-safe corpus)
`build_armb2_corpus.py`	arm-B2 assembly: 217 v28-glm + 60 todo_real + 72 todo_condensed + 87 restraint rows
`qwen38_train.py`	QLoRA train (Unsloth; 214 steps; ckpts 175/200/214)

The scripts carry the author's machine paths (F:/models/..., C:/AI/...) and run on this rig's ROCm stack (Unsloth 2026.8.18, torch 2.11 ROCm, RX 7700 XT). The todo_real/todo_condensed rows derive from DJLougen Hermes traces (Apache-2.0); the restraint rows are real no-call asks from the author's own agent sessions — training data is not published (same policy as v26/v27). Zero author-session tool-call traces entered this corpus except through the published DJLougen lineage.

Convert/quantize: llama-rocmfpx fork,

markdown

convert_hf_to_gguf.py --outtype bf16

→ llama-quantize Q4_0_ROCMFP4_FAST (see GGUF repo).

Quantized version

Q4_0_ROCMFP4_FAST GGUF (ROCmFPX, AMD RDNA3 kernels) is published in the companion repo. ROCmFPX quants target AMD ROCm inference; for portable use, convert from the BF16 merge here.

Acknowledgements

Base model and license inherited from empero-ai/Qwen3.8-9B-Distill (Apache-2.0). Teacher trajectories from GLM-5.3-Flash via the HuggingFace router; real traces from DJLougen Hermes traces (Apache-2.0).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

JamieBradfield

Model Tree

Base

empero-ai/Qwen3.8-9B-Distill

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality