This run
Fresh from Qwen3-8B with YaRN x4 (128k) on v1 + v2 concatenated (1,982 sequences, 109.7M supervised tokens), lr 1e-5, 248 steps (one epoch). Train loss first 6 / last 6 steps 1.21 / 0.78. Finished 2026-10-28.
Table with columns: this model, base | this model | base |
|---|
| held-out NLL, v1 / v2 validation | 0.749 / 0.795 | 1.034 / 1.167 |
| EDIT-Bench thinking / fence (108) | 31 / 34 | 36 / 46 |
| SWE-bench (77), claude-code / cca | 1 / 4 | 3 / 12 |
| Terminal-Bench (56), claude-code / cca | 0 / 0 | 1 / 4 |
Best fit to the traces of any 8B run, worst agent benchmarks: the trajectory text is learned, the agent behaviour is not.
Serving notes
- Config as saved by transformers 5.14.1 from the YaRN x4 base dir:
rope_scaling yarn factor 4.0 over
32,768, rope_theta 1e6, max_position_embeddings 131072. Keep both rope keys (a legacy rope_scaling without
rope_theta resets theta to 10,000 under transformers 5).
- The saved
tokenizer_config.json does not load under transformers 4.x; use the base Qwen/Qwen3-8B tokenizer there.
Training setup (all runs)
Full-parameter trajectory-level SFT with train_cca_sft.py from github.com/rifaaQ/long_ctx_train on Aurora (Intel XPU):
one training sequence per agent context generation, loss on the assistant spans only, 128k context, bf16,
DeepSpeed ZeRO-3 + CPU optimizer offload, node-local Ulysses sequence parallelism (8 ranks), AdamW, cosine schedule,
8 sequences per optimizer step, 1 epoch unless noted. Data: CCA agent trajectories from cca-large-v1
(250 training runs, 1,243 sequences, 61.9M supervised tokens) and cca-large-v2 (182 runs, 739 sequences,
47.7M supervised tokens; disjoint runs, open-weight teachers only).
Research artifact. Evaluation protocol and all numbers: longctx/results.md in the project record (github.com/rifaaQ/long_ctx_train).