Layer-delta analysis (what was measured before merging)
Before merging, the two fine-tunes were compared tensor-by-tensor against their common base in fp32, streamed from safetensors with no full-model load. For every tensor t: ΔD = D(t) − base(t), ΔS = S(t) − base(t), then tensors were grouped (per text layer 000–063, vision groups, embeddings+lm_head, mtp/other) and aggregated.
1. Structural validation. All three models share an identical 1184-tensor skeleton; a strict check of every tensor's dtype and shape ran first (abort on any shape mismatch — none found). The only structural asymmetry was dtype-only (bf16 vs fp32 scale-like tensors), and a small set of tensors present only in DaisyRose was inventoried and handled separately.
2. The two fine-tunes moved in essentially orthogonal directions. The cosine similarity between ΔD and ΔS is ≈ 0 for nearly every layer (|cos| < 0.003 for all 64 text layers; e.g. embeddings_and_lm_head cos = −0.0001). The two fine-tunes are not two points along one training path — they are truly divergent soups, which is exactly the regime where parameter averaging adds rather than cancels knowledge.
3. Delta magnitudes by depth. Both fine-tunes drift more in mid-depth layers: |ΔS| grows from 0.44 (embeddings) to a peak of ~0.51 around layers 24–33, then decays to ~0.26 at the deepest layers; |ΔD| follows the same shape with smaller amplitude (peak ~0.37 at layers 16–26, → 0.28–0.30 late). sabi-noema is the stronger mover overall (norm ~0.44–0.50 vs ~0.34–0.37 for DaisyRose).
4. The 50/50 average is variance-preserving here. For orthogonal deltas, the merged deviation norm is |ΔD + ΔS|/2 = ½·sqrt(|ΔD|² + |ΔS|²) ≈ 0.30 per layer — i.e. the merge retains ~100% of the combined fine-tune signal (uniform-weight averaging never loses combined signal for orthogonal deltas; it loses up to 50% only when deltas are parallel). No layer displayed the "harmful" regime (large positive cosine), so no per-layer reweighting was applied — plain linear averaging everywhere.
Merge procedure
- Streaming merge (one tensor resident at a time), fp32 arithmetic, output cast back to bf16:
out = (D(t) + S(t)) / 2.
- Identical-tensor shortcut: the entire vision tower is bit-identical between the two fine-tune deltas (ΔD = ΔS = 0) — vision weights pass through unchanged.
- Byte-balanced sharding into 3 safetensors shards + regenerated index; 1184/1184 tensors verified post-merge.
- No permutation/alignment issues: both fine-tunes share exact tensor naming and ordering, so the delta analysis doubles as a merge alignment check.
This bf16 checkpoint is the source of truth for the NVFP4 release above.
Usage (vLLM)
vllm serve vince62s/Qwen3.81-27B \ --served-model-name qwen3.81 \ --reasoning-parser qwen3 \ --max-model-len 32768
- Requires ~55 GB+ for weights — use tensor parallelism (2×48 GB) or the NVFP4 version for single Blackwell GPUs.
- Chat template follows the Qwen3.5 thinking format;
enable_thinking defaults to on with xhigh reasoning effort.
- Sampling (
generation_config.json): temperature: 1.0, top_p: 0.95, top_k: 20.
- Multimodal:
image_token_id 248056, architecture Qwen3_5ForConditionalGeneration.
Evaluated on the NVFP4 build (behaviorally equivalent; same merge):
Table with columns: Benchmark, Setting, Result| Benchmark | Setting | Result |
|---|
| GPQA Diamond (mirror, 198 q) | lm-eval 0.4.13, loglikelihood | 43.9% |
| GPQA Diamond (CoT gate, 10 q) | generative, temp 0.6 | 9/10 |
| GPQA Diamond (CoT gate, 10 q) | generative, temp 1.0 (server sampling) | 8/10 |
| Terminal-Bench 2.1 (subset, pass@1) | harbor + terminus-2, 3× task timeouts | ~8/10 early batches; full 89-task suite in progress |
Terminal-Bench notes for reproducibility: Terminus-2 must be told the true context window via --ak model_info={"max_input_tokens":40000,"max_output_tokens":8192} — otherwise it assumes ~1M tokens, overflows a 49k server, and wedges in a summarization retry loop. With the fix, episodes self-trim and complete.
Intent
Qwen3.81 combines DaisyRose-27B's agentic/terminal strengths with sabi-noema's hybrid-attention reasoning base in one open 27B checkpoint, published in both bf16 (this repo) and NVFP4. Analysis scripts (analyze_deltas.py, merge_models.py) replicate the delta analysis and merge with no additional tooling beyond torch and safetensors.