What this model is for
Working as a goose agent, with Atlassian administration, Jira, Confluence, Jira Service Management, Forge and migration knowledge. It is Qwen3.8-27B with a new LoRA adapter trained from the base model. It does not build on v1 (Qwen3.8-27B-Atlassian, the previous release): no v1 weights, no pull toward v1 and no text v1 wrote. In a goose session the model should:
- call only tools that exist in the session
- fill in every required argument (an edit names its file)
- not repeat the same call over and over
- answer directly when told "just tell me" or "no files", without searching or writing files
- ask when a decision belongs to someone else instead of making it
- back what it says with tool output: no file called done that was never written, no count that no tool returned
- never greet, sign or name a person nobody mentioned
For generating Forge apps it is close to v1 but slightly behind: on 35 Forge app briefs (3 samples each) v2 produced 27.33 complete apps of 35 on average against v1's 30, and 29.33 valid manifests against v1's 31.33.
This PEFT adapter on the bf16 base is the reference build of this release. The 8-bit MLX build is not equivalent to it: it matches full precision on general text (KL 0.001, top-1 99.5%), but drifts from it on goose agent contexts up to 16,384 tokens long (KL 0.400, top-1 89.9%); the gate is KL at most 0.067 and top-1 at least 90%. The 6-bit and 4-bit builds are lossier still.
Other files of this release: 8-bit MLX, 6-bit MLX, 4-bit MLX.
Against v1 in one paragraph
On Atlassian knowledge this model is about level with v1: higher on multiple choice (74.4% vs 71.9%; paired difference +2.5 points, 95% CI -3.94 to +8.96, McNemar p 0.54, not statistically significant) and lower on Forge free-form questions (63.3% vs 86.7%; paired difference -23.3 points, 95% CI -38.47 to -8.20, McNemar p 0.016, statistically significant), both on the shipped 8-bit builds. At tool calling it is better: the public BFCL benchmark 91.8 vs 85.7 (right call and arguments) and 85.8 vs 76.7 (declining tools that do not fit), both statistically significant (the 95% bound of each difference favours v2). In the held-out goose sessions these metrics are not shown non-inferior to v1: does not ask the decider the task names, and none of the 4 headline differences is statistically significant. At held-out slip points v2's rate is significantly lower for: searches or writes files after "just tell me" / "no files" (45.2% vs 57.7%, 13 slip points); calls a tool the session does not offer (12.5% vs 22.5%, 50 slip points). For generating Forge apps v1 is still slightly ahead: on 35 Forge app briefs (3 samples each) v2 produced 27.33 complete apps of 35 on average against v1's 30, and 29.33 valid manifests against v1's 31.33.
Why this round starts from the base model
v1 greeted people by forum handle, quoted forum posts and named people nobody mentioned, even in plain admin answers. The cause was in v1's own training data: 131 of its 220 Atlassian Community threads carry a greeting by handle, a handle, a forum quote block or a sign-off (handle 104, greeting 91, closing 35, sign-off 20, quote 11, owner 2). A round that continued from v1 inherited the habit; corrective data lowered it but could not remove it. This round therefore trains a fresh adapter from the base model.
Before training, the base model was checked on the same reply-manners prompts: a person or handle not in the prompt in 0 of 120 replies.
The finished mix was scanned row by row for the same shapes: 0 of 50095 rows (the scan refuses the build on a single hit), and 0 rows were written by v1.
What we did
- Training. A LoRA adapter, rank 128, scale 2.0, on the last 32 blocks, initialised fresh (the first step computes the base exactly), learning rate 2.0e-05, rows up to 65536 tokens, no KL pull toward any earlier model.
- The mix: 50,095 rows: Atlassian and Forge knowledge rewritten from the documentation 30,473; general and thinking-on replay (the base's own answers) 10,579; Forge apps, code and fixes 7,609; admin, organisation and identity rows (no persona; base-written) 778; goose agent sessions 527; voice (the owner's own replies) 129. 36,940 targets were written by the base model itself from a source passage or prompt, 405 goose rows keep their teachers' turns, and the rest are kept rows of the earlier rounds that v1 did not write. Rows dropped while building it: 474 (gen thinking: spare not needed (the target count is met) 275; general: person name the context does not carry (home scan) 103; thinking: person name the context does not carry (home scan) 55; admin: person name the context does not carry (home scan) 27).
- Booster. After the main run, a short continuation of the adapter (1,268 rows, 182 steps: docs 760, forge 194, replay 171, identity 143) taught Forge module keys from developer.atlassian.com sentences and an identity line ("Qwen3.8-27B, fine-tuned by LeanZero", denying being another assistant). It taught the keys behind 20 of the 30 Forge test questions; the other 10 were held out from every booster row (seed 20261004), so the Forge score separates taught keys from held-out ones.
- Forge app fix. A late release check (35 Forge app briefs, each answer checked by Forge's manifest linter, its module allowlist and the TypeScript compiler) showed the booster checkpoint built a complete app only 9.67 times in 35 against v1's 30: the main mix had diluted validated Forge app code to about 4%. A further 388-step continuation (2,714 rows: 2,127 validated Forge app, manifest and fix examples, plus 22% replay of the admin, organisation, identity, knowledge and voice rows) brought it to 27.33. Rows that taught a held-out Forge question key or matched one of the 35 test briefs were removed before training. It cost Forge question answering: 63.3% against 73.3% before this step.
- Checkpoint choice. The released adapter is step 7157 of the main run, plus 91 booster steps, plus the 388 Forge-fix steps (the last of three Forge-fix checkpoints). The owner chose it over the pre-fix checkpoint after reading the trade above. It did not pass every pre-written selection gate: one admin answer was flagged for a code annotation (not a person), one invented date in 320 organisation answers, 2 loops in the 160-answer battery against v1's 1, Forge free-form 63.3% vs v1 90.0% on the training machine, and the thinking-block format leg of the keep suite; the author's name appeared in 1 goose output where nothing in the task grounded it.
Measured against v1
Paired on identical items in bf16 on the training machine. The shipped 8-bit build is not equivalent to it: it matches full precision on general text (KL 0.001, top-1 99.5%), but drifts from it on goose agent contexts up to 16,384 tokens long (KL 0.400, top-1 89.9%); the gate is KL at most 0.067 and top-1 at least 90% (see the release gates). Neither version trained on the goose items or the general tests below; for the Forge questions see the note under Atlassian knowledge.
Goose agent work
Defect rates, lower is better. The goose FINAL split is a set of held-out goose scenarios, read once for this release. Held-out slip points are moments in held-out goose tasks where v1 made a mistake, sampled again for both versions. SUPERIOR / LOWER means v2's rate is lower with p < 0.05 (one-sided); "not shown" means the difference is not significant; NON-INFERIOR means not worse beyond a margin set before the test.
Table with columns: aim, defect measured, test set, v1, v2, verdict| aim | defect measured | test set | v1 | v2 | verdict |
|---|
| calls only tools the session offers | calls a tool the session does not offer | goose FINAL split, 270 items | 0.2% | 0.1% | not shown superior |
| calls only tools the session offers | calls a tool the session does not offer | held-out slip points, 50 points | 22.5% | 12.5% | LOWER |
| fills in every required argument |
Other goose checks:
Table with columns: defect measured, test set, v1, v2, verdict| defect measured | test set | v1 | v2 | verdict |
|---|
| repeats the call that just failed | goose FINAL split, 14 items | 1.8% | 0.0% | NON-INFERIOR |
| writes a tool's output as its own text | goose FINAL split, 270 items | 0.0% | 0.0% | NON-INFERIOR |
| reports a delegate's result no delegate call returned | goose FINAL split, 223 items | 0.0% | 0.0% | NON-INFERIOR |
General skills kept
Table with columns: test, v1, v2, verdict| test | v1 | v2 | verdict |
|---|
| MMLU (general knowledge) (n=1500) | 83.000 | 84.333 | PASS |
| GSM8K (grade-school maths) (n=1319) | 95.754 | 95.603 | PASS |
| IFEval (instruction following) (n=541) | 82.440 | 81.516 | PASS |
| HumanEval (code) (n=164) | 95.732 | 96.341 | PASS |
Base, v1 and v2 on the training machine (bf16)
Table with columns: check, base, v1, v2| check | base | v1 | v2 |
|---|
| Atlassian multiple choice (199) | 68.3% | 69.3% | 76.4% |
| Forge free-form (30) | 0.0% | 90.0% | 63.3% |
| admin answers naming someone (320 sampled + 40 greedy) | 1 | not measured | 1 |
| invented tickets, hosts, ids, dates or people | 28 | not measured | 1 |
Atlassian knowledge, shipped 8-bit
Accuracy, greedy, thinking off. Neither version saw the multiple-choice questions in training. The Forge free-form prompts were unseen too, but the booster taught the module keys behind 20 of the 30 Forge questions; 10 were held out from every booster row. The held-out 10 were not scored separately for both versions in the results this card was built from.
Table with columns: area, questions, v1, v2| area | questions | v1 | v2 |
|---|
| Jira | 46 | 78.3% | 76.1% |
| JSM | 45 | 73.3% | 71.1% |
| Confluence | 45 | 57.8% | 66.7% |
| admin | 45 | 77.8% | 80.0% |
| migration |
Reply manners
The same 60 synthetic prompts for both, no persona; ticket replies out of 20, greedy / sampled:
Table with columns: ticket replies, v1, v2| ticket replies | v1 | v2 |
|---|
| signs with a name | 11 / 11 | 0 / 0 |
| greets a person by name or handle | 20 / 18 | 0 / 0 |
| names a person or handle not in the prompt | 20 / 18 | 0 / 0 |
| claims work it was not told was done | 5 / 5 | 4 / 2 |
Admin and general questions, v2: a person or handle not in the prompt in 0 of 80 replies (v1 7); the author's own name in 0 of 120 replies.
Voice, a minor part of this round (persona prompt, held-out real replies): distance to the author's own style v1 3.069, v2 0.6546 (lower is closer; v2 is closer, and the difference is statistically significant).
Where v2 is worse than v1
- Forge free-form questions, shipped 8-bit: 63.3% vs v1 86.7% (30 questions; paired difference -23.3 points, 95% CI -38.47 to -8.20, McNemar p 0.016, statistically significant).
- Forge free-form questions, bf16 on the training machine: 63.3% vs v1 90.0%.
- Forge module identifiers, shipped 8-bit, thinking on: named before thinking 10 of 13 vs v1 11 of 13, after thinking 0 of 13 vs v1 2 of 13.
- Forge app building: on 35 Forge app briefs (3 samples each) v2 produced 27.33 complete apps of 35 on average against v1's 30, and 29.33 valid manifests against v1's 31.33.
- Long goose agent contexts, 8-bit MLX build (not this adapter, which is the reference build): it matches full precision on general text (KL 0.001, top-1 99.5%), but drifts from it on goose agent contexts up to 16,384 tokens long (KL 0.400, top-1 89.9%); the gate is KL at most 0.067 and top-1 at least 90%.
- IFEval (instruction following): v2 81.52 vs v1 82.44. It passed only under this round's rule (point estimate within 1.5 points of v1); the pre-registered paired bound of 3 points is not met (bound 3.62).
- General skill thinking-on maths, non-empty think blocks: v2 50.333 vs v1 52.333 (FAIL).
- Goose metric does not ask the decider the task names: v2 100.0% vs v1 75.0% (FAIL (possible degradation)).
- Loops in the 160-answer battery: 2 vs v1 1.
- Voice feature greeting: v2 31.2% vs v1 47.9% of replies, outside the author's own range of 35.4%-54.2%.
- Voice feature question: v2 14.6% vs v1 39.6% of replies, outside the author's own range of 29.2%-45.8%.
Release gates
- Merge: KLD(fused 8-bit to base+adapter) 0.037, top-1 99.5% (gate 0.067 / 90%)
- bf16 to 8-bit (MLX bf16 + adapter as the reference): goose agent contexts KL 0.400, top-1 89.9% FAIL; general text KL 0.001, top-1 99.5% PASS; all 5120 positions KL 0.201, top-1 94.7% (gate: KL at most 0.067, top-1 at least 90%); FAIL, released under a waiver (see Limits)
- MTP speculative decoding PASS, LM Studio load FAIL, released under a waiver (see Limits)
- Loops in the 160-answer battery (bf16, training machine): 2 (v1 1, measured on the 8-bit path); on the shipped 8-bit build, greedy 2 loops (base 3); long-context loops 128k: 0 of 1; needle recall 128k 100.0%
- Forge module identifiers, shipped 8-bit, thinking on: named before thinking 10 of 13, after thinking 0 of 13 (v1 11 of 13 / 2 of 13) PASS
- Forge app building, 35 briefs x 3 samples: 27.33 complete apps, 29.33 valid manifests (v1 30 apps, 31.33 manifests) FAIL, released under a waiver (see Limits)
- Privacy: 309 extraction probes (client questions, completions of the voice training text, names behind placeholders, canaries): 0 hits; repeated on the uploaded files of this repo before it was made public.
Use
from transformers import AutoModelForImageTextToTextfrom peft import PeftModelbase = AutoModelForImageTextToText.from_pretrained('Qwen/Qwen3.8-27B', dtype='bfloat16')model = PeftModel.from_pretrained(base, 'Mihai-LeanZero/Qwen3.8-27B-Atlassian-v2-goose-lora-peft')
Limits
- The blind 'which reply did the author write' judge was dropped: a local judge could not tell voices apart once reply length was matched; the voice verdict rests on voice-lint features against the owner's own distribution.
- No reinforcement-learning stage was run.
- The long end-to-end goose session (a Jira migration-readiness brief through the desktop app) was not run for this release.
- The two judged goose metrics (answering the literal ask first, surfacing a conflict instead of picking a side) were not graded for this release.
- The thinking-on maths test of the general suite is left out of the table: 149 of its 300 items errored and were scored 0, so its score, and the think-block rate on the same items, does not measure what it names.
- The LM Studio load gate was not met; the release went ahead under a waiver decided by the release coordinator, under the owner's decision to go ahead. Measured: not tested in LM Studio: LM Studio 0.4.23 on the release machine indexes no local models at all (lms ls lists only LM Link peers), unchanged since 2026-10-03 and identical for any model; the fused 8-bit loads and answers under MLX in every other release gate (KLD vs base+adapter 0.0368 / top-1 99.47%, MTP 1.35-1.56x).
- The 8-bit versus full-precision agreement gate was not met; the release went ahead under a waiver decided by the release coordinator, under the owner's decision to go ahead. Measured: 8-bit vs full-precision MLX with the adapter: general text KL 0.001, top-1 99.5% (pass); long goose contexts KL 0.40, top-1 89.9% (gate 0.067 / 90%); the same pattern measured on the earlier checkpoint, where full-precision MLX itself drifted from the PyTorch reference on the same long contexts.
- The general skills kept gate was not met; the release went ahead under a waiver decided by the release coordinator. Measured: thinking-block format leg 50.33 vs v1 52.33 (the round's rule fails it); every other general skill holds: MMLU 84.33 vs 83.0, GSM8K 95.60 vs 95.75, IFEval 81.52 vs 82.44, HumanEval 96.34 vs 95.73, BFCL 91.79 vs 85.71 and 85.83 vs 76.67.
Where the data came from
- No reply the model was trained to give was written by Claude: the targets are the base model's own answers (knowledge rows rewritten from Atlassian's own documentation text, general and thinking replay, the admin and organisation rows), documentation sentences copied from developer.atlassian.com, the goose teachers' turns, and the owner's own replies.
- No real working sessions are in the training data: the planned slice of treated sessions did not pass its independent privacy audit and was left out. The goose agent data is synthetic, and the voice slice was treated and audited locally (a dictionary scan, an independent audit and extraction probes on the uploaded files all came back clean).
- The voice slice is the owner's own hand-typed replies.
- Teachers for the synthetic goose tasks: DeepSeek V4.1 Flash and MiMo-V2.6-Pro (both MIT-licensed); this round keeps their own prose (the earlier rewording by v1 was dropped with every other v1-written row).
- Some question prompts (the booster's Forge documentation questions and the identity prompts) were written from templates by the build tooling.