Evaluation
Full test splits, chat-format, greedy. Run with scripts/eval_full_suite.sh.
Table with columns: metric, avg-top3| metric | avg-top3 |
|---|
| GSM8K (0-shot pass@1) | 18.38 |
| CitGen | 28.80 |
| SciQ (open-Q pass@1) | 14.90 |
| TextMan summary (chrF++) | 44.07 |
| TextMan rewrite (chrF++) | 48.45 |
| CitMC | 46.70 |
| SciQ-MC | 58.90 |
| ARC-Easy | 41.12 |
| ARC-Challenge | 27.30 |
| OpenBookQA | 35.40 |
PIQA (raw, length-norm logp) | 60.00 |
PIQA (chat-mc, letter generation) | 49.00 |
| GPQA-Diamond | 26.77 |
| SciQ (length-norm logp) | 42.80 |
| CitMC (length-norm logp) | 56.39 |
| ARC (length-norm logp) | 24.25 |
| IFEval-DA prompt-strict | 24.7 |
| IFEval-DA prompt-loose | 25.2 |
| IFEval-DA inst-strict | 38.9 |
| IFEval-DA inst-loose | 40.0 |
Both PIQA modes are listed because they differ by 11 points and the
conventions disagree: raw is the standard PIQA protocol (score each solution
as a continuation, pick the higher length-normalized log-prob), chat-mc asks
for a letter. Neither reproduces the 53.00 on the v31 cards, so the v31 figure
came from a third harness and is not comparable to either.
jensjepsen/danish-icl-schema-format-v3, exact match on the parsed answer.
Table with columns: split, held out, exact, key-set| split | held out | exact | key-set |
|---|
eval_schema | schema | 56.7 | 93.0 |
eval_format | format | 85.3 | 96.4 |
eval_both | schema + format | 47.8 | 86.8 |
eval_format overlaps danish-ner-sft-v1, which trains brace_pair
(2,276 rows) and the [x]...[/x] delimiter pair via spans_bracket. Of its
three held-out formats only kv_eq (82.7) is unseen in this mix.
NER
jensjepsen/danish-ner-sft-v1, exact match on the parsed answer and micro-F1
over (type, span) pairs. Both splits are built from DANSK test text. FAITHFUL
is span-wrap only: the answer both strips back to the source passage and
carries at least one well-formed tag pair.
jensjepsen/danish-lm-400m-sft-v31-avg-top3 is the same mix without
danish-ner-sft-v1 or danish-icl-schema-format-v3; both models were scored
by the same script on the same day.
Table with columns: split, metric, v31-avg-top3, v33-avg-top3| split | metric | v31-avg-top3 | v33-avg-top3 |
|---|
eval | exact | 0.2 | 58.7 |
| entity-F1 | 0.7 | 70.0 |
| span faithful | 0 | 73 |
eval_format | exact |
eval — formats seen in training, n=1500
Table with columns: format, v31 exact, v33 exact, v33 F1| format | v31 exact | v33 exact | v33 F1 |
|---|
brace_pair | 0.0 | 69.9 | 77.6 |
tsv | 0.0 | 66.7 | 74.5 |
numbered | 0.0 | 64.2 | 72.3 |
|
v33 span faithfulness on this split is 73% (strip 86, tagged 87, bare 12);
v31 is 0% (strip 2, tagged 25, bare 2) — it emits tag-like output but does not
reproduce the passage verbatim.
eval_format — formats held out of both danish-ner-sft-v1 and
danish-icl-schema-format-v3 training splits, n=1500
Table with columns: format, v31 exact, v33 exact, v33 F1, v33 faithful| format | v31 exact | v33 exact | v33 F1 | v33 faithful |
|---|
bracket_pair | 0.0 | 60.5 | 71.3 | — |
kv_eq | 0.0 | 53.1 | 66.4 | — |
spans_brace | 0.0 |
Key-value formats transfer to unseen delimiters; span-wrap does not. Three
span-wrap delimiters are trained (spans_angle, spans_bracket,
spans_paren) and the fourth returns the passage untagged in 70% of rows.
Table with columns: prompt mode, eval exact, eval_format exact| prompt mode | eval exact | eval_format exact |
|---|
icl | 60.4 | 42.9 |
both | 63.1 | 38.0 |
instruction | 48.4 | 36.0 |
Checkpoint comparison
Full test splits, scripts/eval_downstream_once.py, ICL capped at 1000 rows.
Table with columns: model, GSM8K, CitGen, SciQ, IFEval inst-strict, ICL, mean| model | GSM8K | CitGen | SciQ | IFEval inst-strict | ICL | mean |
|---|
step-25683 | 17.2 | 25.4 | 12.3 | 37.3 | 51.9 | 28.82 |
step-29352 | 15.3 | 26.8 | 13.0 |
Training
Table | |
|---|
| base | jensjepsen/danish-lm-400m-base-ropext8048-v1 |
| tokenizer | jensjepsen/danish-tokenizer |
| rows | 1,878,629 train / 98,876 eval |
| epochs | 3 (44,031 steps) |
| batch | 128 x grad-accum 1, max_length 8048 |
| optimizer | adamw_torch_fused, fp32 master weights, bf16 autocast |
| LR | 3e-5, , 500 warmup |
wandb: da_sft_v33_full_mix20_bs128_fa2_compile
Datasets
jensjepsen/danish-metamath-gsmjensjepsen/danish-algebra-sft-v5-mixedjensjepsen/danish-arith-chain-sft-v1jensjepsen/danish-wiki-grounded-sft-v3jensjepsen/danish-text-to-question-v2jensjepsen/danish-sciqjensjepsen/danish-gsm8kjensjepsen/danish-instruction-following-v4jensjepsen/danish-wiki-closedqa-v1