Kentucky-Open-Science

KOS-V5-Base

README

License: apache-2.0

At a glance

Table

Parameters	3.72B (36 layers × 2560 hidden, 32/8 GQA, head_dim 128, SwiGLU, tied embeddings)
Training tokens	235.2B (one complete epoch over a 54-source medical/biomedical corpus)
Final train loss	≈1.16
Context	trained at 24,576 tokens (whole-document); `max_position_embeddings` 32,768
RoPE	standard 1D, θ = 25,000
Tokenizer	custom 32k byte-level BPE ("v5-32k"), `add_bos_token=False`, single special token `<
Precision / stack	bf16, 24×H200, DeepSpeed ZeRO-1, FlashAttention-2
Architecture class	`Qwen3ForCausalLM` (stock, exports with no custom modelling code)

Evaluation

96 metrics across 19 tests; 94 ranked. Scoring convention used throughout:

A metric is a WIN if KOS-V5 places in the top quartile of the pool (rank 1–4 of 17), and a LOSS if it places in the bottom quartile (rank 14–17 of 17). Ranks 5–13 are mid-pool, neither.

Result: 28 wins, 16 losses, 50 mid-pool.

📖 Every test is defined in Test definitions below: what it measures, on what data, and which direction is better. The tier codes (T1, T1b, T2 …) used in the tables are all explained there.

Every comparator was trained on 1.3–153× more data than KOS-V5 (0.3–36T tokens vs our 0.235T).

The comparator pool (16 external models + KOS-V5 = 17)

Biomedical specialists (5)

Table with columns: model, tokens
model	tokens
BioMedLM	300B (PubMed)
MedGemma-4B-PT	Gemma-3-4B 4T + medical image-text
Meditron-7B	Llama-2-7B 2T + 48B medical
AdaptLLM-Medicine-LLM	LLaMA-1-7B 1T + 2B adaptation
PMC-LLaMA-7B	LLaMA-7B 1T + 79B biomedical

General-domain (11)

Table with columns: model, tokens
model	tokens
Llama-3.2-1B / Llama-3.2-3B / Llama-3.1-8B	9T / 9T / 15T
Gemma-3-1B-PT / Gemma-3-4B-PT	2T / 4T
Gemma-2-9B	8T
Qwen2.5-7B	18T
Qwen3-4B-Base	36T
StableLM-2-1.6B	2T
MiMo-7B-Base	25T
GigaChat3-10B-A1.8B-base	5.5T (synthetic-heavy)

(KOS-V4, our own previous base, is held out of the ranking pool and reported separately below.)

Wins: top quartile, ranks 1-4 (28 metrics)

Tier codes (T1, T2 …) are defined in Test definitions.

Table with columns: Metric, Value, Rank
Metric	Value	Rank
BPB, 5-corpus mean (held-out medical text)	0.4635	1 / 17
BPB radiology	0.5132	1 / 17
BPB chest-xray	0.6688	1 / 17
BPB clinical narratives	0.4179	1 / 17
Collapsed-head fraction (T2)	0.0069

Bits-per-byte is tokenizer-agnostic, so it is a fair cross-model number, and the distillation confound in the pool flatters the trillion-token externals, not a from-scratch model. These BPB placements are therefore conservative, not inflated.

Losses: bottom quartile, ranks 14-17 (16 metrics)

Tier codes are defined in Test definitions.

Table with columns: Metric, Value, Rank
Metric	Value	Rank
MedHALT false-confidence test (T8-FCT)	0.0280	16 / 17 (worst placement)
Single-token rate, aggregate (T1b-STRR)	0.0481	15 / 17
STRR, clinical / biomedical	0.1257 / 0.0525	15 / 17
STRR, radiology / chest-xray / textbooks / macro	0.1921 / 0.2274 / 0.1693 / 0.1534	14 / 17
Fragment score (T1b)	2.5855	14 / 17
RadGraph2-CXR micro-F1 (DR.3)

The losses are concentrated, not diffuse: 7 of 16 are the single-token-rate/fragmentation family (a deliberate tokenizer trade-off, see Known issues), 3 are short-context needle cells, and the rest are MedHALT-FCT, two general-domain reasoning sets, one HoC probe and one RadGraph micro-F1.

Mid-pool: neither a win nor a loss (50 metrics)

Tier codes are defined in Test definitions.

Reported because they are frequently mistaken for one or the other:

Table with columns: Metric, Value, Rank
Metric	Value	Rank
Calibration, mean ECE	0.1209	9 / 17
Long-context needle, 3-depth mean	0.9333	11 / 16
RadGraph-XL / CXR / RadGraph2 macro-F1	0.7463 / 0.7599 / 0.7661	7 / 8 / 10 of 17
MedQA / MedMCQA / PubMedQA	0.3339 / 0.3373 / 0.6680	12 / 13 / 12 of 17
BPB biomedical-lit / textbooks	0.1108 / 0.6066	13 / 11 of 17

Closed-book medical MCQ (mean 0.4352 over 16 tasks) sits mid-pool: below the frontier fleet, well above KOS-V4. It is a post-training problem, not a token-budget one.

Test definitions

What each tier actually measures. "dir" is the direction that is better. Every metric is ranked against the same 17-model pool unless marked reference-only.

Block A: general LM diagnostics (58 metrics)

T1 · Bits-per-byte on held-out medical text (the headline; dir: lower)

Negative log-likelihood divided by UTF-8 bytes, not tokens, so it is tokenizer-agnostic and directly comparable across models with different vocabularies. Measured on five held-out medical corpora, none of which appear in training: radiology_reports, chest_xray_reports, clinical_narratives, biomedical_literature, medical_textbooks, plus their mean. Caveat we state against ourselves: a model distilled from a larger teacher can print lower BPB regardless of true quality, which flatters the trillion-token externals, not a from-scratch model , so our BPB placements are conservative.

T1b · Tokenizer statistics (dir: STRR higher; fragment & tokens-per-byte lower)

STRR (Single-Token Rate Ratio): the fraction of medical terms the tokenizer encodes as one token, over a union vocabulary of 21,978 medical terms across the 5 domains. Reported per domain, plus a macro mean and a union-weighted aggregate.
fragment_score: average number of subword pieces per medical term.
tokens_per_byte: raw compression efficiency (how compactly the tokenizer encodes text).

T2 · Attention pathology (dir: lower for sinks/collapse; higher for min-entropy)

Per-(layer, head) attention diagnostics over medical text:

collapsed_head_frac: fraction of heads whose attention has degenerated to a near-constant pattern.
bos_sink_mass: attention mass dumped on the first token (the classic "attention sink").
layers_with_sink: fraction of layers exhibiting a sink.
entropy_norm_min: the lowest normalised attention entropy in the model (higher = no head has collapsed to a single position).

T3 · Representation flow (dir: lens accuracy higher; dead-neuron lower)

peak_lens / final_lens accuracy. Logit-lens probe accuracy, i.e. how well the correct token can be read out of intermediate layers (peak across depth, and at the final layer).
lens_AUC: area under the depth-vs-accuracy curve.
max_dead_neuron_frac: the worst layer's fraction of neurons that never activate.

T4 · Representation geometry (dir: RankMe higher; isotropy lower = less anisotropic)

Measured at 5 proportional depths:

RankMe: the effective rank (soft rank of the singular-value spectrum) of the hidden representations. Higher = the model uses more of its representational space rather than collapsing into a low-dimensional cone.
isotropy: directional uniformity of the embedding cloud.

T5 · WeightWatcher power-law α (structural maturity)

Fits a power law to each layer's weight spectrum. α ∈ [2, 6] is the healthy band; α > 6 means under-trained, α < 2 over-trained.

safe_layer_ratio (dir: higher): fraction of layers inside the healthy band.
under_ratio (dir: lower): fraction of layers that are under-trained.
median_α / min_α. Reference-only, deliberately unranked. α is a target band, not a monotone quantity, so "more is better" is meaningless. (Ranking it higher-is-better would literally reward being under-trained, and would have improved our reported result from rank 15 to rank 3. We declined to take the favourable reading of an ill-defined metric.)

T9 · Calibration (dir: lower)

Expected Calibration Error (ECE), averaged over 15 multiple-choice calibration tasks: the gap between the model's confidence and its actual accuracy. Low ECE = the model's confidence means something.

T10 · Context-length scaling + needle-in-a-haystack

Bucket BPB (dir: lower): BPB measured at {L/8, L/4, L/2, L} where L = min(model's max position, 4096), so every model is judged inside its own usable window. A separate KOS-only extension probes the full 32,768-token window.
Needle retrieval (dir: higher): a numeric medical fact is hidden at depth 0.1 / 0.5 / 0.9 inside a long context and must be retrieved, at fixed lengths 512 / 1,024 / 2,048 / 4,096 tokens. Reported per length, per depth, and as a 3-depth mean.

T11 · Demographic bias (dir: disparity lower; frac_stereotyped closer to 0.5)

CrowS-Pairs: for each minimally-different sentence pair (stereotyping vs anti-stereotyping), which does the model prefer? |disparity| measures how strongly it prefers one; frac_stereotyped is the fraction of pairs where the stereotyping sentence wins (0.5 = no preference).

Block B: medical knowledge and entity extraction (32 metrics)

T6 · Closed-book multiple choice (dir: higher)

16 log-likelihood MCQ tasks, scored with the standard lm-evaluation-harness (no chain-of-thought, no prompting tricks, this is a base model):

Medical specialist: MedQA (4-option, USMLE-style), MedMCQA, PubMedQA, and the 8 MMLU medical subjects (anatomy, clinical knowledge, college biology, college medicine, high-school biology, medical genetics, nutrition, professional medicine, virology).
Augmented / regional / expert: MedXpertQA-Text, AfriMed-QA.
General-reasoning controls: PIQA, HellaSwag.

T6b · MedThink-LL (dir: rank-1 higher; margin lower)

A PubMedQA-substitute: the correct answer must be ranked above 4 distractors by log-likelihood. rank1_acc = how often it is; mean_logprob_margin = the confidence gap.

T7 · BLURB linear probes (dir: higher)

A linear probe is trained on the frozen model's representations, this reads the semantic geometry of the upper layers, independent of any generation ability:

BC5CDR-Chem: chemical NER (macro / micro F1).
NCBI-Disease: disease NER (macro / micro F1).
BIOSSES: biomedical sentence-similarity (Pearson / Spearman).
HoC: Hallmarks-of-Cancer document classification (micro / macro / samples F1).
BLURB mean. The headline figure: the mean of BC5CDR-macro, NCBI-macro, BIOSSES-Spearman and HoC-micro.

T8 · Med-HALT hallucination resistance (dir: higher)

FCT (false-confidence test): the question's premise is wrong; the model should decline. A base model has no refusal prior, which is exactly why this is our worst placement.
NOTA (none-of-the-above): the correct answer is "none of the above".
FQT (fake-question test): the question is fabricated; it should be rejected.
PMID2Title: recall a paper's title from its PubMed ID (a memorisation probe; degenerate , every model in the pool scores 0.0, so it is excluded from the tally).

Block C: radiology-specific (6 metrics)

DR.1 / DR.2 / DR.3 · RadGraph entity + relation extraction (dir: higher)

Structured extraction of clinical entities and their relations from radiology reports, scored as macro-F1 and micro-F1:

DR.1, RadGraph-XL: chest CT + abdomen-pelvis CT + brain MRI.
DR.2, RadGraph-CXR: chest x-ray (50 MIMIC + 50 CheXpert).
DR.3, RadGraph2-CXR: adds the CHAN (change) relation class.

DPM.1 · PMR-Q&A (reference-only)

English physiotherapy / rehabilitation Q&A, rank-1 accuracy. Unranked: too few comparators have a score for it to be a meaningful ranking.

How ranks are computed

Pool = KOS-V5 + the 16 external models = 17. KOS-V4 is excluded from the pool, it is our own prior model, and is reported separately in its own head-to-head below.
Each metric is ranked in its own correct direction (lower-is-better for BPB, ECE, sinks, fragmentation; higher-is-better for accuracy, F1, RankMe, STRR).
Denominators vary where a comparator has no score for a metric: Gemma-2-9B has no T10 result and BioMedLM is architecturally capped at 1,024 tokens (so it is skipped in the longer needle tables). That is why some ranks read "of 16" or "of 12" rather than "of 17", the rank is always taken over whichever models actually have that metric.
Two metrics are reference-only (the WeightWatcher α pair) and one is degenerate (PMID2Title, where all 17 models score 0.0), leaving 93 metrics in the tally.

KOS-V5 vs KOS-V4: a corpus trade-off, now settled

Both models are ours, both are finished, and both were run on the identical harness. KOS-V5 is better on 17 of 32 compared metrics, KOS-V4 on 15.

The token-budget alibi is dead: KOS-V5 finished on 235.2B tokens vs KOS-V4's 180.3B (1.30×) at a larger parameter count. Every remaining KOS-V4 win is a win on the merits.

Table with columns: KOS-V5 (final, 235B), KOS-V4 (final, 180B), Better
	KOS-V5 (final, 235B)	KOS-V4 (final, 180B)	Better
Medical MCQ mean (16 tasks)	0.4352	0.3319	V5 (+0.103; 12 of 16 tasks)
MMLU clinical knowledge	0.5057	0.3094	V5 (+0.196)
MMLU college medicine	0.4335	0.2312	V5 (+0.202)
MMLU professional medicine

The honest reading. KOS-V4's narrow radiology/clinical-dominant corpus still produces the better text model of radiology and clinical notes. KOS-V5's broader 54-source corpus produces the better medical reasoner: on the recall-heavy MCQ tasks V4 sat close to chance and V5 does not. This is a corpus-composition difference, not a maturity gradient, and further pretraining was never going to close it: V5 would have needed to recover 0.0326 BPB, and its entire final leg moved it by 0.0001. Which model is "better" depends on the deployment.

The run converged before it finished. Read this before choosing a checkpoint

The identical 16-tier battery was run at six checkpoints (90B / 120B / 150B / 190B / 220B / 235B).

Table with columns: axis, 90B, 120B, 150B, 190B, 220B, 235B (final)
axis	90B	120B	150B	190B	220B	235B (final)
BPB (5-corpus, lower better)	.4890	.4781	.4740	.4647	.4636	.4635
MCQ (16 tasks, higher better)	.3478	.3737	.4071	.4449	.4332

On the final leg, not one headline axis improved. BPB moved +0.0001 (zero to three decimals). MCQ peaked at 190B and never recovered: the final model ends 0.0097 below its own peak, with 12 of 16 tasks below their peak-checkpoint value. Needle ended exactly where it began at 90B. Calibration ended worse than the checkpoint before it. The learning-rate integral is 100% spent.

⚠️ The newest checkpoint is NOT the knowledge-best checkpoint. The 190B checkpoint is KOS-V5's knowledge-best, permanently. Choose your fine-tuning initialisation by axis: the 190B checkpoint for medical knowledge/MCQ; this final checkpoint for BPB, long-context and general LM health. Do not assume "newest = best."

Both MCQ moves (the 220B drop and the 235B bounce) sit inside MMLU-subset sampling noise. What survives noise is the plateau: knowledge accuracy stopped improving 45B tokens before the end.

Known issues

1. The tokenizer does not split numbers, and this cannot be fixed in this model.

The v5-32k BBPE has no digit rule: its pre-tokenizer is plain byte-level, so digit runs merge into single tokens by BPE frequency.

Table with columns: tokenizer, digit rule, multi-digit tokens, max digit-run, 10 / 100 / 1000, tokens/number
tokenizer	digit rule	multi-digit tokens	max digit-run	`10` / `100` / `1000`	tokens/number
KOS-V5	none	963	5	one token each	1.80
Qwen3-4B-Base	yes, every digit split	0	1

The consequence is compositional: 10, 100 and 1000 are three unrelated atomic symbols that share no substructure. The model cannot see that they differ by a factor of ten; it must learn each magnitude as a separate lexical item, and rare numbers fragment on arbitrary BPE boundaries while common ones do not. In medicine, doses, lab values, vitals, this is exactly the regime where digit-level arithmetic and magnitude comparison are known to degrade.

Stated with its limit: this is a measured property of the frozen tokenizer, not a measured downstream failure. Our evaluation suite contains no arithmetic or numeric-reasoning tier, so we neither quantify the cost nor claim there is none. The tokenizer is baked into the trained (tied) embedding table, so it is not fixable by fine-tuning, removing it requires retraining with a digit-splitting pre-tokenizer. Treat any numeric medical task on this model with that in mind.

2. Low medical single-token rate (STRR: bottom quartile, rank 15/17). Deliberate: BBPE spends its 32k vocabulary on byte-level compression rather than memorising whole medical terms. It single-tokens only ~19% of radiology and ~23% of chest-xray terms. The trade appears sound, tokens-per-byte is a top-quartile 4/17 and BPB is 1/17, but it is a trade, and it is the same trade as the digit issue.

3. MedHALT-FCT (false-confidence test) is the worst placement in the model (rank 16/17). A base model with no refusal prior has essentially no mechanism for declining a question whose premise is false, and KOS-V5 answers anyway. This is the axis a grounded medical model most needs to improve, and it is a target for the instruction/RL stage rather than for more pretraining.

4. Long-context needle retrieval never improved across the entire run (mid-pool, 11/16). More pretraining will not fix it; it is a post-training problem.

5. Attention health is leading but degrading. KOS-V5 still leads the pool on T2, but bos-sink mass rose 0.1619 → 0.2541 and layers-carrying-a-sink rose 0.0830 → 0.3330 across training. The lead is real but it shrank; it did not consolidate.

Usage

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Kentucky-Open-Science/KOS-V5-Base"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")

prompt = "IMPRESSION: The chest radiograph demonstrates"
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Notes

Works with transformers 4.x and 5.x out of the box. Verified on 4.57.1 and 5.2.0: the snippet above runs unchanged on both.
Base model, no chat template. It completes text. Prompt it as a completion model, or post-train it.
One special token. <|endoftext|> (id 0) serves eos/bos/pad and the document separator.

Intended use

A research base model for medical/biomedical NLP: the initialisation for domain SFT/RL, a subject for interpretability and tokenizer/corpus research, and a from-scratch reference point against distilled and continued-pretrained medical models. It is not instruction-tuned and is not a question-answering system. Its closed-book medical MCQ accuracy sits mid-pool, and the known issues above, especially the number tokenization, bear directly on any numeric medical task.

Training details

Table

tokens / step	1 × 8 × 24 × 24,576 = 4,718,592
optimizer	AdamW β=(0.9, 0.95), grad-clip 1.0
LR schedule	cosine → 0, peak 3.0e-4, 1% warmup (integral 100% spent)
batch	micro_batch 1 × grad_accum 8 × 24 GPUs, seq 24,576 (≈4.72M tokens/step)
corpus	54 sources, English-only, deduped → decontaminated; whole documents (any document over 24,576 tokens is dropped, never split); neat-packed with a 4-D block-diagonal segment mask (no cross-document attention; position ids reset per document)
`tokenizer.json` md5	`9c9df0404f6aae96dba5f3785e8b4c9d`

Full 22-page evaluation report, all 96 metrics, all 17 models, every rank re-derived from source: kos_v5_brief.pdf, included in this repository.

Attribution

University of Kentucky

University of Louisville

Kentucky Center for Digital Innovation

Citation

bibtex
@misc{kos_v5_base_2026,
  title  = {KOS-V5-Base (Catbird): a from-scratch 3.72B medical language model},
  author = {Kentucky Open Science},
  year   = {2026},
  note   = {University of Kentucky (College of Medicine Office for Research;
            Center for Clinical and Translational Science) and
            University of Louisville (Kentucky Center for Digital Innovation)},
  url    = {https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Kentucky-Open-Science

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

At a glance

Table

Parameters	3.72B (36 layers × 2560 hidden, 32/8 GQA, head_dim 128, SwiGLU, tied embeddings)
Training tokens	235.2B (one complete epoch over a 54-source medical/biomedical corpus)
Final train loss	≈1.16
Context	trained at 24,576 tokens (whole-document); `max_position_embeddings` 32,768
RoPE	standard 1D, θ = 25,000
Tokenizer	custom 32k byte-level BPE ("v5-32k"), `add_bos_token=False`, single special token `<
Precision / stack	bf16, 24×H200, DeepSpeed ZeRO-1, FlashAttention-2
Architecture class	`Qwen3ForCausalLM` (stock, exports with no custom modelling code)

Evaluation

96 metrics across 19 tests; 94 ranked. Scoring convention used throughout:

A metric is a WIN if KOS-V5 places in the top quartile of the pool (rank 1–4 of 17), and a LOSS if it places in the bottom quartile (rank 14–17 of 17). Ranks 5–13 are mid-pool, neither.

Result: 28 wins, 16 losses, 50 mid-pool.

📖 Every test is defined in Test definitions below: what it measures, on what data, and which direction is better. The tier codes (T1, T1b, T2 …) used in the tables are all explained there.

Every comparator was trained on 1.3–153× more data than KOS-V5 (0.3–36T tokens vs our 0.235T).

The comparator pool (16 external models + KOS-V5 = 17)

Biomedical specialists (5)

Table with columns: model, tokens
model	tokens
BioMedLM	300B (PubMed)
MedGemma-4B-PT	Gemma-3-4B 4T + medical image-text
Meditron-7B	Llama-2-7B 2T + 48B medical
AdaptLLM-Medicine-LLM	LLaMA-1-7B 1T + 2B adaptation
PMC-LLaMA-7B	LLaMA-7B 1T + 79B biomedical

General-domain (11)

Table with columns: model, tokens
model	tokens
Llama-3.2-1B / Llama-3.2-3B / Llama-3.1-8B	9T / 9T / 15T
Gemma-3-1B-PT / Gemma-3-4B-PT	2T / 4T
Gemma-2-9B	8T
Qwen2.5-7B	18T
Qwen3-4B-Base	36T
StableLM-2-1.6B	2T
MiMo-7B-Base	25T
GigaChat3-10B-A1.8B-base	5.5T (synthetic-heavy)

(KOS-V4, our own previous base, is held out of the ranking pool and reported separately below.)

Wins: top quartile, ranks 1-4 (28 metrics)

Tier codes (T1, T2 …) are defined in Test definitions.

Table with columns: Metric, Value, Rank
Metric	Value	Rank
BPB, 5-corpus mean (held-out medical text)	0.4635	1 / 17
BPB radiology	0.5132	1 / 17
BPB chest-xray	0.6688	1 / 17
BPB clinical narratives	0.4179	1 / 17
Collapsed-head fraction (T2)	0.0069

Losses: bottom quartile, ranks 14-17 (16 metrics)

Tier codes are defined in Test definitions.

Table with columns: Metric, Value, Rank
Metric	Value	Rank
MedHALT false-confidence test (T8-FCT)	0.0280	16 / 17 (worst placement)
Single-token rate, aggregate (T1b-STRR)	0.0481	15 / 17
STRR, clinical / biomedical	0.1257 / 0.0525	15 / 17
STRR, radiology / chest-xray / textbooks / macro	0.1921 / 0.2274 / 0.1693 / 0.1534	14 / 17
Fragment score (T1b)	2.5855	14 / 17
RadGraph2-CXR micro-F1 (DR.3)

Mid-pool: neither a win nor a loss (50 metrics)

Tier codes are defined in Test definitions.

Reported because they are frequently mistaken for one or the other:

Table with columns: Metric, Value, Rank
Metric	Value	Rank
Calibration, mean ECE	0.1209	9 / 17
Long-context needle, 3-depth mean	0.9333	11 / 16
RadGraph-XL / CXR / RadGraph2 macro-F1	0.7463 / 0.7599 / 0.7661	7 / 8 / 10 of 17
MedQA / MedMCQA / PubMedQA	0.3339 / 0.3373 / 0.6680	12 / 13 / 12 of 17
BPB biomedical-lit / textbooks	0.1108 / 0.6066	13 / 11 of 17

Closed-book medical MCQ (mean 0.4352 over 16 tasks) sits mid-pool: below the frontier fleet, well above KOS-V4. It is a post-training problem, not a token-budget one.

Test definitions

What each tier actually measures. "dir" is the direction that is better. Every metric is ranked against the same 17-model pool unless marked reference-only.

Block A: general LM diagnostics (58 metrics)

T1 · Bits-per-byte on held-out medical text (the headline; dir: lower)

T1b · Tokenizer statistics (dir: STRR higher; fragment & tokens-per-byte lower)

STRR (Single-Token Rate Ratio): the fraction of medical terms the tokenizer encodes as one token, over a union vocabulary of 21,978 medical terms across the 5 domains. Reported per domain, plus a macro mean and a union-weighted aggregate.
fragment_score: average number of subword pieces per medical term.
tokens_per_byte: raw compression efficiency (how compactly the tokenizer encodes text).

T2 · Attention pathology (dir: lower for sinks/collapse; higher for min-entropy)

Per-(layer, head) attention diagnostics over medical text:

collapsed_head_frac: fraction of heads whose attention has degenerated to a near-constant pattern.
bos_sink_mass: attention mass dumped on the first token (the classic "attention sink").
layers_with_sink: fraction of layers exhibiting a sink.
entropy_norm_min: the lowest normalised attention entropy in the model (higher = no head has collapsed to a single position).

T3 · Representation flow (dir: lens accuracy higher; dead-neuron lower)

peak_lens / final_lens accuracy. Logit-lens probe accuracy, i.e. how well the correct token can be read out of intermediate layers (peak across depth, and at the final layer).
lens_AUC: area under the depth-vs-accuracy curve.
max_dead_neuron_frac: the worst layer's fraction of neurons that never activate.

T4 · Representation geometry (dir: RankMe higher; isotropy lower = less anisotropic)

Measured at 5 proportional depths:

RankMe: the effective rank (soft rank of the singular-value spectrum) of the hidden representations. Higher = the model uses more of its representational space rather than collapsing into a low-dimensional cone.
isotropy: directional uniformity of the embedding cloud.

T5 · WeightWatcher power-law α (structural maturity)

Fits a power law to each layer's weight spectrum. α ∈ [2, 6] is the healthy band; α > 6 means under-trained, α < 2 over-trained.

safe_layer_ratio (dir: higher): fraction of layers inside the healthy band.
under_ratio (dir: lower): fraction of layers that are under-trained.
median_α / min_α. Reference-only, deliberately unranked. α is a target band, not a monotone quantity, so "more is better" is meaningless. (Ranking it higher-is-better would literally reward being under-trained, and would have improved our reported result from rank 15 to rank 3. We declined to take the favourable reading of an ill-defined metric.)

T9 · Calibration (dir: lower)

T10 · Context-length scaling + needle-in-a-haystack

Bucket BPB (dir: lower): BPB measured at {L/8, L/4, L/2, L} where L = min(model's max position, 4096), so every model is judged inside its own usable window. A separate KOS-only extension probes the full 32,768-token window.
Needle retrieval (dir: higher): a numeric medical fact is hidden at depth 0.1 / 0.5 / 0.9 inside a long context and must be retrieved, at fixed lengths 512 / 1,024 / 2,048 / 4,096 tokens. Reported per length, per depth, and as a 3-depth mean.

T11 · Demographic bias (dir: disparity lower; frac_stereotyped closer to 0.5)

Block B: medical knowledge and entity extraction (32 metrics)

T6 · Closed-book multiple choice (dir: higher)

16 log-likelihood MCQ tasks, scored with the standard lm-evaluation-harness (no chain-of-thought, no prompting tricks, this is a base model):

Medical specialist: MedQA (4-option, USMLE-style), MedMCQA, PubMedQA, and the 8 MMLU medical subjects (anatomy, clinical knowledge, college biology, college medicine, high-school biology, medical genetics, nutrition, professional medicine, virology).
Augmented / regional / expert: MedXpertQA-Text, AfriMed-QA.
General-reasoning controls: PIQA, HellaSwag.

T6b · MedThink-LL (dir: rank-1 higher; margin lower)

A PubMedQA-substitute: the correct answer must be ranked above 4 distractors by log-likelihood. rank1_acc = how often it is; mean_logprob_margin = the confidence gap.

T7 · BLURB linear probes (dir: higher)

A linear probe is trained on the frozen model's representations, this reads the semantic geometry of the upper layers, independent of any generation ability:

BC5CDR-Chem: chemical NER (macro / micro F1).
NCBI-Disease: disease NER (macro / micro F1).
BIOSSES: biomedical sentence-similarity (Pearson / Spearman).
HoC: Hallmarks-of-Cancer document classification (micro / macro / samples F1).
BLURB mean. The headline figure: the mean of BC5CDR-macro, NCBI-macro, BIOSSES-Spearman and HoC-micro.

T8 · Med-HALT hallucination resistance (dir: higher)

FCT (false-confidence test): the question's premise is wrong; the model should decline. A base model has no refusal prior, which is exactly why this is our worst placement.
NOTA (none-of-the-above): the correct answer is "none of the above".
FQT (fake-question test): the question is fabricated; it should be rejected.
PMID2Title: recall a paper's title from its PubMed ID (a memorisation probe; degenerate , every model in the pool scores 0.0, so it is excluded from the tally).

Block C: radiology-specific (6 metrics)

DR.1 / DR.2 / DR.3 · RadGraph entity + relation extraction (dir: higher)

Structured extraction of clinical entities and their relations from radiology reports, scored as macro-F1 and micro-F1:

DR.1, RadGraph-XL: chest CT + abdomen-pelvis CT + brain MRI.
DR.2, RadGraph-CXR: chest x-ray (50 MIMIC + 50 CheXpert).
DR.3, RadGraph2-CXR: adds the CHAN (change) relation class.

DPM.1 · PMR-Q&A (reference-only)

English physiotherapy / rehabilitation Q&A, rank-1 accuracy. Unranked: too few comparators have a score for it to be a meaningful ranking.

How ranks are computed

Pool = KOS-V5 + the 16 external models = 17. KOS-V4 is excluded from the pool, it is our own prior model, and is reported separately in its own head-to-head below.
Each metric is ranked in its own correct direction (lower-is-better for BPB, ECE, sinks, fragmentation; higher-is-better for accuracy, F1, RankMe, STRR).
Denominators vary where a comparator has no score for a metric: Gemma-2-9B has no T10 result and BioMedLM is architecturally capped at 1,024 tokens (so it is skipped in the longer needle tables). That is why some ranks read "of 16" or "of 12" rather than "of 17", the rank is always taken over whichever models actually have that metric.
Two metrics are reference-only (the WeightWatcher α pair) and one is degenerate (PMID2Title, where all 17 models score 0.0), leaving 93 metrics in the tally.

KOS-V5 vs KOS-V4: a corpus trade-off, now settled

Both models are ours, both are finished, and both were run on the identical harness. KOS-V5 is better on 17 of 32 compared metrics, KOS-V4 on 15.

The token-budget alibi is dead: KOS-V5 finished on 235.2B tokens vs KOS-V4's 180.3B (1.30×) at a larger parameter count. Every remaining KOS-V4 win is a win on the merits.

Table with columns: KOS-V5 (final, 235B), KOS-V4 (final, 180B), Better
	KOS-V5 (final, 235B)	KOS-V4 (final, 180B)	Better
Medical MCQ mean (16 tasks)	0.4352	0.3319	V5 (+0.103; 12 of 16 tasks)
MMLU clinical knowledge	0.5057	0.3094	V5 (+0.196)
MMLU college medicine	0.4335	0.2312	V5 (+0.202)
MMLU professional medicine

The run converged before it finished. Read this before choosing a checkpoint

The identical 16-tier battery was run at six checkpoints (90B / 120B / 150B / 190B / 220B / 235B).

Table with columns: axis, 90B, 120B, 150B, 190B, 220B, 235B (final)
axis	90B	120B	150B	190B	220B	235B (final)
BPB (5-corpus, lower better)	.4890	.4781	.4740	.4647	.4636	.4635
MCQ (16 tasks, higher better)	.3478	.3737	.4071	.4449	.4332

⚠️ The newest checkpoint is NOT the knowledge-best checkpoint. The 190B checkpoint is KOS-V5's knowledge-best, permanently. Choose your fine-tuning initialisation by axis: the 190B checkpoint for medical knowledge/MCQ; this final checkpoint for BPB, long-context and general LM health. Do not assume "newest = best."

Both MCQ moves (the 220B drop and the 235B bounce) sit inside MMLU-subset sampling noise. What survives noise is the plateau: knowledge accuracy stopped improving 45B tokens before the end.

Known issues

1. The tokenizer does not split numbers, and this cannot be fixed in this model.

The v5-32k BBPE has no digit rule: its pre-tokenizer is plain byte-level, so digit runs merge into single tokens by BPE frequency.

Table with columns: tokenizer, digit rule, multi-digit tokens, max digit-run, 10 / 100 / 1000, tokens/number
tokenizer	digit rule	multi-digit tokens	max digit-run	`10` / `100` / `1000`	tokens/number
KOS-V5	none	963	5	one token each	1.80
Qwen3-4B-Base	yes, every digit split	0	1

4. Long-context needle retrieval never improved across the entire run (mid-pool, 11/16). More pretraining will not fix it; it is a post-training problem.

Usage

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Kentucky-Open-Science/KOS-V5-Base"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")

prompt = "IMPRESSION: The chest radiograph demonstrates"
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Notes

Works with transformers 4.x and 5.x out of the box. Verified on 4.57.1 and 5.2.0: the snippet above runs unchanged on both.
Base model, no chat template. It completes text. Prompt it as a completion model, or post-train it.
One special token. <|endoftext|> (id 0) serves eos/bos/pad and the document separator.

Intended use

Training details

Table

tokens / step	1 × 8 × 24 × 24,576 = 4,718,592
optimizer	AdamW β=(0.9, 0.95), grad-clip 1.0
LR schedule	cosine → 0, peak 3.0e-4, 1% warmup (integral 100% spent)
batch	micro_batch 1 × grad_accum 8 × 24 GPUs, seq 24,576 (≈4.72M tokens/step)
corpus	54 sources, English-only, deduped → decontaminated; whole documents (any document over 24,576 tokens is dropped, never split); neat-packed with a 4-D block-diagonal segment mask (no cross-document attention; position ids reset per document)
`tokenizer.json` md5	`9c9df0404f6aae96dba5f3785e8b4c9d`

Full 22-page evaluation report, all 96 metrics, all 17 models, every rank re-derived from source: kos_v5_brief.pdf, included in this repository.

Attribution

University of Kentucky

University of Louisville

Kentucky Center for Digital Innovation

Citation

bibtex
@misc{kos_v5_base_2026,
  title  = {KOS-V5-Base (Catbird): a from-scratch 3.72B medical language model},
  author = {Kentucky Open Science},
  year   = {2026},
  note   = {University of Kentucky (College of Medicine Office for Research;
            Center for Clinical and Translational Science) and
            University of Louisville (Kentucky Center for Digital Innovation)},
  url    = {https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base}
}