Base model and loading contract
Base: Qwen/Qwen3.5-9B, revision c202236235762e1c871ad0ccb60c8ee5ba337b9a.
Full-language fine-tune, not a LoRA adapter. Input protocol
raw-text-bill-source-native-structure-0.2, prompt variant
bill-indexed-text-structure-v1.
Use the exact trained system prompt and message encoding below. User input is a
sparse start:end|JSON-string lexical index of the bill's House OLM visible-text
projection; unlisted characters are ASCII spaces. Offsets are zero-based,
end-exclusive Unicode character positions. Output is
{"tree":[tag,start,end,attributes,children]}. Do not replace the trained
interface with an arbitrary plain-text prompt.
This model does not accept plain bill text. At training and evaluation time
the user message was a lossless sparse lexical index of the source bill. Passing
raw text, or applying output offsets to the indexed message, is outside the
trained contract and should not be expected to reproduce the reported accuracy.
The output is a tree whose spans index the original raw text.
System prompt
Use this frozen prompt byte-for-byte (implemented as SYSTEM in
scripts/bill_structure_input.py; shipped as PROMPT_TRAINED.txt):
Convert indexed bill text into a structure tree. Return only {"tree":[tag,start,end,attributes,children]}. Every node has that five-field shape. Spans are zero-based Unicode character offsets into the recovered raw bill text, end exclusive; they are not offsets into this indexed message. The root covers the complete recovered raw text. Children are ordered, nonoverlapping and contained within their parent. All text is copied from source spans, not rewritten. Tags name the legislative hierarchy and its headings, enumerators, and text. Attributes are a JSON object; children is an array. Input fragments are start:end|JSON-string. Unlisted character positions are ASCII spaces; all other whitespace is explicitly listed. Fill unlisted positions with spaces to recover the exact stated text length. Fragment boundaries are lexical only, not XML or provision boundaries. Return JSON only, with no explanations.
Do not add a role wrapper, a schema description, or Markdown fences.
The standalone file contains exactly 912 UTF-8 bytes, with no trailing newline.
Its raw-file SHA-256 is
aeadf0c99b2899af415893a19fa3ece882c73974dca62969df13ead5197e91fa.
Read the file without stripping whitespace or adding a newline. Download the
prompt and weights from the same pinned repository revision:
import hashlibfrom pathlib import Pathfrom huggingface_hub import HfApi, hf_hub_download repo_id = "thurstonpowers/xll-bills-qwen3.5-9b"revision = HfApi().model_info(repo_id).sha # resolve once; retain this commit for reproducibilityprompt_bytes = Path(hf_hub_download(repo_id, "PROMPT_TRAINED.txt", revision=revision)).read_bytes()assert hashlib.sha256(prompt_bytes).hexdigest() == "aeadf0c99b2899af415893a19fa3ece882c73974dca62969df13ead5197e91fa"system_prompt = prompt_bytes.decode("utf-8")# Pass revision=revision when loading this repo's tokenizer and model as well.
User message
The user content is a header line followed by one fragment per line:
Document type: bills. Text length: <N> Unicode characters.<start>:<end>|<JSON string>...
start is inclusive and end exclusive; both are zero-based Unicode character
offsets into the original raw bill text. The fragments cover every non-ASCII-space
character. Any position not listed is an ASCII space (U+0020). Non-ASCII
whitespace, including no-break space (U+00A0) and narrow no-break space
(U+202F), is listed explicitly. The | separator is followed by a JSON string
with ensure_ascii=True; decode it with a JSON parser, do not strip quotes.
import jsonimport re FRAGMENT = re.compile(r'\s+|\w+|[^\w\s]') def encode(raw, document_type='bills'): assert document_type == 'bills' fragments = [ f'{m.start()}:{m.end()}|{json.dumps(m.group(), ensure_ascii=True)}' for m in FRAGMENT.finditer(raw) if m.group().strip(' ') # omit ASCII-space-only runs only ] header = (f'Document type: {document_type}. ' f'Text length: {len(raw)} Unicode characters.') return header + '\n' + '\n'.join(fragments)
Encode the exact text you want the spans to refer to. Do not normalize, collapse,
strip, or re-wrap it first. raw is a Python str; len(raw) counts Unicode
code points. Example for the fictional text § 1. Example rule.:
Document type: bills. Text length: 18 Unicode characters.0:1|"\u00a7"2:3|"1"3:4|"."5:12|"Example"13:17|"rule"17:18|"."
Chat template and decoding
system_prompt = ... # the exact text abovemessages = [ {'role': 'system', 'content': system_prompt}, {'role': 'user', 'content': encode(raw, 'bills')},]input_ids = tokenizer.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, enable_thinking=False,)
Load the full fine-tune with Qwen/Qwen3.5-9B at revision
c202236235762e1c871ad0ccb60c8ee5ba337b9a. Use BF16, greedy decoding
(do_sample=False), and no grammar or output repair. The trained context is
32,768 tokens with no truncation; the benchmark uses per-bin completion caps of
1,536 / 2,304 / 4,608 / 6,656 / 16,384 tokens and treats cap hits as failures.
Reject inputs that do not fit; do not truncate them.
Reading the output
Parse exactly one JSON object of the form
{"tree":[tag,start,end,attributes,children]}. Spans are zero-based,
end-exclusive Unicode offsets into the original raw text, not into the
encoded user message. The root span must be [0, len(raw)]; children are
ordered, non-overlapping, and contained in their parent. Validate the tree and
retain human review. A syntactically valid JSON object is not necessarily a valid
or correct tree.
Training
- 8,335 bills (House OLM visible-text projection,
bill-indexed-text-structure-v1),
converted to indexed inputs with no truncation; longest encoded example 32,755
tokens; every example fits the 32,768-token context.
- Oversized bills were dropped and backfilled from the same provision-count bins
with bills that fit (421 training and 43 evaluation replacements), preferring
larger bills; per-bin counts preserved. Train and evaluation documents are
disjoint by ID family and exact text.
- Full language parameters trained; vision frozen. FP32 training parameters,
BF16 compute; Adafactor (factored FP32,
scale_parameter=False,
relative_step=False), learning rate 5e-6, 5%-of-updates warmup (105 steps for
the 1-epoch run, 209 for the 2-epoch run), cosine decay to 0.1 of peak;
effective batch four; seed 102301.
- Two candidates on identical data and recipe, differing only in epochs:
- 1-epoch: 2,084 optimizer updates, 45.7 h on one NVIDIA GB10.
- 2-epoch (released): 4,168 optimizer updates, 92.7 h, all 16,670 example
passes,
complete: true with reload verification and no interruptions.
- Both runs used a resumable worker with periodic FP32+optimizer+RNG checkpoints;
the retained checkpoints are exact-trajectory resumable (verified bit-exact
reload and loss continuation). Retained export is BF16 safetensors.
Evaluation — fitting-only 200-bill cohort
A new, complexity-balanced cohort of 200 held-out bills (40 per bin: shortest,
short, middle, long, longest by provision count), disjoint from training. The
reference is the House XML. Success at the document level requires a valid tree
(root span [0, len(raw)], ordered non-overlapping contained children), and
exact additionally requires the parsed tree to equal the reference tree.
Timeouts and completion-cap hits count as failures in the full 200-document
denominator. All arms ran on identical inputs, prompts, decoding (greedy,
thinking disabled, BF16), and batched protocol.
Deadline calibration (read before deploying). The trained model generates
full trees at roughly 9 tokens/second/document on a GB10. Original per-batch
deadlines of 120–420 s produced a misleading 0/200: the model was completing
trees but being cut off mid-output. Measured worst-case requirement was 3,904 s
on a 300 s base (13.0×), so the reported protocol uses 14× deadlines
(1,680–5,880 s per batch). Budget wall-clock accordingly at inference time; a
model that is "slow" here is still working.
Main comparison (14× deadlines, 200 documents)
Table with columns: Model, Valid trees, Exact trees, Node P / R / F1, Edge F1| Model | Valid trees | Exact trees | Node P / R / F1 | Edge F1 |
|---|
| Trained 1-epoch (this model) | 191/200 (95.5%) | 154/200 (77.0%) | .998 / .914 / .954 | .953 |
| GPT-6.1 Sol (OpenRouter, zero-shot) | 84/200 (42.0%) | 0/200 (0.0%) | .131 / .057 / .079 | .069 |
| Vanilla base (untrained Qwen3.5-9B) | 1/200 (0.5%) | 0/200 (0.0%) | .000 / .000 / .000 | .000 |
Per-bin exact trees (trained 1-epoch): shortest 37/40, short 34/40, middle
32/40, long 26/40, longest 25/40. Per-bin valid trees: 40/40, 40/40, 40/40,
35/40, 36/40. Three documents ended on the completion cap; all three were
emitting correct tree prefixes when the shared batch cap clipped them (two were
within ~500 and ~180 tokens of finishing — see supplement below).
Solo full-window supplement (the 3 capped documents)
Rerun individually with each document's full generation window and a 20-minute
allowment. 1-epoch: 3/3 completed with EOS, 3/3 valid trees, 2/3 exact;
generated lengths within 1-2 tokens of the reference. 2-epoch (released):
identical outcomes - 3/3 completed, 3/3 valid, the same 2/3 exact (the same
single document diverges). Counting these, the released checkpoint's ceiling on
this cohort is 197+3 = 200/200 completions and 156+2 = 158/200 (79.0%) exact.
The strict protocol numbers above stand unchanged; this supplement is reported
separately, never merged.
Epoch comparison (same cohort, same 14× protocol)
Table with columns: Model, Valid trees, Exact trees, Node P / R / F1, Edge F1| Model | Valid trees | Exact trees | Node P / R / F1 | Edge F1 |
|---|
| Trained 2-epoch (released) | 197/200 (98.5%) | 156/200 (78.0%) | .997 / .967 / .982 | .979 |
| Trained 1-epoch (ablation) | 191/200 (95.5%) | 154/200 (77.0%) | .998 / .914 / .954 | .953 |
Released (2-epoch) per-bin exact trees: shortest 36/40, short 34/40, middle
33/40, long 29/40, longest 24/40; per-bin valid trees 40/40, 40/40, 40/40,
38/40, 39/40; three documents ended on the completion cap (the same batch-min
window artifact documented below). The second epoch's gain is recall-dominant
(+5.3 points, precision unchanged): the model finds more of the tree rather
than guessing more. The largest per-bin gain is the long bin (26 → 29 exact);
shortest and longest each trade one exact match.
Task comparison and frontier caveats
These are same-cohort, frozen-prompt results, not a demonstrated best-prompt
frontier comparison. GPT-6.1 Sol received the identical indexed prompts via
OpenRouter, temperature 0, model-default reasoning, an output cap matched to the
per-bin protocol, and no output repair; total API spend $6.52. It stopped early
voluntarily (finish_reason=stop, well under cap) at roughly 54–86% of the
reference output length, produced unbalanced bracket structure on 116/200
documents, and — even where JSON parsed — used its own invented node vocabulary
rather than the contract's, which is why exact matches and F1 are near zero.
Zero exact success does not imply no useful structure or inferior general legal
ability; prompt suitability, tokenization, stopping, and runtime differences
limit the comparison. This is not an optimized frontier benchmark or a general
intelligence ranking.
Runtime and reproducibility
Evaluation used BF16 without quantization, greedy batched generation (batch 16
by token-budget), 32,768-token context, JSON-object early stopping, and 14×
per-bin deadlines on NVIDIA GB10 (Transformers 5.15.1, PyTorch 2.13.0+cu130).
The trained arm completed both benchmark arms and scoring without timeouts.
Quantized variants and other hardware are not qualified. BF16 weights are
roughly 18 GiB before activation/cache overhead. Deployment note: at ~9 tokens
per second per document, a 6,000-token tree needs ~11 minutes of wall clock;
size request deadlines accordingly.
Evidence to be packaged at publish time: cohort and prediction hashes for every
arm (SHA256SUMS in the evaluation job), training reports for both candidates,
and the deadline-calibration analysis. The original 1×-deadline run (0/200,
194 timeouts) is retained as evidence of the deadline artifact, not of model
quality.
Selection, limitations, and license
The 1-epoch and 2-epoch candidates share data, recipe, and seed; the 2-epoch
checkpoint is selected for release on the epoch comparison above (strictly
better on valid trees, exact trees, node F1, and edge F1). Development
evaluation is not an unbiased test estimate. No complete training
corpus or evaluation predictions are packaged. The Senate-side bills, amendments,
and non-OLM sources, OCR/PDF input, arbitrary state conventions, and
long-context or quantized deployment are not validated. Tags describe the
legislative hierarchy as encoded in this corpus; full USLM schema compliance and
legal accuracy are not claimed. Use official source text as authoritative;
review all model-generated structure.
Apache-2.0; the base-model license will be included as LICENSE.
Replication dataset
Training and evaluation data: thurstonpowers/xll-bills-structure-indexed.
Use the model-specific contract and splits described in the dataset card.