Model summary
Table | |
|---|
| Developed by | Ashish Ranjan |
| Model type | LoRA adapter for a vision-language model (image + text in, text out) |
| Fine-tuned from | Qwen/Qwen3.5-9B (9.45 B parameters) |
| Task | Page-by-page extraction of a document's nested structure into JSON |
| Input | One page image (rendered at 200 DPI) + a short text describing where the previous page left off ("open path") + the extraction system prompt |
| Output | One JSON object per page (typically ~1,200 tokens, at most 4,096). The included code merges the pages into one tree per document |
| Language | English |
| Adapter | rank 16, 43.3 M trainable parameters (0.46%), 173 MB |
| License | Apache-2.0 |
| GPU memory | about 21.5 GB with the default settings: works on 48 GB and 80 GB GPUs, not on a 24 GB L4 (details) |
| Release | v1.1, 2026-10-18 (parsing fix, same weights; see Changelog). First release v1.0, 2026-10-16. A static model trained on an offline dataset |
Highlights
- Structure: node F1 of 0.89, against 0.36 for the original model, on two insurance policies held out from training (34 pages).
- Nesting: 87% of nodes sit under the right parent, against 59% for the original model, and 95% have the right type.
- Verbatim text: 99.8% whole-document word similarity. Only 4 of the 1,301 reference nodes are missing.
- Beyond the training domain: on six unlabelled documents it had never seen (88 pages, including commercial forms, a card agreement and a federal regulation), it captured 98% of the printed words, against 43% for the original model.
- Behaves normally without the extraction prompt: no ordinary question was answered with JSON, visual-QA accuracy is 32/40 against 34/40, and thinking mode is intact.
- Fast and cheap: 8.6 s per page on one H100. One training run took 33 minutes; the whole project cost an estimated $12 of cloud GPU time.
Contents
How it works · Getting started · Evaluation · Uses · Bias, risks and limitations · Training details · Technical specifications · Environmental impact · Troubleshooting · Files · License and data provenance · Citation · Changelog · Glossary
How it works
A policy is too long for one generation, and its nesting runs across pages. The model therefore reads one page per request. Each request also carries a short description of where the previous page left off, called the open path.
page 1 ─► model ─► page tree ─► merge ─► document tree ─► open path ─┐ │page 2 + open path ◄─────────────────────────────────────────────────┘ ─► model ─► page tree ─► merge ─► …
- The open path is the chain of nodes still open at the end of the previous page, from the document root down through the last child at each level. Each entry holds the node's type and its label, or its title, or the first words of its text. The deepest entry also gets a
tail (its last 150 characters), so the model can tell whether the new page continues that sentence.
- The page tree. The model writes only what is printed on the current page. A node that continues from an earlier page is written as a stub along the left edge, for example
{"cont": true, "type": "clause", "label": "12."}. A stub never repeats earlier text. If the page starts mid-sentence, the deepest stub's text holds just the continuation.
- Merging is done by deterministic Python code, not by the model (
page_merge.py):
- Each stub is matched to the open node at the same level.
- Continuation text is appended to that node, and new nodes are attached after it.
- The next page's open path is then read off the merged tree.
Example: held-out GEICO auto policy, page 5
Page 5 of GEICO's Maine Family Automobile Insurance Policy (form A-30ME), a publicly filed policy form, shown here for illustration only (not affiliated with or endorsed by GEICO). This policy was held out from training.
The open path given to the model was built from its own output for pages 3–4:
[{"type": "document"}, {"type": "section", "title": "Section I — Liability Coverages"}, {"type": "subsection", "title": "Exclusions — When Section I Does Not Apply"}, {"type": "clause", "label": "12.", "tail": "of a non-owned auto or temporary substitute auto that is designed for use principally off public roads that is not registered for use on public roads."}]
The start of the model's output, after the bracket repair:
{"cont": true, "type": "document", "children": [ {"cont": true, "type": "section", "title": "Section I — Liability Coverages", "children": [ {"cont": true, "type": "subsection", "title": "Exclusions — When Section I Does Not Apply", "children": [ {"type": "clause", "label": "13.", "text": "We do not cover bodily injury or property damage that results from nuclear exposure or explosion including resulting fire, radiation or contamination is not covered."}, {"type": "clause", "label": "14.", "text": "We do not cover bodily injury or property damage that results from bio-chemical attack or exposure to bio-chemical agents is not covered."}, {"type": "clause", "label": "15.", "text": "Section I does not apply:", "children": [ {"type": "item", "label": "(a)", "text": "To bodily injury or property damage caused by an auto driven in or preparing for any racing, speed or demolition contest or stunting activity of any nature, whether or not prearranged or organized."}, {"type": "item", "label": "(b)", "text": "To the operation or use of a motor vehicle on a track designed primarily for racing or high speed driving. ..."}]}, …
How to read this:
- The page continues the open Exclusions subsection, so it starts with three stubs and then adds exclusions 13 onwards.
- The repeated "…is not covered." in exclusions 13 and 14 is printed that way in the policy; the model copies text exactly as printed.
- The raw output had a closing-bracket miscount, which
repair_json fixed. The raw and parsed outputs are both in examples/geico_p5.json.
Getting started
Requirements
- Hardware: a Linux machine with an NVIDIA GPU with more than 24 GB of memory. With the default settings the model needs about 21.5 GB: 17.8 GB of weights once loaded, plus vLLM's working memory and the cache for one page.
- Software: vLLM 0.30.0, pypdfium2 and Pillow (see
requirements.txt).
- Tested end to end from this repository (2026-10-16, fresh install on one H100): download, then
extract.py on pages 3–8 of the held-out GEICO policy.
- All 6 pages attached, and 4 of the 6 outputs were byte-identical to the evaluation run.
- Node F1 on those pages was 0.868, against 0.861 in the evaluation run.
- Greedy decoding on GPUs isn't bit-exact across library versions and batch sizes, so expect small differences like these.
- v1.1 changed only the parsing code. Replaying the saved model outputs through the v1.1
extract.py reproduces the re-scored trees exactly for all 8 test documents.
Which GPU? Measured 2026-10-16 with the published extract.py on held-out GEICO pages:
Table with columns: GPU, Usable memory, Result, Speed, one document, GPU cost per page*| GPU | Usable memory | Result | Speed, one document | GPU cost per page* |
|---|
| NVIDIA H100 | 80 GB | ✅ works | 8.6 s/page | ≈ $0.009 |
| NVIDIA L40S | 44 GB | ✅ works | 32 s/page | ≈ $0.018 |
| NVIDIA L4 | 22 GB | ❌ out of memory at start-up, also at --gpu-memory-utilization 0.95 | – |
*At Modal's on-demand prices. The H100 is the cheapest per page because it generates almost 4× faster.
Other GPUs (calculated, not tested): any card with 32 GB or more should work. 24 GB consumer cards (RTX 3090 / 4090) have slightly more usable memory than the L4, so they're borderline. 16 GB cards can't hold the weights.
Run on a PDF
pip install -U "huggingface_hub[cli]"hf download Ashish-Ranjan/Qwen3.5-9B-Policy-Structure-LoRA --local-dir policy-structurecd policy-structurepip install -r requirements.txtpython extract.py my_policy.pdf --out-dir out
What it does:
- Renders each page at 200 DPI, the resolution the model was trained on.
- Runs the page loop.
- Writes
out/my_policy.json, the merged document tree.
- Writes
out/my_policy.pages.json, which holds for every page the raw model output, whether its brackets were repaired, merge repairs, output tokens and timing.
Options:
--pages 3-18 processes only some pages.- Passing several PDFs batches them together.
--adapter <folder> loads the adapter from a local copy.
Use from Python
from extract import render_pdf, extract, vllm_generator generate = vllm_generator("Ashish-Ranjan/Qwen3.5-9B-Policy-Structure-LoRA") # loads Qwen3.5-9B + this adapter in vLLMdocs = {"policy": render_pdf("policy.pdf", pages=range(3, 19))}result = extract(docs, generate) # pages in order; documents batched togethertree = result["policy"]["tree"]
Everything the evaluation used is in this repo.
Table with columns: Part, Content| Part | Content |
|---|
| System message | system_prompt.txt, verbatim (599 tokens). It switches on extraction mode; without it the model behaves like the original. |
| User message, first page | the page image, then Open path: [] (this is the first page)\n\nExtract this page. |
| User message, later pages | the page image, then Open path at the end of the previous page:\n{path}\n\nExtract this page., where {path} is page_merge.path_text(open_path(tree)) |
| Thinking | : |
Keep the bracket repair. With free decoding the model sometimes miscounts closing brackets at the end of deeply nested pages (9 of 16 GEICO pages and 7 of 18 Encompass pages needed it).
Every node is {type, label, title, text, children}, and empty fields are left out. There are no ids; assign them downstream if you need them.
Table with columns: Field, Meaning| Field | Meaning |
|---|
type | document, part, section, subsection, clause, definition, subclause, item, paragraph |
label | the enumerator exactly as printed: "A.", , , |
Conventions the model learned:
- Headings of equal visual weight are siblings.
- A list is nested under the heading or lead-in sentence (ending in ":") that introduces it.
- The first enumerated level under a section or subsection is a
clause; deeper levels are items.
- Page furniture is left out: running headers and footers, page numbers, form numbers, copyright lines, insurer address blocks and endorsement banners.
Evaluation
Testing data, factors and metrics
Testing data:
Table with columns: Set, What, How it's scored| Set | What | How it's scored |
|---|
| Held-out documents | 2 complete insurance policies never seen in training: GEICO Family Auto (single column, pages 3–18) and Encompass Special Home (two columns, pages 3–20). 34 pages, 1,301 reference nodes | Against hand-authored reference trees |
| Unseen documents | 6 more documents, every page including covers and contents pages (88 pages): Liberty Mutual homeowners and auto, ISO commercial general liability and commercial property forms, a Capital One card agreement, and Regulation E (12 CFR 1005) | Label-free checks against each page's PDF text layer (no reference trees exist) |
| Behaviour checks | 18 ordinary questions about held-out policy pages, 40 held-out visual-QA questions from the_cauldron, 10 thinking-mode text prompts | Against the original model |
Factors:
- Layout: one, two or three columns.
- Document type: personal-lines insurance, commercial insurance, a card agreement, a federal regulation.
- Page kind: body pages, covers, contents pages.
- Decoding and merge settings, for the ablation.
Metrics:
Table with columns: Metric, What it measures| Metric | What it measures |
|---|
| Node precision / recall / F1 | Are the document's units found, as separate nodes? Nodes are matched in order by label plus the opening of their title and text. |
| Right parent (relaxed, strict) | Is each matched node nested under the right parent? |
| Node type accuracy | Section, clause, item and so on. |
| Text exact, word similarity | Is the text verbatim? |
| Pages attached | Did every page's output parse and join the tree? |
| Words captured, words not on page | Label-free completeness and invented text, against the PDF text layer with headers and footers removed. |
All runs went page by page, exactly as in How it works. Each page's prompt carried the open path built from the model's own earlier pages, never from the reference.
Results: held-out documents
Table with columns: Metric, GEICO original, GEICO fine-tuned, Encompass original, Encompass fine-tuned| Metric | GEICO original | GEICO fine-tuned | Encompass original | Encompass fine-tuned |
|---|
| Pages attached | 16/16 | 16/16 | 18/18 | 18/18 |
| Nodes (reference / predicted) | 585 / 220 | 585 / 614 | 716 / 343 | 716 / 723 |
| Precision | 0.650 | 0.845 | 0.551 |
- Original model: the same prompts with JSON-constrained decoding, its best setting. With free decoding it scores lower (F1 0.22 / 0.36), because most of its pages don't parse.
- Fine-tuned model: the released pipeline: free decoding, then bracket repair, then the tolerant merge.
- Right parent, strict: the matched node's parent is the matched counterpart of its reference parent. This check cascades: if a parent is written slightly differently (for example
"4" instead of "4."), all of its children count as wrong.
- Right parent, relaxed: the parent is the same node, ignoring label punctuation, or has the same heading or text start. Only 1.5% (GEICO) and 4% (Encompass) of nodes are genuinely under the wrong node.
- Unmatched reference nodes (66 GEICO, 54 Encompass): almost all are present but merged into a neighbouring node's text, or split differently. Only 4 of 1,301 reference nodes are missing outright.
Results: unseen documents
On GEICO, where the right answer is known, the reference tree scores 99.8% captured / 0.1% not on page, and the fine-tuned model 99.7% / 0.1%. So these two checks track quality closely.
Table with columns: Document (not in training), Pages attached, fine-tuned vs original, Words captured, Words not on page| Document (not in training) | Pages attached, fine-tuned vs original | Words captured | Words not on page |
|---|
| Liberty Mutual homeowners (three columns) | 18/18 vs 13/18 | 99.7% vs 46% | 2.3% vs 6.5% |
| Liberty Mutual auto | 12/13 vs 5/13 | 95.7% vs 27% | 0.3% vs 7.0% |
| ISO commercial general liability, CG 00 01 (watermarked) | 20/20 vs 4/20 | 96.4% vs 18% | 1.8% vs 4.2% |
| ISO commercial property, CP 00 17 + a second form |
What we found:
- Text capture generalises. 79 of 88 pages scored ≥ 95%, across a three-column layout, a watermark, the small print of a federal regulation and a pricing table.
- Nesting is the weak spot on unfamiliar layouts. On some pages the model closes one bracket level too few in the middle of the page. Every later node on that page then lands one level too deep, for example Part B nested inside Part A. It happened most on Regulation E (9 of 15 pages) and the commercial general liability form (8 of 20).
- Contents pages: one was lost, because the model copied the dot leaders ("Agreement . . . .") until it hit the 4,096-token cap. Another came out as empty headings.
- Regulation-style run-in labels such as "(b) Electronic fund transfer—(1) Definition." are nested incorrectly.
- A second form inside the same PDF is attached beside the first form's sections instead of under a new node.
The live demo shows these outputs page by page under "More documents"; per-document numbers are in results/unseen_docs.csv.
Ablation: what each part of the pipeline adds
Table with columns: Decoding and merge (fine-tuned adapter), GEICO F1, Encompass F1, Pages attached| Decoding and merge (fine-tuned adapter) | GEICO F1 | Encompass F1 | Pages attached |
|---|
| Free decoding, strict merge | 0.761 | 0.808 | 28 / 34 |
| JSON-constrained decoding, strict merge | 0.866 | 0.818 | 33 / 34 |
| Free decoding + bracket repair + lenient merge (v1.0) | 0.866 | 0.921 | 34 / 34 |
| The same + duplicate-key-safe parsing (released, v1.1) | |
Constrained decoding guarantees valid JSON, but on dense two-column pages it stops at a premature close and loses the right-hand column. The repair-based pipeline keeps that column.
Table with columns: Check, Original, Fine-tuned| Check | Original | Fine-tuned |
|---|
| Ordinary questions about held-out policy pages answered with extraction JSON (should be 0) | 0 / 18 | 0 / 18 |
| Held-out visual QA from the_cauldron, relaxed match | 34 / 40 | 32 / 40 |
| Thinking-mode prompts that still produce reasoning | 10 / 10 | 10 / 10 |
Learning-rate sweep and checkpoint selection
Table with columns: Learning rate, Validation loss, epoch 1 / 2 / 3, Full-document F1 at epoch 3 (free decoding, GEICO / Encompass)| Learning rate | Validation loss, epoch 1 / 2 / 3 | Full-document F1 at epoch 3 (free decoding, GEICO / Encompass) |
|---|
| 5e-5 | 0.0171 / 0.0137 / 0.0132 | 0.546 / 0.360 |
| 1e-4 | 0.0150 / 0.0121 / 0.0116 | 0.685 / 0.752 |
| 2e-4 (released, epoch 3) | 0.0141 / 0.0117 / 0.0116 | 0.761 / 0.808 |
- Validation loss is teacher-forced on the 16 GEICO pages. It was 0.0623 for the original model.
- F1 in this table is plain free decoding before bracket repair. It is dominated by pages lost to bracket errors, which the released pipeline recovers.
- Selection: lowest validation loss together with the best full-document F1. Because the checkpoint was chosen on the same two held-out documents, their scores are slightly optimistic; the six unseen documents were not used for any choice.
Uses
Direct use
Turning born-digital, English insurance policies and similarly nested contracts into one JSON tree per document, with extract.py or your own loop (see Prompt format).
Downstream use
- Structure-aware chunking for retrieval (RAG): one chunk per section or clause, with its heading path.
- Clause lookup and side-by-side comparison of policies.
- A starting point for fine-tuning on other nested document types with the same page-by-page format.
Out-of-scope use
- Automated coverage, claims, underwriting or legal decisions without human review.
- Legal or insurance advice.
- Scanned, handwritten or non-English documents, and table-heavy documents: grids are not supported, and the schema has no table type.
- General chat or image questions. These still work, but use the base model for them.
Bias, risks and limitations
- Small reference-scored test. Structure is scored against reference trees on only two documents (34 pages), both US insurance policies. The six unseen documents (88 pages) were checked with label-free measures only.
- Insurance-centred training. Training used 15 documents, mostly US personal-lines insurance plus one credit-card agreement. Layouts and drafting styles from insurers or document types outside that set nest less reliably. Flat or lightly structured documents may come out over-nested.
- Pages the model hasn't seen. Cover pages, contents pages, declarations pages and data tables were not in training. Contents pages with dot leaders can make the model copy the leaders until the token cap, losing the page. Key-value tables are flattened into items.
- Granularity and nesting. The model sometimes:
- merges two paragraphs into one node;
- folds several heading levels into one title;
- attaches a lead-in sentence at the wrong level;
- drops a label's punctuation (
"4" vs "4."), because the training labels were inconsistent about this.
- Brackets.
repair_json fixes missing or extra closers at the end of a page. It can't fix a pair missing in the middle of a page, which pushes the rest of that page one level too deep.
Recommendations
- Always parse with
parse_page and merge with merge_lenient (extract.py does). Together they repair miscounted brackets, never drop a key the model wrote twice, and never drop a page.
- Skip contents pages, or stop generation on a long run of
. . ., and split PDFs that bundle several forms into one document per form.
- Check outputs cheaply without labels:
- compare the output's words with the PDF text layer (the "words captured" check);
- flag numbering gaps or restarts among siblings ("1, 2, 4" or a list starting at 3), which reliably point to nesting slips.
- Before relying on it for a new document type, score a few documents against hand-checked trees.
- Keep a human in the loop for any decision that depends on the extracted structure.
Training details
Training data
Table with columns: Part, Conversations, What| Part | Conversations | What |
|---|
| Extraction | 265 (55%) | One page image + open path → that page's JSON, from 15 documents (below) |
| Contrast | 71 (15%) | Pages from the same documents with ordinary prompts (summarise, answer a question, list the headings, transcribe, generic JSON request) → normal answers, so JSON comes out only when the extraction prompt is used |
| Retention, images | 99 (20%) | General image questions (natural images, charts, documents, infographics, OCR, diagrams, screens) sampled from the_cauldron |
| Retention, text | 49 (10%) | English chat prompts from oasst2 |
Training documents: 15 policy forms and agreements: forms from US insurers (plus one Canadian policy), some of them ISO standard forms, and one bank credit-card agreement. Declarations pages and other policyholder-specific pages were excluded. The pages, targets, held-out reference trees and the exact training mix are available for non-commercial research in the gated dataset Policy-Structure-Pages. There, training documents are identified by kind only.
Table with columns: Kind, Documents, Pages| Kind | Documents | Pages |
|---|
| Homeowners | 6 | 131 |
| Personal auto | 5 | 39 |
| Renters, with endorsements | 1 | 39 |
| Dwelling fire | 1 | 27 |
| Manufactured home | 1 | 22 |
| Credit-card agreement | 1 | 7 |
13 of the 15 documents are two-column layouts. Drafting styles range from ISO standard forms to insurers' own plain-language and narrative styles. Enumeration styles include A. / 1. / a. / (1) / (a) / 1) / a).
- Data collection method: hybrid. The extraction data comes from public policy documents, mostly from a US state insurance regulator's policy-comparison filings, and one bank's published agreement. The retention prompts and images come from public datasets.
- Labelling method:
- Extraction targets: human. The author wrote a reference tree for each document from its page images, using the PDF text layer for exact wording.
- Contrast and retention answers: synthetic. The original Qwen3.5-9B generated them itself (self-distillation), about 75% with thinking on, so the adapter doesn't unlearn reasoning.
Training procedure
Preprocessing:
- Each reference tree was aligned to the page breaks and sliced into per-page targets with continuation stubs. Merging the slices reproduces every tree exactly.
- Node types were normalised with one rule across documents.
- Five near-duplicate pages were dropped (4-gram Jaccard ≥ 0.7).
- Pages were rendered at 200 DPI. A page image is 3,657 tokens and is never downscaled.
Hyperparameters (ms-swift 4.5.3; full arguments in training/swift_args.json):
Table with columns: Setting, Value| Setting | Value |
|---|
| Method | LoRA on the language model only; vision encoder and projector frozen |
| Target modules | all linear layers: attention q/k/v/o, Gated DeltaNet in_proj_qkv/z/b/a + out_proj, MLP gate/up/down |
| Rank / alpha / dropout | 16 / 32 / 0.05 |
| Learning rate | 2e-4, cosine schedule, 5% warmup (swept over 5e-5, 1e-4, 2e-4) |
| Optimizer | AdamW (fused), betas 0.9 / 0.95, weight decay 0.1, gradient clip 1.0 |
| Batch | 1 per device × 8 gradient accumulation = 8 |
| Epochs / steps | 3 / 183 |
Speeds, sizes, times
Table | |
|---|
| Training time | 33 minutes per run on one H100 (183 steps); the three-run sweep ran in parallel |
| Peak training memory | 68.5 GiB |
| Adapter size | 173 MB (43.3 M parameters) |
| Sequence lengths | extraction targets median 1,073 tokens (max 2,316); extraction conversations median 5,455; longest conversation 8,058 |
| Inference, one document | 8.6 s per page on one H100 (≈ 1,170 output tokens per page at 108 tokens/s); 32 s per page on one L40S (≈ 39 tokens/s) |
| Inference, batched | ≈ 2.6 s per page of throughput with several documents in flight |
| Model load | ≈ 3.6 minutes on a fresh machine |
The training log is in training/train_log.jsonl.
Technical specifications
Model architecture and objective
- Base model: Qwen3.5-9B, a natively multimodal model. Its language model mixes Gated DeltaNet (linear attention) layers with full-attention layers and supports a 262K-token context.
- What the adapter changes: low-rank updates to every linear layer of the language model. The vision encoder and projector are unchanged.
- Objective: supervised fine-tuning with next-token cross-entropy on the assistant's tokens.
Compute infrastructure
- Hardware: NVIDIA H100 80 GB on Modal, one GPU per training run. Self-distillation answers were generated on an L40S.
- Software:
- training: ms-swift 4.5.3, transformers 5.16.1, peft 0.20.0, flash-linear-attention 0.5.2;
- inference and evaluation: vLLM 0.30.0.
Environmental impact
Estimated with the Machine Learning CO₂ Impact method (Lacoste et al., 2019): GPU hours × power draw × grid carbon intensity.
Table | |
|---|
| Hardware type | NVIDIA H100 80 GB (plus an L40S for data generation) |
| Hours used | ≈ 1.7 GPU-hours for the released learning-rate sweep (3 runs × 33 minutes). ≈ 3.5 GPU-hours for the whole project, including data generation and every evaluation (estimated from the cloud bill) |
| Cloud provider | Modal |
| Compute region | not recorded |
| Carbon emitted | ≈ 0.5 kg CO₂eq for the sweep, ≈ 1 kg CO₂eq for the whole project (assuming 0.7 kW per GPU and 0.4 kg CO₂eq/kWh) |
Troubleshooting
- vLLM fails while compiling a FlashInfer sampler (no
nvcc on the machine): set VLLM_USE_FLASHINFER_SAMPLER=0. The evaluation ran this way.
- Out of memory while vLLM starts: the GPU is too small. The model needs about 21.5 GB, so a 24 GB L4 fails; use a 32 GB or larger card. On a borderline card,
--gpu-memory-utilization 0.95 gives vLLM a little more room. max_model_len is 12,288 tokens: one page image plus prompt plus up to 4,096 output tokens.
- A page is missing from the tree: look at its entry in
<name>.pages.json. An error means the output couldn't be parsed even after repair. output_tokens = 4096 means generation hit the cap, usually on a contents page. A non-empty duplicate_keys means the model wrote a field twice on one node; both values were kept (see Changelog).
- Output is prose instead of JSON: make sure the system message is exactly , the adapter is loaded and thinking is off.
Files in this repository
Table with columns: File, What| File | What |
|---|
adapter_model.safetensors, adapter_config.json | The LoRA adapter (PEFT format) |
extract.py | PDF → nested JSON with vLLM: renders pages, runs the page loop, writes the tree plus per-page logs |
page_merge.py | parse_page, repair_json, keep_duplicates, merge_lenient, strip_cont, open_path, : the same reassembly code as the training and evaluation pipeline (standard library only) |
License and data provenance
- Licence: the adapter weights and the code in this repository are released under Apache-2.0, the same licence as the base model, Qwen3.5-9B.
- Training documents: insurance policy forms (most from a US state insurance regulator's public policy-comparison filings) and a public card agreement. Their copyrights stay with their owners, and some include ISO copyrighted material. This model repository doesn't contain the documents. The training and evaluation data is available separately in the gated, research-only dataset Policy-Structure-Pages. The single example page above is shown for illustration.
- Trademarks and affiliation: this project is not affiliated with or endorsed by any insurer or other company named in this card. Names are used only to identify the publicly filed documents the model was tested on.
- Retention data: prompts and images come from HuggingFaceM4/the_cauldron, whose subsets carry their own licences, and from OpenAssistant/oasst2 (Apache-2.0). The answers were generated by Qwen3.5-9B. None of that data is redistributed here.
Changelog
v1.1 (2026-10-18): parsing fix, same model weights.
- What was wrong: when a clause has text both before and after its list, the model sometimes writes the node's
text field twice, the lead-in before children and the closing words after them. Standard JSON parsing keeps only the last value of a repeated key, so v1.0's parser silently dropped the lead-in.
- Where it happened: on the held-out Encompass policy it removed "Injury arising out of a premises:" from a personal-injury exclusion. In the unseen-documents test it also overwrote a list of 3 nodes (Liberty Mutual homeowners) and 3 table row names (Capital One).
- The fix:
parse_page keeps every value. A second text becomes a trailing paragraph, which is how the reference trees store such text, and a second children list is appended. Each page's log reports any repeated keys, so this can't happen silently.
- Effect (saved outputs re-scored, no retraining):
- the Encompass clause now matches the reference exactly;
- Encompass node F1 0.921 → 0.923 and text similarity 0.997 → 0.998;
- unseen documents: Liberty Mutual homeowners 99.6% → 99.7% of words captured, Capital One 99.6% → 99.7%;
v1.0 (2026-10-16): first release.
Citation
BibTeX:
@misc{ranjan2026policystructure, title = {Qwen3.5-9B Policy Structure: page-by-page nested JSON extraction of insurance policies with a LoRA adapter}, author = {Ranjan, Ashish}, year = {2026}, howpublished = {\url{https://huggingface.co/Ashish-Ranjan/Qwen3.5-9B-Policy-Structure-LoRA}}}
APA: Ranjan, A. (2026). Qwen3.5-9B Policy Structure: page-by-page nested JSON extraction of insurance policies with a LoRA adapter [Computer software]. Hugging Face. https://huggingface.co/Ashish-Ranjan/Qwen3.5-9B-Policy-Structure-LoRA
Glossary
Table with columns: Term, Meaning| Term | Meaning |
|---|
| Node | One unit of the document: a section, clause, item, paragraph and so on |
| Reference tree | A hand-authored, correct tree for a whole document, used for training targets and scoring |
| Open path | The chain of nodes still open at the end of the previous page; it goes into the next page's prompt |
| Stub | A {"cont": true, ...} node in a page's output that names an open node the page continues, instead of repeating it |
| Bracket repair | repair_json: fixes miscounted closing brackets in a page's output before merging |
| Lenient merge | merge_lenient: attaches a page to the tree; a stub that doesn't match becomes a new node instead of dropping the page |
Written by Ashish Ranjan. For questions, problems or results on other document types, open a discussion in this repository's Community tab.
Acknowledgements
- Qwen team: Qwen3.5-9B.
- ModelScope: ms-swift.
- The vLLM project.
- Modal: the GPUs.
- HuggingFaceM4 (the_cauldron) and OpenAssistant (oasst2): retention data.
- Thinking Machines' "LoRA Without Regret": the LoRA hyperparameter choices.