Results (held-out, measured identically base vs fine-tuned)
Measured on 150–200 held-out schemas the model never trained on. Fine-tuned
result independently cross-checked via the merged fp16 model (74% nested).
Table with columns: Metric, Base Qwen3-1.7B, Fine-tuned, Δ| Metric | Base Qwen3-1.7B | Fine-tuned | Δ |
|---|
Nested / $ref schemas | 58% | 72–74% | +14–16 |
| Flat schemas | 87% | 92% | +5 |
| Overall | 72.5% | 82–83% | +10 |
| Parse-fail rate | 4% | 2.5–4% | ~flat |
Official results — JSONSchemaBench test split (n=360, native/unconstrained)
Base Qwen3-1.7B vs fine-tuned, identical prompt + Draft-2020-12 validator, greedy.
Table with columns: Dataset, Base, Fine-tuned, Gain| Dataset | Base | Fine-tuned | Gain |
|---|
| Github_trivial | 49% | 82% | +33% |
| Github_easy | 78% | 93% | +16% |
| Github_medium | 42% | 56% | +13% |
| Github_hard | 11% | 24% | +13% |
McNemar paired test: 76 schemas fixed, 14 broke, p=1.78e-11 (highly significant).
Our base measurement aligns with the published Llama-3.2-1B native numbers
(confirming comparability); the fine-tune beats published small-model native
compliance on nearly every dataset.
Why this matters
Small models are widely deployed for narrow, high-volume, structured tasks —
but their published native schema-compliance collapses on complex schemas
(JSONSchemaBench reports ~13–38% for small models on the hardest datasets).
This fine-tune lifts native compliance on exactly that hard regime, so the
model is more usable without attaching a constrained-decoding engine
(and better as a base with one).
How to use
from transformers import AutoModelForCausalLM, AutoTokenizer
import json, torch
m = AutoModelForCausalLM.from_pretrained("ujwal00/qwen3-1.7b-json", dtype=torch.float16, device_map="auto")
tok = AutoTokenizer.from_pretrained("ujwal00/qwen3-1.7b-json")
SYSTEM = ("You are a precise JSON generator. Given a JSON Schema, output ONE JSON value "
"that strictly validates against it. Output ONLY the JSON value.")
schema = {"type":"object","properties":{"name":{"type":"string"},"age":{"type":"integer"}},"required":["name","age"]}
msgs = [{"role":"system","content":SYSTEM},
{"role":"user","content":f"Generate a JSON value that validates against this JSON Schema:\n\n{json.dumps(schema)}\n\nReturn only the JSON value."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=False, return_tensors="pt").to(m.device)
out = m.generate(ids, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
Low-VRAM machines (e.g. a 6 GB laptop GPU): add low_cpu_mem_usage=True to
from_pretrained (and optionally max_memory={0: "5GiB", "cpu": "6GiB"}) so
loading doesn't exhaust memory. On a normal 8 GB+ GPU the snippet above works as-is.
Verified live output
Table with columns: Schema, Model output, Valid?| Schema | Model output | Valid? |
|---|
{name:str, age:int} (required both) | {"name": "sample_value", "age": 50} | ✅ |
| nested: user{id,email:email-format, roles:enum[]} | {"user": {"id": 50, "email": "user@example.com", "roles": ["user"]}} | ✅ |
Limitations (stated honestly)
- Numbers are native / unconstrained; constrained decoding scores differently and is out of scope.
- Trained on JSONSchemaBench-style schemas; may generalize less to very different schema dialects.
- Ultra-large schemas (>4000 chars) were excluded from training data.
- Reported gains are on held-out gold; the official test-split table is the definitive comparison.
Reproducibility
Data recipe, generator, validator, training + eval code, and manifest are in the
project repo (jsonc/, kaggle/). Seed=42; prompt hashed; validator format
self-tested. Base and fine-tuned measured with identical prompt/validator/greedy.