ichetandhembre

ifstruct-lora-adaptor

README

License: apache-2.0

IFStruct v1.0 result

Scored on the full 2,000-row test set of LiquidAI/ifstruct-v1.0 with the benchmark's own validator (byte-identical to Liquid4All/ifstruct).

Table with columns: Model, pass@1, JSON, YAML
Model	pass@1	JSON	YAML
this adapter	96.45	96.6	96.3
Qwen3-4B-Instruct-2507 (base; greedy, earlier run at max_tokens 4096)	76.15	76.4	75.9

Eval setup: greedy decoding (temperature 0), 1 sample per prompt, max_tokens 16000, no system prompt, the model's chat template (non-thinking model), no constrained decoding, vLLM 0.24. Full per-row outputs (prompt, response, validator errors) are in eval/.

Remaining failures (71/2000) are concentrated in prompts whose payload is long free text with heavy escaping (email threads, terminal sessions, dialogue). 18 of them are degenerate repetition loops inside a string value that run to the token cap.

Training

Method: GRPO (verl 0.9.0), CISPO policy loss, KL loss (low_var_kl, 0.01) to the base model, zero-variance group filtering, 16 prompts × 16 rollouts per step, max response 4096 tokens.
Reward: binary pass/fail from the IFStruct validator. No judge model.
Stage 1: fresh LoRA, lr 1e-4, rollout temperature 1.0 → 1.5, 105 steps on 2,406 synthetic prompts.
Stage 2 (this checkpoint): warm start from stage 1, lr 5e-5, rollout temperature 1.5, step 110 on 4,293 synthetic prompts (2 epochs).
Training prompts are synthetic. None of the 2,000 benchmark prompts or their entity types appear in the training data (checked by exact prompt and prompt-prefix match).

Disclosure: how the benchmark was used

This is not a blind held-out score. The benchmark influenced development in two ways:

Data targeting. The synthetic data generator was steered toward the failure types the model showed on this benchmark (failure-mode mix matched to the previous checkpoint's benchmark failures). No benchmark rows were copied into training.
Checkpoint selection. Checkpoints were compared on validation slices drawn from this benchmark (≈288 of the 2,000 rows) and on full-benchmark evaluations.

Expect a lower number on genuinely unseen structured-output distributions.

Usage

python

from peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer base = "Qwen/Qwen3-4B-Instruct-2507"tok = AutoTokenizer.from_pretrained(base)model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")model = PeftModel.from_pretrained(model, "ichetandhembre/ifstruct-lora-adaptor")

With vLLM: vllm serve Qwen/Qwen3-4B-Instruct-2507 --enable-lora --max-lora-rank 8 --lora-modules ifstruct=ichetandhembre/ifstruct-lora-adaptor.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

ichetandhembre

Model Tree

Base

Qwen/Qwen3-4B-Instruct-2507

Adapter

this model

Input Modalities

Text

Output Modalities