k3dani

Qwen3.6-35B-A3B-invoice-decision-lora

README

License: apache-2.0

Write-up

Blog: Where is it worth training the model? (magyarul): this adapter, and a second round where the same recipe applied to tool selection lost to the calibrated base model.
Study (Hungarian, long form, with methodology and limitations): docai.hu/kutatas/hol-erdemes-tanitani
Evidence: pre-registered protocols, code and per-item read-outs in docai-evals

Results (synthetic test set, vLLM, all four test strata pooled)

τ is chosen on the validation split for 95% target precision (one-sided Clopper–Pearson).

Table with columns: method, coverage@95, precision reached, AURC, ECE, X recall
method	coverage@95	precision reached	AURC	ECE	`X` recall
base model, calibrated label readout	0.616	0.975	0.0273	0.043	0.434
this adapter (mean of 3 seeds)	0.752	0.979	0.0018	0.010	0.957

The AURC curve ends at each method's own non-X coverage; cut at a common 0.745 it is 0.0108 vs. 0.0012–0.0019. The baseline's temperature was fitted on single-permutation readouts; refitted on the permutation average, its ECE is 0.026.

Gain over the calibrated readout: +13.7 coverage points (95% CI [4.8; 21.0]); +9.8 [1.7; 15.5] on two category roots held out from training.
Unseen suppliers: the original supplier stratum turned out not to be held out, so a separate split with 16 new suppliers was built and frozen before measuring these same adapters on it (test_new_suppliers in the dataset). There the gain is +19.5 points (95% CI [10.2; 29.2], clustered by supplier), positive for all 16 suppliers: 0.568 → 0.758–0.769.
The gain comes mostly from recognising "none of these": coverage@95 is capped by the non-X share (~0.75), and the adapter reaches it. The size of the gain depends on how often X is the right answer: +13.7 points at 25% X, +3.5 at 10%.
At a 90% precision target the two methods cover the same share; the adapter does it at 0.979 precision instead of 0.928. At 70% coverage, its error rate is 0.4–0.5% against 4.8%.
The data are synthetic. Validate on your own labelled sample before relying on the threshold.

Serving note — read before co-hosting

The adapter works in vLLM when loaded at runtime. Its effect matches the HF/peft reference (Pearson r = 0.976 on the label-logit shifts), and agreement between the two engines does not degrade.

Enabling LoRA changes the base model's outputs too. In vLLM 0.30, --enable-lora changes the compute path for every request: the compiled graph, and this MoE model's FP8 expert kernel (DeepGEMM → Triton, "fused MoE LoRA implementation"). Requests that do not use the adapter therefore run on different numerics:

on a 1000-item subset, 5.8–7.7% of base-model top labels changed, concentrated on uncertain items;
LoRA-enabled instances landed in one of several numeric modes from restart to restart, while instances without LoRA reproduced exactly;
we found no detectable loss of accuracy (−0.1 to +0.1 points, 95% CI about ±1.3, serial traffic only), and the distance to the HF reference did not grow;
we did not measure mixed batches (adapter and base requests together) or chat decode throughput.

This refuted the pre-registered condition that LoRA-free requests stay unchanged on a shared instance. If you co-host the adapter with existing traffic, expect uncertain base-model decisions to reshuffle and restart-level bitwise reproducibility to be lost. A separate instance avoids both.

Training

Base: Qwen/Qwen3.6-35B-A3B-FP8, trained on a BF16 copy dequantized from the FP8 checkpoint, served on the FP8 checkpoint (no fake-quant needed).
LoRA r = 16, α = 32, dropout 0.05, 1 epoch, lr 1e-4. Targets: attention q/k/v/o, GatedDeltaNet in_proj_qkv/z/a/b and out_proj, and the shared-expert gate/up/down. The routed experts carry no LoRA.
Loss at the decision position only: cross-entropy over the labels present (label smoothing ε = 0.05) + 0.5 · Brier; soft targets on items flagged ambiguous.
Three seeds. Seed noise on validation (2 points of coverage@95) exceeded the differences between the tuned configurations.

How to use

Prompt: decision_prompt.py builds the exact prompt the adapter was trained and tested with. It uses the chat template with thinking disabled, lists the options as A–J in your order, and puts X = "Egyik sem" last. The decision is the first generated token.
vLLM (the measured path): start the FP8 base with --enable-lora --max-lora-rank 16 and load vllm/ at runtime (/v1/load_lora_adapter). Then call /v1/completions with the token ids as prompt, max_tokens = 1, temperature = 0 and logprob_token_ids = the ids of the labels present (decision_prompt.decide). Apply a softmax over those log-probabilities after dividing them by the temperature.

Repository layout

Table with columns: path, content
path	content
`adapter_config.json`, `adapter_model.safetensors`	seed 1, peft variant (the main release)
`vllm/`	seed 1, vLLM variant
`seed2/`, `seed3/` (each with `vllm/`)	the two other training seeds; coverage@95 on the test ranges 0.745–0.761 across seeds
`calibration.json`	temperature and τ@95 / τ@90 per seed (synthetic validation split, vLLM)

autotrust/JEV-27B follows the same serving idea at a general scale. A decision LoRA (its "System 1") and the untouched base model ("System 2") run on one vLLM engine and are routed per request; the LoRA is distilled from a hosted decision model. This adapter differs in three ways:

it is task-specific and Hungarian;
it has an explicit "none of these" label, and its gain comes from that label;
it is evaluated by coverage at a target precision against gold labels, with pre-registered held-out strata.

Our serving finding applies to that pattern too. In vLLM 0.30, enabling LoRA on an FP8 MoE model changes the numerics of requests that do not use the adapter.

Limitations

Trained and tested on synthetic Hungarian procurement data from one business profile.
Ambiguous lines (e.g. size variants) are classified confidently, usually choosing one of the acceptable options (97.5–98.3% on the test set). They are not abstained on.
Coverage@95 has poor resolution (MDE ~11–12 points), so AURC is the co-primary endpoint.

Citation

bibtex

@misc{kiss2026decisionlora,  title        = {A decision LoRA on a served chat model: calibrated abstention for Hungarian invoice-line                  to catalog matching},  author       = {Kiss, D{\'a}niel},  organization = {K3Net Kft.},  year         = {2026},  howpublished = {\url{https://huggingface.co/k3dani/Qwen3.6-35B-A3B-invoice-decision-lora}}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

k3dani

Model Tree

Base

Qwen/Qwen3.6-35B-A3B-FP8

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Write-up

Blog: Where is it worth training the model? (magyarul): this adapter, and a second round where the same recipe applied to tool selection lost to the calibrated base model.
Study (Hungarian, long form, with methodology and limitations): docai.hu/kutatas/hol-erdemes-tanitani
Evidence: pre-registered protocols, code and per-item read-outs in docai-evals

Results (synthetic test set, vLLM, all four test strata pooled)

τ is chosen on the validation split for 95% target precision (one-sided Clopper–Pearson).

Table with columns: method, coverage@95, precision reached, AURC, ECE, X recall
method	coverage@95	precision reached	AURC	ECE	`X` recall
base model, calibrated label readout	0.616	0.975	0.0273	0.043	0.434
this adapter (mean of 3 seeds)	0.752	0.979	0.0018	0.010	0.957

Gain over the calibrated readout: +13.7 coverage points (95% CI [4.8; 21.0]); +9.8 [1.7; 15.5] on two category roots held out from training.
Unseen suppliers: the original supplier stratum turned out not to be held out, so a separate split with 16 new suppliers was built and frozen before measuring these same adapters on it (test_new_suppliers in the dataset). There the gain is +19.5 points (95% CI [10.2; 29.2], clustered by supplier), positive for all 16 suppliers: 0.568 → 0.758–0.769.
The gain comes mostly from recognising "none of these": coverage@95 is capped by the non-X share (~0.75), and the adapter reaches it. The size of the gain depends on how often X is the right answer: +13.7 points at 25% X, +3.5 at 10%.
At a 90% precision target the two methods cover the same share; the adapter does it at 0.979 precision instead of 0.928. At 70% coverage, its error rate is 0.4–0.5% against 4.8%.
The data are synthetic. Validate on your own labelled sample before relying on the threshold.

Serving note — read before co-hosting

The adapter works in vLLM when loaded at runtime. Its effect matches the HF/peft reference (Pearson r = 0.976 on the label-logit shifts), and agreement between the two engines does not degrade.

on a 1000-item subset, 5.8–7.7% of base-model top labels changed, concentrated on uncertain items;
LoRA-enabled instances landed in one of several numeric modes from restart to restart, while instances without LoRA reproduced exactly;
we found no detectable loss of accuracy (−0.1 to +0.1 points, 95% CI about ±1.3, serial traffic only), and the distance to the HF reference did not grow;
we did not measure mixed batches (adapter and base requests together) or chat decode throughput.

Training

Base: Qwen/Qwen3.6-35B-A3B-FP8, trained on a BF16 copy dequantized from the FP8 checkpoint, served on the FP8 checkpoint (no fake-quant needed).
LoRA r = 16, α = 32, dropout 0.05, 1 epoch, lr 1e-4. Targets: attention q/k/v/o, GatedDeltaNet in_proj_qkv/z/a/b and out_proj, and the shared-expert gate/up/down. The routed experts carry no LoRA.
Loss at the decision position only: cross-entropy over the labels present (label smoothing ε = 0.05) + 0.5 · Brier; soft targets on items flagged ambiguous.
Three seeds. Seed noise on validation (2 points of coverage@95) exceeded the differences between the tuned configurations.

How to use

Prompt: decision_prompt.py builds the exact prompt the adapter was trained and tested with. It uses the chat template with thinking disabled, lists the options as A–J in your order, and puts X = "Egyik sem" last. The decision is the first generated token.
vLLM (the measured path): start the FP8 base with --enable-lora --max-lora-rank 16 and load vllm/ at runtime (/v1/load_lora_adapter). Then call /v1/completions with the token ids as prompt, max_tokens = 1, temperature = 0 and logprob_token_ids = the ids of the labels present (decision_prompt.decide). Apply a softmax over those log-probabilities after dividing them by the temperature.

Repository layout

Table with columns: path, content
path	content
`adapter_config.json`, `adapter_model.safetensors`	seed 1, peft variant (the main release)
`vllm/`	seed 1, vLLM variant
`seed2/`, `seed3/` (each with `vllm/`)	the two other training seeds; coverage@95 on the test ranges 0.745–0.761 across seeds
`calibration.json`	temperature and τ@95 / τ@90 per seed (synthetic validation split, vLLM)

it is task-specific and Hungarian;
it has an explicit "none of these" label, and its gain comes from that label;
it is evaluated by coverage at a target precision against gold labels, with pre-registered held-out strata.

Our serving finding applies to that pattern too. In vLLM 0.30, enabling LoRA on an FP8 MoE model changes the numerics of requests that do not use the adapter.

Limitations

Trained and tested on synthetic Hungarian procurement data from one business profile.
Ambiguous lines (e.g. size variants) are classified confidently, usually choosing one of the acceptable options (97.5–98.3% on the test set). They are not abstained on.
Coverage@95 has poor resolution (MDE ~11–12 points), so AURC is the co-primary endpoint.

Citation

bibtex

@misc{kiss2026decisionlora,  title        = {A decision LoRA on a served chat model: calibrated abstention for Hungarian invoice-line                  to catalog matching},  author       = {Kiss, D{\'a}niel},  organization = {K3Net Kft.},  year         = {2026},  howpublished = {\url{https://huggingface.co/k3dani/Qwen3.6-35B-A3B-invoice-decision-lora}}}

Qwen3.6-35B-A3B-invoice-decision-lora

README

Write-up

Results (synthetic test set, vLLM, all four test strata pooled)

Serving note — read before co-hosting

Training

How to use

Repository layout

Related work

Limitations

Citation

Explore FriendliAI today

README

Write-up

Results (synthetic test set, vLLM, all four test strata pooled)

Serving note — read before co-hosting

Training

How to use

Repository layout

Related work

Limitations

Citation