Model details
Table | |
|---|
| Base model | Qwen/Qwen3-8B (8.2B parameters, Apache 2.0) |
| Method | Supervised fine-tuning with LoRA (PEFT 0.21.2), non-thinking mode |
| Trainable parameters | 174.6M (rank 64 on all attention and MLP projections, about 2.1% of the base) |
| Variants | LoRA adapter (this repo); merged bf16 weights: JulianAT/sv-lotse-qwen3-8b-sft-merged |
| Task | Multi-turn tool calling: clarify the situation, call the Lotse tools, finish with submit_plan |
| Languages | German (primary, incl. Austrian dialect and plain language), English, Turkish, Ukrainian, Hungarian, Arabic, Romanian, Bosnian/Croatian/Serbian, Polish |
| Context | Trained on sequences of up to 8,192 tokens |
| Training data | JulianAT/at-benefits (synthetic, CC BY 4.0) |
| Developer | Julian Schmidt (team Attention Seekers), for the Dachverband KI Challenge, Use Case 1 |
| License | Apache 2.0 |
Role in SV KI-Lotse
SV KI-Lotse is a voice-first guide through Austrian social insurance. Eligibility rules,
offices and document requirements live in a deterministic, cited rule engine, not in the model.
The model plays the pilot: it asks only the questions the engine needs, calls the tools, and
submits a plan with the benefits, the responsible office per benefit, the required documents, the
next step and citations. Every fact in the plan is meant to come from a tool output.
Table with columns: Tool, Purpose| Tool | Purpose |
|---|
lotse_catalog_findBenefits | Find benefits and their sourced eligibility criteria |
lotse_catalog_getBenefit | Read one benefit record |
lotse_office_resolve | Resolve the responsible office from carrier, subject status and region |
lotse_documents_checklist | List required documents and how to obtain them |
lotse_kb_search | Retrieve sourced passages from the knowledge base |
submit_plan | Submit the final plan |
The exact tool schemas used in training are stored in the tools column of every row of the
dataset's dialogues config.
Training data
All examples come from the train split of JulianAT/at-benefits. The data is
synthetic: plans, offices, documents and citations are computed by the product's rule engine,
never by a language model; language models only wrote surface text. No real person is described.
- Dialogues, tier O (oracle): the gym oracle's trajectories; template questions in German,
English or Turkish, exact tool calls and tool outputs.
- Dialogues, tier A (verbalised): oracle trajectories whose natural-language turns were
written by a model in nine languages and passed an LLM judge. A sample contains medical
questions that the pilot hands off to a doctor or 1450.
- Dialogues, tier B (teacher): a strong model piloting against a simulated user with the real
tools, kept only at gym reward of at least 0.9.
- Situations: single-turn examples. A persona card and a model-written situation text go in,
the engine's gold plan comes out.
All counts below are from data_report.json.
Dialogue selection
Table with columns: Step, Tier O, Tier A, Tier B, Total| Step | Tier O | Tier A | Tier B | Total |
|---|
| Train-split dialogues | 5,000 | 250 | 1 | 5,251 |
| Dropped, longer than 8,192 tokens | 2,118 | 126 | 0 | 2,244 |
| Fit | 2,882 | 124 | 1 | 3,007 |
| Used | 301 | 124 | 1 | 426 |
Tier O was subsampled from 2,882 to 301 dialogues to stay within a 3,000,000-token budget. No
dialogue was dropped for medical-boundary flags.
Training mix
Table with columns: Source, Examples, Note| Source | Examples | Note |
|---|
| Dialogues, tier O | 301 | |
| Dialogues, tier A | 248 | 124 dialogues, upsampled 2x |
| Dialogues, tier B | 1 | |
| Situations | 333 | every model-written situation text in the train split |
| Total | 883 | |
Table with columns: Statistic, Value| Statistic | Value |
|---|
| Training tokens | 2,765,532 |
| Loss tokens (assistant turns) | 1,234,396 (44.6%) |
| Example length p50 / p90 / max | 3,524 / 5,168 / 6,042 tokens |
| Validation examples | 172 |
| Data hash (sha256) | af973a4a64fd92bc9cc0dfbb2e8ec468521cb8d355ad3cb4cd0c7ece1e43f95c |
Languages (dialogues counted once, before upsampling)
Table with columns: Language, Dialogues, Situations| Language | Dialogues | Situations |
|---|
| German (de) | 273 | 116 |
| English (en) | 60 | 39 |
| Turkish (tr) | 58 | 28 |
| Ukrainian (uk) | 10 | 12 |
| Hungarian (hu) | 6 | 16 |
| Romanian (ro) | 5 | 40 |
| Arabic (ar) | 5 | 23 |
| Bosnian/Croatian/Serbian (bcs) | 5 | 29 |
| Polish (pl) | 4 | 30 |
| Total | 426 | 333 |
Situations by life situation: illness 161, family with child 54, prevention 52, after hospital
30, low income 24, uninsured 12.
Training procedure
Table with columns: Setting, Value| Setting | Value |
|---|
| LoRA rank / alpha | 64 / 128 (scale 2.0) |
| LoRA dropout | 0.05 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Bias / DoRA / rsLoRA | none / no / no |
| Max sequence length | 8,192 tokens |
| Chat template | Qwen3, tool schemas passed as tools=, enable_thinking=False |
| Tool outputs | each tool message compacted to at most 250 characters |
| Tier upsampling | A 2x, O 1x, B 1x |
| Token budget | 3,000,000 |
The 250-character tool-output limit came from a sweep on a sample of dialogues: at 1,500
characters 25% of them fit into 8,192 tokens (median 8,852 tokens), at 250 characters 59% (median
5,094 tokens). Learning rate, schedule and epoch count are not recorded in the published
artifacts and are not listed here.
Final validation loss: 0.0359 on the 172 validation examples. This is the token-level loss on
held-out examples from the same synthetic distribution, much of it template and engine-generated
text. It shows the model has fit the format; it is not a task metric and says nothing about plan
correctness on new situations or with real users.
Evaluation
Task evaluations have not been run yet. There are no accuracy, tool-call or safety numbers for
this model, and none should be inferred from the validation loss.
Planned, on the AT-Benefits gym, whose reward terms are already computed for every dataset row:
- Plan accuracy: benefit F1, office accuracy, document F1, next-step correctness, citation
validity, excess questions.
- Tool-call validity: parseable calls, schema-valid arguments, ids copied verbatim from tool
outputs.
- Safety boundary: medical-boundary red-team cases (no diagnosis, dosage, interpretation of
findings or risk assessment).
- Generalisation: the held-out care family (
heldout_family split) and the non-German
languages.
Results will be added to this card once measured.
How to use
Use the Qwen3 chat template with the tool schemas passed as tools= and enable_thinking=False,
as in training, and keep tool outputs at 250 characters or fewer. The model either asks the person
a question (plain text) or emits <tool_call> blocks; the run ends when it calls submit_plan and
writes its closing answer.
Requires transformers>=4.56 and peft. adapter_config.json records the base model as a local path from the training machine, so load Qwen/Qwen3-8B explicitly and apply the adapter on top instead of using AutoPeftModelForCausalLM.
import jsonimport re import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer BASE = "Qwen/Qwen3-8B"ADAPTER = "JulianAT/sv-lotse-qwen3-8b-sft" tokenizer = AutoTokenizer.from_pretrained(ADAPTER)model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, ADAPTER).eval() SYSTEM = ( "Du bist SV KI-Lotse, ein Assistent für österreichische Sozialversicherungs-Leistungen. " "Kläre die Situation mit wenigen, gezielten Fragen, nutze die Werkzeuge für Leistungen, " "zuständige Stellen und Unterlagen, und reiche am Ende mit submit_plan einen Plan ein. " "Jede Leistung braucht eine Quelle aus den Werkzeugen. Keine medizinischen Einschätzungen: " "keine Diagnosen, Dosierungen, Befundinterpretationen oder Risikobewertungen.\n" "Fallkontext: caseId=<case-id>, subjectId=<subject-id>.\n" "Antworte in der Sprache der Person; bei Bedarf in Einfacher Sprache.") # Minimal excerpt. For best results pass all six tools with the exact training schemas:# row = next(iter(load_dataset("JulianAT/at-benefits", "dialogues", split="val", streaming=True)))# TOOLS = json.loads(row["tools"])TOOLS = [ { "type": "function", "function": { "name": "lotse_catalog_findBenefits", "description": ( "Find administrative benefits and sourced eligibility criteria. Omit lifeSituation " "when its exact catalog value is unknown. Copy a returned item's id verbatim into " "benefitId for subsequent tools; never invent, translate or add a ref prefix to an id." ), "parameters": { "type": "object", "properties": { "lifeSituation": {"type": "string"}, "goalText": {"type": "string"}, "subject": { "type": "object", "properties": { "ageBand": {"type": "string"}, "carrier": {"type": "string"}, "region": {"type": "string"}, "status": {"type": "string"}, }, "additionalProperties": False, }, }, "required": ["goalText", "subject"], "additionalProperties": False, }, }, }, { "type": "function", "function": { "name": "submit_plan", "description": "Submit the final plan. Every benefit must be backed by citations returned by tools.", "parameters": { "type": "object", "properties": { "benefits": {"type": "array", "items": {"type": "string"}}, "office_per_benefit": {"type": "object", "additionalProperties": {"type": "string"}}, "documents_per_benefit": { "type": "object", "additionalProperties": {"type": "array", "items": {"type": "string"}}, }, "next_step": { "type": "object", "properties": { "benefitId": {"type": "string"}, "stepId": {"type": "string"}, "description": {"type": "string"}, }, }, "citations": {"type": "array", "items": {"type": "string"}}, }, "required": ["benefits", "office_per_benefit", "documents_per_benefit", "next_step", "citations"], }, }, },] TOOL_OUTPUT_CHARS = 250 # same limit as in training def generate(messages): inputs = tokenizer.apply_chat_template( messages, tools=TOOLS, enable_thinking=False, add_generation_prompt=True, return_dict=True, return_tensors="pt", ).to(model.device) out = model.generate(**inputs, max_new_tokens=1024, do_sample=False) return tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True) def parse_tool_calls(text): return [json.loads(c) for c in re.findall(r"<tool_call>\s*(.*?)\s*</tool_call>", text, re.S)] messages = [ {"role": "system", "content": SYSTEM}, {"role": "user", "content": "Ich bin in Pension, wohne in Graz und war im Urlaub in Kroatien beim Arzt. Was mache ich mit der Rechnung?"},] for _ in range(32): reply = generate(messages) calls = parse_tool_calls(reply) if not calls: break # a question for the person or the closing answer messages.append({"role": "assistant", "content": "", "tool_calls": [{"type": "function", "function": c} for c in calls]}) for call in calls: output = run_tool(call["name"], call["arguments"]) # your tool backend, returns a string messages.append({"role": "tool", "content": output[:TOOL_OUTPUT_CHARS]}) print(reply)
vLLM
vllm serve Qwen/Qwen3-8B \ --enable-lora --lora-modules sv-lotse=JulianAT/sv-lotse-qwen3-8b-sft --max-lora-rank 64 \ --max-model-len 8192 \ --enable-auto-tool-choice --tool-call-parser hermes
--max-lora-rank 64 is required (the vLLM default is 16). The adapter's chat template is identical to the base model's, so the server needs no template override.
from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")resp = client.chat.completions.create( model="sv-lotse", messages=messages, # system prompt, turns and tool messages as above tools=TOOLS, temperature=0, extra_body={"chat_template_kwargs": {"enable_thinking": False}},)message = resp.choices[0].message # message.tool_calls holds the parsed calls
--max-model-len 8192 matches the training length. The base model supports longer contexts, but
this fine-tune has not seen them.
Training compacted every tool message to at most 250 characters, so full-length tool outputs are
out of distribution. Apply the same limit at inference. The repository's compact_json_text
(ml/sft/data.py) shortens JSON
outputs to the limit while keeping them valid JSON; the plain cut in the example above is the
simplest fallback. Return a short result (for example {"ok": true}) for submit_plan; the model
then writes its closing message to the person.
Intended use
- Research on tool-calling agents for benefits navigation.
- The pilot model inside SV KI-Lotse or a comparable harness, where the tools are backed by a rule
engine and the server checks every plan before anything is shown.
- A starting point for further training, such as multi-turn GRPO on the AT-Benefits gym described
in the paper.
Out of scope: standalone use without the tools, eligibility decisions, medical, legal or financial
advice, and production or public deployment before the task evaluations above exist.
Limitations
- Not evaluated. No task metrics exist yet (see Evaluation).
- Synthetic data, rule-engine labels. Every label comes from the product's rule engine and
catalog. Errors or gaps in those rules are learned as truth; the labels are navigation hints, not
eligibility decisions. Benefit, office and document ids reflect the catalog at data-generation
time.
- No human study. The model has not been tested with real users or real conversations.
- Small training set. 883 examples: 426 unique dialogues, 301 of them template text from the
oracle, and 333 single-turn situations.
- German-centric. 273 of 426 dialogues are German. Ukrainian, Hungarian, Romanian, Arabic,
Bosnian/Croatian/Serbian and Polish have 4 to 10 dialogues each; expect weaker behaviour outside
German, English and Turkish.
- Coverage. The care family (area D, e.g. Pflegegeld) is held out of the training split by
design. The situations cover illness, family with child, prevention, after hospital, low income
and uninsured.
- Truncated tool outputs. At 250 characters, long outputs (such as a catalog search returning
many benefits) shrink to a short preview, while the training targets still list every id. The
model therefore partly relies on catalog ids memorised during training. Validate every benefit,
office and document id against the catalog before use.
- Mode and length. Trained in non-thinking mode only and on sequences of up to 8,192 tokens.
Safety
This model does not give medical, legal or financial advice and does not decide eligibility. It
must never diagnose, recommend doses, interpret findings or assess medical risk. Medical questions
go to the treating doctor or, in Austria, to the health hotline 1450. The training data includes
dialogues in which the pilot hands such questions off, but boundary compliance has not been
measured. Deploy only behind server-side boundary checks, as SV KI-Lotse does.
Citation
The paper describes the data, the gym and the SFT and GRPO recipe. It predates this checkpoint and
reports no results for it.
@misc{schmidt2026lotserl, title = {Lotse-RL: Executable Eligibility Rules as Verifiable Ground Truth for Benefits Navigation in Austrian Social Insurance}, author = {Schmidt, Julian}, year = {2026}, howpublished = {Preprint}, url = {https://github.com/Julian-AT/sv-lotse/blob/main/paper/lotse-rl.pdf}}
Acknowledgements
- Base model: Qwen3-8B by the Qwen team (Apache 2.0).
- Built for the KI Challenge of the Dachverband der Sozialversicherungsträger (Use Case 1).
- Training data:
JulianAT/at-benefits (CC BY 4.0), generated from the SV KI-Lotse
rule engine and catalog. Related:
JulianAT/at-benefits-personas
and JulianAT/at-sv-law.
- Austrian law texts in the project's corpora: Datenquelle: Rechtsinformationssystem des Bundes
(RIS), Bundeskanzleramt Österreich. Licence CC BY 4.0.