Enriqueag26

Qwen3.8-27B-switchboard-lora-v2

README

License: apache-2.0

What it was trained on

168 verified multi-turn banking trajectories (synthetic tasks on the τ²/τ³-bench banking_knowledge domain, distilled from Qwen3.8-Max and GLM-5.3, gold-verified by DB end state). Every assistant text turn was rewritten into a spoken phone register (numbers, dates, identifiers and tool names preserved verbatim and validated), and every silent tool-call turn received a short spoken say-before-do line.
182 synthetic phone-call traces (Spanish and English; banking, telecom, retail) covering identifier capture and validation on the phone: DNI/NIE/passport/IBAN/phone numbers spoken in groups, with dots, letter-by-word, wrong control letters, ASR mishearings, chunked read-back before any write.
954 training windows of up to 16k tokens rendered with the Qwen3.8 chat template with enable_thinking=False; 2 epochs, cosine LR 1e-4, 64k-token gradient accumulation, custom HF + peft loop (loss masks on assistant turns only). Dev loss 0.564 → 0.369 (best at epoch 1) → 0.387 (end of epoch 2).
Not trained on the 97 τ³-Banking test tasks or on GDPval gold tasks. Synthetic tasks were decontaminated against the test set (5-gram Jaccard, gold-action signature, entity overlap). Track B: the knowledge base is the real Rho-Bank KB of the benchmark; entities are fresh.

How to serve it (important)

bash
vllm serve Qwen/Qwen3.8-27B --enable-lora --lora-modules student=<this repo> --max-lora-rank 64 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --max-model-len 262144

Request with "model": "student" and "chat_template_kwargs": {"enable_thinking": false}, temperature 0. Tensor keys carry the base_model.model.model.language_model.layers.* prefix that vLLM's Qwen3_5ForConditionalGeneration LoRA loader expects; an adapter saved with the plain model.layers.* prefix is loaded but silently never applied (that cost us a day). We have not validated LoRA + MTP speculative decoding together; serve without --speculative-config until you have run a greedy student-vs-base A/B.

Measured behaviour (greedy, thinking off)

Table with columns: Base Qwen3.8-27B, thinking off, This adapter, thinking off, Base, thinking on
	Base Qwen3.8-27B, thinking off	This adapter, thinking off	Base, thinking on
τ³-Banking pass^1, 97 tasks, AA config, 1 trial	22/97 (22.7%)	18/97 (18.6%), paired 90% CI [-11.3, +3.1] pts	44/97 (45.4%)
Tool-call turns preceded by a spoken sentence	16%	100% (1655/1655)	–
Text turns containing markdown	65%	1% (8/544)	–
Blind judge (GLM-5.3) on 17 held-out phone traces vs gold	preferred 2

Example (held-out Spanish trace). Caller: "Quería repasar los últimos movimientos, que me ha salido un cargo que no reconozco." Adapter: "Buenos días. Lo vemos ahora mismo. ¿Me dice su DNI y su fecha de nacimiento?" Base: "Buenos días. Con gusto le ayudo a revisar ese cargo. Primero necesito verificar su identidad. ¿Me podría decir su número de documento (DNI, NIE o pasaporte) y su fecha de nacimiento?"

Known limitations

Benchmark accuracy is not above the base. With thinking off it is at base level (18 vs 22 of 97, within noise); the base with thinking on is far ahead. Typical errors: wrong card/account tier from the knowledge base, wrong computed amounts, incomplete multi-item workflows.
Retry loops on tool errors. In 7 of 97 benchmark conversations the agent retried a failing action against an explicit tool error ("Error: Account eligibility requirements not met.") with a new spoken line each time until the step limit or the 262k context. Use a no-repeat guard in production, and expect v3 to train on error-recovery turns.
Trained and evaluated only in chat form with an LLM user simulator; no ASR/TTS in the loop, no latency tuning, no NVFP4 quantization yet.
Style evaluation uses our own held-out synthetic phone traces and our own gold; PhoneBench itself is not publicly runnable, so no comparison to PhoneLLM is claimed.

Files

adapter_model.safetensors (800 LoRA tensors, 435M parameters), adapter_config.json (peft 0.20).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Enriqueag26

Model Tree

Base

Qwen/Qwen3.8-27B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality