GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
devenshah21
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
devenshah21
Model Tree
Qwen/Qwen2.5-Coder-7B-Instruct
Input Modalities
Output Modalities
Supported Functionality
Graded by result-set comparison against gold on DuckDB (TPC-H SF=0.1) — the same grader that scores the production Claude agent at 91.7%.
| Model | Execution accuracy | Valid SQL |
|---|---|---|
| Qwen2.5-Coder-7B (base, zero-shot) | 54.5% | 90.9% |
| Qwen2.5-Coder-7B (base, 3-shot) | 61.8% | 94.5% |
| This adapter (SFT, zero-shot) | 74.5% | 98.2% |
| Claude (full production agent) | 91.7%¹ | — |
¹ Full agent: retrieval + few-shot + a 3-attempt self-healing loop. A product comparison, not a like-for-like model comparison.
Per-tier (SFT): simple_select 10/10 · aggregation 10/10 · single_join 8/10 · window_function 6/10 · multi_hop 5/10 · simulation 2/5.
Run zero-shot on Spider dev (1,032 queries, 20 unseen SQLite databases) with a generic prompt — no house-style rules — to test whether fine-tuning damaged general SQL ability:
| Model | Spider dev accuracy |
|---|---|
| Qwen2.5-Coder-7B (base) | 82.4% |
| This adapter (SFT) | 82.7% |
+20pp in-domain for +0.3pp (i.e. zero, within noise) out-of-domain. No catastrophic forgetting: the adapter learned this schema's conventions, not a narrower notion of SQL.
python
from peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizerbase = "Qwen/Qwen2.5-Coder-7B-Instruct"model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(model, "devenshah21/sqlforge-qwen7b-sft")tok = AutoTokenizer.from_pretrained(base)messages = [{"role": "system", "content": HOUSE_STYLE_SYSTEM_PROMPT}, # see Serving contract{"role": "user", "content": "Which region has the lowest revenue?"},]prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)out = model.generate(**tok(prompt, return_tensors="pt").to(model.device),max_new_tokens=512, do_sample=False)print(tok.decode(out[0], skip_special_tokens=True))
Serve with vLLM (as used for all evals):
bash
vllm serve Qwen/Qwen2.5-Coder-7B-Instruct \--enable-lora --lora-modules sqlforge=devenshah21/sqlforge-qwen7b-sft \--max-lora-rank 32 --port 8000
The adapter was trained and benchmarked (74.5%) with one specific prompt shape: a
house-style system prompt (schema + the categorical-literal / revenue-formula / join-path
rules) and the user question, zero-shot, temperature 0. Sending few-shot examples or a
different prompt is an unmeasured configuration and will not reproduce the numbers above —
fine-tuning already absorbed what few-shot was teaching. The exact prompt builder lives in
phase0/run_baseline.py (build_system_prompt).
| Method | bf16 LoRA (no quantization) |
| LoRA | r=32, α=64, dropout 0.05, all 7 projections (q,k,v,o,gate,up,down) |
| Data | 2,314 execution-validated synthetic pairs (distilled from Claude, gated on DuckDB execution + house-style checks) |
| Schedule | 2 epochs, lr 2e-4 cosine, effective batch 16, max_seq 2048, completion-only loss |
| Hardware | 1× A40 (48GB) |
churn_risk='High' vs
'HIGH_RISK'), output-column shape, and simulation/CTE format — see the project repo's
failure-mode analysis.