blaze-star

qwen2.5-1.5b-sql-qlora

README

License: apache-2.0

Results

Held-out test split, 1,000 examples never seen in training.

Table with columns: Model, Exact match, Token F1, Format compliance
Model	Exact match	Token F1	Format compliance
Base, 0-shot (4-bit)	49.7%	0.925	26.7%
Base, 3-shot (4-bit)	52.3%	0.921	99.3%
QLoRA fine-tuned	74.8%	0.973	99.9%
Delta vs 0-shot	+25.1 pts	+0.049	+73.2 pts

Fine-tuning improved exact match by +25.1 points (49.7% -> 74.8%), a +51% relative gain.

Usage

The model expects the chat template with this system prompt — it was trained with it, and accuracy drops without it.

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct", dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(base, "blaze-star/qwen2.5-1.5b-sql-qlora")
tok = AutoTokenizer.from_pretrained("blaze-star/qwen2.5-1.5b-sql-qlora")

python
SYSTEM = ("You are a text-to-SQL engine. Given a SQLite schema and a question, reply with a "
          "single SQL query that answers the question. Output only the SQL query: no "
          "explanation, no comments, no markdown code fences.")

schema = "CREATE TABLE head (age INTEGER)"
question = "How many heads of the departments are older than 56?"

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": f"Schema:\n{schema}\n\nQuestion: {question}\n\nSQL:"},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=96, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
# SELECT COUNT(*) FROM head WHERE age > 56

Use greedy decoding (do_sample=False). Sampling hurts exact match on this task.

What the metric actually measures

Primary metric is normalized exact match against the dataset's reference SQL: lowercased, whitespace collapsed, " unified to ', backticks/brackets stripped, spacing normalized around operators and punctuation. It is strict — a semantically equivalent query that differs in alias naming (AS p vs AS T1) or literal quoting (= '15' vs = 15) counts as a miss.

That strictness is the point, but it must be read correctly: the base model already produces largely correct SQL content (token F1 0.925 before any training). Much of the headroom is conformance to this dataset's canonical SQL style, which is exactly what task-specific fine-tuning buys you. To keep that claim honest this project reports three separate baselines rather than one:

Table with columns: Baseline, Exact match, Format compliance, What it controls for
Baseline	Exact match	Format compliance	What it controls for
4-bit, 0-shot	49.7%	26.7%	matched conditions — same quantization the adapter trains on
4-bit, 3-shot	52.3%	99.3%	isolates output formatting from SQL convention
fp16, 0-shot	57.3%	99.4%	strongest untrained configuration

The 3-shot baseline is the important control. Three in-context examples raise format compliance to 99.3% — the model stops wrapping output in markdown fences almost entirely — yet exact match moves only to 52.3%. Formatting was therefore not the bottleneck, and gains above that line are genuine SQL-convention learning, not prompt-format cleanup.

Both models receive an identical prompt and identical output post-processing (fence stripping, leading-prose removal, first-statement extraction), so neither is advantaged by the harness. Secondary metrics: order-insensitive token F1 over SQL tokens (partial credit) and format compliance (fraction of raw generations that were already bare SQL).

Data and leakage control

b-mc2/sql-create-context — natural-language question + CREATE TABLE schema -> SQLite query.

The 78,577 raw rows are deduplicated on a SHA-1 of the normalized (question, schema) pair (4 exact duplicates dropped), shuffled with seed 42, and then test is carved off first, before val and train. Splits: 12,000 train / 750 val / 1,000 test.

Split disjointness is asserted at build time and recorded in data/split_report.json:

json
{"train_test_overlap": 0, "val_test_overlap": 0, "train_val_overlap": 0}

prepare_data.py raises if any of these is non-zero, so a leaking split cannot be trained on. The test split was used only by evaluate.py, never by train.py.

Training

Table

Base model	`Qwen/Qwen2.5-1.5B-Instruct`
Method	QLoRA — frozen 4-bit NF4 base (double quant, bf16 compute) + LoRA adapters
LoRA	r=16, alpha=32, dropout=0.05, on `q,k,v,o,gate,up,down_proj`
Trainable params	18.5M of 1.56B (1.18%)
Optimizer	paged AdamW 8-bit, lr 0.0002, cosine schedule, 3% warmup, grad-clip 0.3
Effective batch	32 (16 x 2 accumulation)
Epochs

Loss is computed only on the assistant turn, so the model is never rewarded for reproducing the schema or the question.

Quantization: latency, VRAM, and quality

Merged fp16 model re-quantized with bitsandbytes and benchmarked on the same A100. Latency is a single request generating exactly 64 tokens (20 runs after 3 warmups); quality is exact match on the first 300 test examples.

Table with columns: Precision, Weights VRAM, Peak VRAM, Latency (bs=1, 64 tok), Decode tok/s, Batch-16 tok/s, Exact match
Precision	Weights VRAM	Peak VRAM	Latency (bs=1, 64 tok)	Decode tok/s	Batch-16 tok/s	Exact match
fp16	3.09 GB	3.3 GB	1862.4 ms	34.4	436.5	76.7% (n=300)
8bit	1.8 GB	2.1 GB	27963.2 ms	2.3	16.7

Limitations

Single-table, synthetic-ish schemas. sql-create-context schemas are small CREATE TABLE statements derived from WikiSQL/Spider. Performance will not transfer directly to large multi-table production warehouses.
Exact match is style-sensitive. A correct query written in a different but valid style scores zero. Token F1 is reported alongside for this reason.
No execution-based evaluation. Queries are compared as strings, not run against a database, so semantic equivalence is undercounted.
4-bit inference costs accuracy. The base model loses ~8 points of exact match going from fp16 to 4-bit (57.3% -> 49.7%); see the quantization table for the fine-tuned model's own fp16/8-bit/4-bit spread.
English only, and the model emits SQLite dialect.

Intended use

Converting natural-language questions into SQLite queries over small, explicitly-provided schemas — a component inside a larger system that supplies the schema and validates or sandboxes the generated query. Do not execute generated SQL against a production database without validation; the model can emit syntactically valid queries that are semantically wrong.

Training code

Full, reproducible pipeline: https://github.com/harshb20/qwen2.5-1.5b-sql-qlora

Citation

bibtex
@misc{qwen25-1.5b-sql-qlora,
  title  = {QLoRA text-to-SQL fine-tune of Qwen2.5-1.5B-Instruct},
  author = {harshb20},
  year   = {2026},
  url    = {https://github.com/harshb20/qwen2.5-1.5b-sql-qlora}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

blaze-star

Model Tree

Base

Qwen/Qwen2.5-1.5B-Instruct

Adapter

this model

Input Modalities

Text

Output Modalities