emonHasan01

qwen2.5-3b-text2sql

README

License: apache-2.0

Results

Metrics

Table with columns: exact match, valid SQL, token F1, mean output chars
	exact match	valid SQL	token F1	mean output chars
`unsloth/Qwen2.5-3B-Instruct`	7.7%	98.7%	83.1	75
this model	74.3%	100.0%	97.5	75

Exact match is computed on normalised ASTs, not strings. Both queries are parsed with sqlglot and re-emitted in a canonical form, so differences in case, whitespace and quoting do not count as errors.

The same output cleaning was applied to both models — markdown fences stripped, leading prose removed, first statement taken. Without that the comparison would mostly measure whether a model wraps its answer in backticks. The gain above is about the SQL being right.

Progress

Measured at three points: untrained, step 150, and step 500.

Per-query, not just the aggregate

Per-query

Table with columns: queries
	queries
both correct	22
only this model correct	201
only the base correct	1
neither correct	76

Samples

Table with columns: Question, Reference SQL, Base correct, This model correct
Question	Reference SQL	Base correct	This model correct
How many Attendances have a Score of 4 – 5? Question 4	`SELECT MAX(attendance) FROM table_name_88 WHERE score = "4 –`	no	no
When did woking compete?	`SELECT date FROM table_name_92 WHERE opponent = "woking"`	no	no
What is the local name given to the city of Canberra?	`SELECT capital___endonym__ FROM table_1008653_1 WHERE capita`	no

Loss

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLMimport torch tok = AutoTokenizer.from_pretrained("emonHasan01/qwen2.5-3b-text2sql")model = AutoModelForCausalLM.from_pretrained("emonHasan01/qwen2.5-3b-text2sql", torch_dtype=torch.float16,                                             device_map="auto") schema = "CREATE TABLE head (age INTEGER, head_id VARCHAR)"question = "How many heads of the departments are older than 56?" messages = [    {"role": "system", "content": "You convert questions into a single SQLite query. Reply with the query only: no explanation, no markdown, no commentary."},    {"role": "user", "content": f"Schema:\n{schema}\n\nQuestion: {question}"},]prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)out = model.generate(**inputs, max_new_tokens=128, do_sample=False)   # greedyprint(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Use greedy decoding. Sampling introduces column and table names that are not in the schema. The system prompt and user template are in prompt_format.json and were present in every training example.

Training

Table

Base model	`unsloth/Qwen2.5-3B-Instruct` (3.1B parameters)
Method	LoRA at 16-bit via Unsloth
Trainable	29,933,568 parameters (0.97%)
Rank / alpha / dropout	16 / 16 / 0
Target modules	`q,k,v,o,gate,up,down`
Learning rate	0.0002, linear, `adamw_8bit`
Batch	2 × 4 = 8 effective

Why this task worked when a previous attempt did not

An earlier experiment distilled reasoning traces into Llama-3.2-3B and GSM8K accuracy fell from 75.0% to 65.0%. That model had already been post-trained on maths by its authors, leaving little headroom and plenty to disturb.

Fine-tuning pays off where the base model's output does not match the target form. Asked for SQL, an instruct model returns a markdown fence, a preamble and an explanation, and it reaches for columns the schema does not contain. Those are form errors, and they are exactly what supervised fine-tuning fixes — which is why the gain here is +66.6 points rather than a regression.

This model was only published because a gate in the training notebook required at least a +5 point improvement over the baseline before uploading.

Limitations

Single-table, synthetic schemas. sql-create-context pairs one CREATE TABLE with one question. Multi-table joins, real production schemas, CTEs and window functions are out of distribution.
Exact match is strict and also too lenient. It misses semantically equivalent queries written differently, and it credits queries that parse identically but would error against a real database. Nothing here was executed against actual data.
Evaluated on 300 queries, so differences under a couple of points are noise.
No SQL injection or safety review. Do not route model output straight into a database without validation and a read-only connection.
Greedy only. Sampled decoding was not evaluated and hallucinates schema elements.
A 3B model will not match a frontier model on hard SQL.

Reproducing

One Colab notebook on a Tesla T4 produced the baseline, both training stages, all four charts and this card. Raw numbers in results.json.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

emonHasan01

Model Tree

Base

unsloth/Qwen2.5-3B-Instruct

Fine-tuned

this model

Input Modalities

Text

Output Modalities