Official BIRD results (hidden test set)
The BIRD team scored the model on the hidden test set: 32 candidates per question, execution repair and self-judge, on a single A100 80 GB.
Table with columns: metric, total (1,789)| metric | total (1,789) |
|---|
| Execution Accuracy (EX) | 75.68 (simple 82.93 / moderate 72.79 / challenging 57.19) |
| Soft F1 | 76.86 |
| R-VES | 68.10 |
How it compares
On the single-model leaderboard it shares 8th place with Databricks RLVR 32B and is ahead of 57 of the 65 other entries. On the overall leaderboard, which includes multi-model pipelines and proprietary systems, it is ahead of 105 entries. Among entries with 4B parameters or fewer, only Native mini (4B) scores higher.
All scores are BIRD test-set execution accuracy from the single-model leaderboard at bird-bench.github.io, taken at the time of writing. Markers show how many candidates each entry samples per question, using the leaderboard's own categories: no marker = one candidate, * Few = 2–7, ** Many = 8–32.
Frontier and large general-purpose LLMs:
Table with columns: model, size, test EX| model | size | test EX |
|---|
| flash-sqlite-4b (this model) | 4B | 75.68** |
| GPT-5.5-xhigh | proprietary | 72.55 |
| Claude Opus 4.6 | proprietary | 70.15 |
| Qwen3-Coder-480B-A35B | 480B MoE | 68.14 |
| Claude 4.5 Sonnet | proprietary | 66.85 |
| GLM-4.7 | large open-weight | 62.94 |
| DeepSeek-R1 | 671B MoE | 60.93 |
| Kimi-K2-Thinking | ~1T MoE | 59.87 |
Specialised text-to-SQL models (selection):
Table with columns: model, size, test EX| model | size | test EX |
|---|
| SiriusAI-SQL-9B (Tencent) | 9B | 77.08** |
| Native mini | 4B | 76.86** |
| flash-sqlite-4b (this model) | 4B | 75.68** |
| Databricks RLVR 32B | 32B | 75.68* |
| Sophon-Text2SQL-32B (ByteDance) | 32B | 74.79** |
| Arctic-Text2SQL-R1-32B (Snowflake) | 32B | 73.84* |
| flash-sqlite (our 2B) | 2B | 73.73** |
| Arctic-Text2SQL-R1-7B (Snowflake) | 7B | 70.43* |
| XiYanSQL-QwenCoder-32B (Alibaba) | 32B | 69.03* |
BIRD dev results
Dev revision bird-sql-dev-1106: 1,532 questions with executable gold SQL, execution accuracy with set comparison. All rows come from the same run of the inference package.
Table with columns: inference, EX, simple, moderate, challenging| inference | EX | simple | moderate | challenging |
|---|
| 32 candidates + repair + self-judge | 71.41 | 80.79 | 70.88 | 37.39 |
| 16 candidates + repair + self-judge | 71.15 | 80.44 | 70.43 | 37.83 |
| majority vote over 32 repaired candidates, no judge | 69.13 | 78.46 | 67.49 | 37.39 |
Without the repair step, the same model reaches 70.54 with 32 candidates + self-judge and 68.32 with plain majority voting. Repair adds about 0.6–0.9 EX on dev; the self-judge adds about 2.3 EX over majority voting.
How it works
For each question:
- Generate 32 candidates at temperature 1.0, top_p 0.95, top_k 20, with a 2,048-token limit and non-thinking mode.
- Repair: every distinct candidate runs read-only against the database (30 s timeout).
- If it fails (SQL error, timeout, no SQL found) or returns an empty or all-NULL result, the model continues the chat with its own answer and the execution feedback: the error message, or the row count, column names and first 10 rows, plus a fixed instruction to check the result and either repeat the SQL or fix it.
- Repair is greedy, at most 2 rounds per candidate. It stops early when the model repeats its SQL or the new SQL returns a non-empty result.
- Group the repaired candidates by their result set, compared as sets with
1 == 1.0. Candidates that still fail are dropped. A group's votes are the number of candidates in it.
- Self-judge, only if more than one result group remains: the same model gets one prompt per group with the schema and question, the group's SQL and a 5-row preview of its result. P(Yes) is read from the first generated token, restricted to Yes / No and renormalised.
- Select the group with the highest votes × P(Yes).
On BIRD dev this costs on average 39.1 model calls per question: 32 generations, 4.7 repair calls (only 592 of 1,534 questions needed any repair; identical candidates share one repair) and 2.4 judge calls (54 % of questions needed no judge call because all executable candidates agreed).
Speed and efficiency
- One GPU, no API: the bf16 weights need about 9 GB. Generation, execution, repair and judging all run locally; no data leaves the machine.
- Batched sampling: the 32 samples per question come from one request (
n=32); with prefix caching, the long schema prompt is prefilled once and shared by all samples and by the judge prompts of the same question.
- End to end: all 1,534 BIRD dev questions take about 4 h 15 min on one 96 GB workstation GPU: about 3 h to generate 49k candidates, about 1 h 10 min for the repair rounds, and a few minutes to execute and judge.
Training
Training is supervised fine-tuning only. Every SQL target is execution-verified: a chain-of-thought trace was only kept if its final SQL, run on the real database, returned the gold result set. The traces were rejection-sampled from a Qwen 27B teacher and follow a structured chain of thought: question and hint mapping, schema selection, join path, filters, aggregation, edge-case checks, then the final SQL.
Stage 1: structured-CoT SFT from Qwen3.5-4B on 20,940 traces: BIRD train 6,779, Spider 4,046 and 10,115 synthetic examples over generated databases (schema-linking traps, expert-style, noisy-hint and formula questions). The best checkpoint (epoch 1.75) was kept.
Stage 2: joint generator + judge SFT on 40,452 rows:
- SQL generation traces (BIRD train, Spider and synthetic databases), with per-column descriptions in the prompt;
- pointwise critiques: each judges a single candidate query and its execution result and ends in a Yes/No verdict. The teacher wrote them without seeing the label; only critiques whose verdict matched the execution-based label were kept;
- listwise judge rows: several candidates (grouped by execution result) are compared, then the model chooses one or answers "none".
Hyperparameters for stage 2: full fine-tune in bf16, learning rate 2e-5 (cosine schedule), 50 warmup steps, effective batch size 32, NEFTune α = 5, maximum sequence length 16,384. The released checkpoint is step 3,476 (about 2.75 of 3 planned epochs).
The prompts contain per-column descriptions read from each BIRD database's database_description/*.csv files. No BIRD dev or test data was used for training.
Usage
The model expects the OmniSQL-style prompt below and answers with its reasoning followed by a fenced SQL block. Always take the last ```sql block. Use non-thinking mode (enable_thinking=False).
Task Overview:You are a data science expert. Below, you are provided with a database schema and a natural language question. Your task is to understand the schema and generate a valid SQL query to answer the question. Database Engine:SQLite Database Schema:{db_details}This schema describes the database's structure, including tables, columns, primary keys, foreign keys, and any relevant relationships or constraints. Question:{question} Hint: {evidence} Instructions:- Make sure you only output the information that is asked in the question. If the question asks for a specific column, make sure to only include that column in the SELECT clause, nothing more.- The generated query should return all of the information asked in the question without any missing or extra information.- Before generating the final SQL query, please think through the steps of how to write the query. Output Format:In your answer, please enclose the generated SQL query in a code block:```sql-- Your SQL query
Take a deep breath and think step by step to find the correct SQL query.
`{db_details}` is the schema as `CREATE TABLE` statements, with the column descriptions as comments and three example rows per table. ### Quick start (vLLM) ```pythonfrom vllm import LLM, SamplingParams llm = LLM("AlioLeuchtmann/flash-sqlite-4b", max_model_len=32768)tok = llm.get_tokenizer()prompt = tok.apply_chat_template([{"role": "user", "content": PROMPT}], tokenize=False, add_generation_prompt=True, enable_thinking=False) # greedy single answerout = llm.generate([prompt], SamplingParams(temperature=0.0, max_tokens=2048))sql = out[0].outputs[0].text.split("```sql")[-1].split("```")[0].strip() # 32 candidates for repair + voting + self-judgecands = llm.generate([prompt], SamplingParams(n=32, temperature=1.0, top_p=0.95, top_k=20, max_tokens=2048))
vLLM ≥ 0.25 is required for the Qwen3.5 hybrid architecture.
Limitations
- Trained and evaluated on SQLite only; other dialects (PostgreSQL, BigQuery, Snowflake) are not covered.
- The schema prompt must fit the context window. Very large schemas need truncation or schema filtering.
- A sizeable share of the remaining dev errors comes from ambiguous or incorrect gold SQL in BIRD itself, so dev EX understates practical accuracy on clean questions, and gains from further tuning on dev are noisy.
License
Apache 2.0, the same as the base model Qwen/Qwen3.5-4B.