AlioLeuchtmann

flash-sqlite

README

License: apache-2.0

Official BIRD results (hidden test set)

The BIRD team scored the model on the hidden test set: 32 candidates per question plus self-judge, on a single A100 80 GB.

Table with columns: metric, simple (949), moderate (555), challenging (285), total (1,789)
metric	simple (949)	moderate (555)	challenging (285)	total (1,789)
Execution Accuracy (EX)	80.19	71.53	56.49	73.73
Soft F1	80.50	72.88	58.49	74.63
R-VES	73.54	64.29	51.55	67.17

How it compares

At 2B parameters, flash-sqlite scores higher on the BIRD test set than every frontier LLM baseline on the leaderboard:

GPT-5.5, Claude Opus 4.6, Claude 4.5 Sonnet, Qwen3-Coder-480B, GLM-4.7, DeepSeek-R1 and Kimi-K2-Thinking;
and specialised open models up to 32B.

On the single-model leaderboard it is ahead of 50 of the 66 other entries. On the overall leaderboard, which includes multi-model pipelines and proprietary systems, it is ahead of more than 90.

All scores are BIRD test-set execution accuracy. Markers show how many candidates each entry samples per question, using the leaderboard's own categories:

no marker: one candidate;
* Few: 2–7 candidates;
** Many: 8–32 candidates.

Frontier and large general-purpose LLMs (single-model leaderboard):

flash-sqlite vs frontier LLM baselines on the BIRD test set

Table with columns: model, provider, size, test EX
model	provider	size	test EX
flash-sqlite (this model)	—	2B	73.73**
GPT-5.5-xhigh	OpenAI	proprietary	72.55
Claude Opus 4.6	Anthropic	proprietary	70.15
Qwen3-Coder-480B-A35B	Alibaba	480B MoE	68.14

Accuracy per parameter. The chart shows every single-model entry with a published size. flash-sqlite lies on the Pareto frontier, so no smaller entry scores higher. It is the highest-scoring model at 2B parameters or fewer.

BIRD test EX by model size, single-model leaderboard

Specialised text-to-SQL models (single-model leaderboard, selection):

Table with columns: model, size, test EX
model	size	test EX
Native mini	4B	76.86**
Arctic-Text2SQL-R1-32B (Snowflake)	32B	73.84*
flash-sqlite (this model)	2B	73.73**
Databricks RLVR 32B	32B	73.56
Arctic-Text2SQL-R1-14B (Snowflake)	14B	72.22*
Arctic-Text2SQL-R1-7B (Snowflake)

flash-sqlite samples 32 candidates and selects one with its own judge (see How it works). All figures are from the public leaderboard at bird-bench.github.io, taken at the time of writing. The earlier version of this model (2B, majority vote over 7 candidates) scored 68.47*.

Speed and efficiency

32 candidates plus a judge sounds expensive. With a 2B model and the right serving setup, it is not.

One GPU, no API: the bf16 weights need about 4.5 GB. Everything runs locally on a single GPU: generation, execution and judging. No data leaves the machine.
32 candidates in parallel:
- The 32 samples per question come from one request (n=32), and vLLM batches them together.
- With prefix caching, the long schema prompt is prefilled once and shared by all 32 samples.
- The judge prompts of the same question also share their schema prefix.
Throughput: about 6,400–7,600 output tokens/s on one 96 GB workstation GPU, which is 8–11 finished candidates per second.
The judge is almost free:
- It reads one token (P(Yes)) per result group instead of generating text.
- On BIRD dev, 38% of questions needed no judge call at all, because all executable candidates agreed.
- On average there were 2.45 judge calls per question (3,763 calls for 949 contested questions).
- The whole judge pass over BIRD dev took .

BIRD dev results

These come from the dev revision bird-sql-dev-1106: 1,532 questions with executable gold SQL, scored by execution accuracy with set comparison. The three candidate-based rows come from the same generation run; the greedy row comes from a separate single-answer run.

Table with columns: inference, EX, simple, moderate, challenging
inference	EX	simple	moderate	challenging
32 candidates + self-judge	67.49	78.11	63.66	35.22
16 candidates + self-judge	66.91	77.65	63.21	33.91
majority vote over 32, no judge	62.92	74.27	56.88	32.17

On dev, the self-judge adds 4.6 EX over plain majority voting with the same 32 candidates. Its gain grows with the number of candidates.

How it works

For each question:

Generate 32 candidates at temperature 1.0, top_p 0.95, top_k 20, with a 2,048-token limit and non-thinking mode.
Execute each candidate read-only against the database, with a 30 s timeout.
- Candidates are grouped by their result set, compared as sets with 1 == 1.0.
- Candidates that fail to run are dropped.
- A group's votes are the number of candidates in it.
Self-judge — only if more than one result group remains.
- The same model gets one prompt per group: the schema and question, the group's SQL, and a 5-row preview of its result.
- P(Yes) is read from the first generated token, restricted to Yes / No and renormalised.
Select the group with the highest votes × P(Yes).

Training

Training is supervised fine-tuning only: no reinforcement learning, no preference optimisation and no reward model. Every SQL target is execution-verified. A chain-of-thought trace was only kept if its final SQL, run on the real database, returned the gold result set.

Stage 1: synthetic pre-training.

Data: 57,178 execution-verified synthetic examples, over generated databases and questions designed to match BIRD's style.
Training: 2 epochs.

Stage 2: joint generator + judge SFT. 27,179 rows in total:

14,327 SQL generation traces (BIRD train and synthetic databases):
- The traces follow a structured chain of thought: question and hint mapping, schema selection, join path, filters, aggregation, edge-case checks, then the final SQL.
- They were rejection-sampled from a Qwen 27B teacher.
8,314 pointwise critiques:
- Each one judges a single candidate query and its execution result, and ends in a Yes/No verdict.
- The teacher wrote them without seeing the label. Only critiques whose verdict matched the execution-based label were kept.
4,538 listwise judge rows:
- Several candidates (grouped by execution result) are compared, then the model chooses one or answers "none".

Hyperparameters for stage 2:

Full fine-tune in bf16, 3 epochs.
Learning rate 2e-5 (cosine schedule), 50 warmup steps.
Effective batch size 32.
NEFTune α = 5.
Maximum sequence length 16,384.

The prompts contain per-column descriptions, read from each BIRD database's database_description/*.csv files.

Usage

The model expects the OmniSQL-style prompt shown below and answers with its reasoning followed by a fenced SQL block. Always take the last ```sql block. Use non-thinking mode (enable_thinking=False).

Prompt format

text

Task Overview:You are a data science expert. Below, you are provided with a database schema and a natural language question. Your task is to understand the schema and generate a valid SQL query to answer the question.Database Engine:SQLiteDatabase Schema:{db_details}This schema describes the database's structure, including tables, columns, primary keys, foreign keys, and any relevant relationships or constraints.Question:{question}Hint: {evidence}Instructions:- Make sure you only output the information that is asked in the question. If the question asks for a specific column, make sure to only include that column in the SELECT clause, nothing more.- The generated query should return all of the information asked in the question without any missing or extra information.- Before generating the final SQL query, please think through the steps of how to write the query.Output Format:In your answer, please enclose the generated SQL query in a code block:```sql-- Your SQL query

Take a deep breath and think step by step to find the correct SQL query.

text

`{db_details}` contains, for each table:- the `CREATE TABLE` statement;- a comment block of column descriptions (where the database ships them);- a comment block with 3 example rows.### Generation (vLLM)```pythonimport refrom transformers import AutoTokenizerfrom vllm import LLM, SamplingParamsmodel_id = "AlioLeuchtmann/flash-sqlite"tok = AutoTokenizer.from_pretrained(model_id)llm = LLM(model=model_id, dtype="bfloat16")def chat(prompt):    return tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False,                                   add_generation_prompt=True, enable_thinking=False)def last_sql(text):    blocks = re.findall(r"```sql\s*(.*?)```", text, re.S)    return blocks[-1].strip() if blocks else ""prompt = ...  # the prompt above, filled with the schema, question and hint# a single answergreedy = llm.generate([chat(prompt)], SamplingParams(temperature=0.0, max_tokens=2048))print(last_sql(greedy[0].outputs[0].text))# 32 candidates for voting + self-judgesamples = llm.generate([chat(prompt)], SamplingParams(n=32, temperature=1.0, top_p=0.95, top_k=20, max_tokens=2048))candidates = [last_sql(o.text) for o in samples[0].outputs]

Self-judge

Execute the candidates and group them by result set. Then score one representative query per group with the judge prompt below. Use max_tokens=1, allowed_token_ids=[id("Yes"), id("No")] and logprobs=2, and renormalise P(Yes) over the two tokens.

text

Task Overview:You are a data science expert. Below, you are provided with a database schema, a natural language question, a candidate SQL query and the result of executing it. Your task is to decide whether the candidate SQL query answers the question correctly.Database Engine:SQLite{the "Database Schema:" ... "Question:" part of the generation prompt}Candidate SQL:```sql{sql}

Execution result ({nrows} rows, first 5 shown): {preview}

Does the candidate SQL query answer the question correctly, returning exactly the requested information? Answer with one word: Yes or No.

text

Pick the group with the highest `votes × P(Yes)`. If two groups tie, prefer the one with more votes, then the one that appeared first.### Tested environment`vllm==0.25.1` and `transformers==5.14.1`, which support the Qwen3.5 hybrid architecture. A 2B model in bf16 needs about 4.5 GB of GPU memory for the weights. All 1,534 BIRD dev questions (49k samples plus judging) take about 95 minutes on a single 96 GB workstation GPU.## Intended use and limitations- **Built for:** SQLite text-to-SQL on analytical questions in the style of BIRD, in English, optionally with an evidence hint. It is not a general chat model.- **The judge needs execution:** it scores a query *together with its result*, so the database has to be available at inference time. It has not been calibrated as a standalone classifier. Use it in the votes × P(Yes) combination it was tuned for.- **Dialects:** trained and evaluated only on SQLite. Other dialects, such as PostgreSQL or BigQuery, will need adaptation.- **Hard questions:** accuracy on the challenging tier (56.5 EX on test) is still well below the simple tier. Multi-step queries remain the main weakness of a 2B model.- **Safety:** generated SQL should run read-only. This model was never trained to avoid writing or destructive statements.## AcknowledgementsBuilt on Qwen3.5-2B by the Qwen team. Evaluated by the BIRD team ([bird-bench.github.io](https://bird-bench.github.io)). Developed by Alio Leuchtmann.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

AlioLeuchtmann

Model Tree

Base

Qwen/Qwen3.5-2B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Official BIRD results (hidden test set)

The BIRD team scored the model on the hidden test set: 32 candidates per question plus self-judge, on a single A100 80 GB.

Table with columns: metric, simple (949), moderate (555), challenging (285), total (1,789)
metric	simple (949)	moderate (555)	challenging (285)	total (1,789)
Execution Accuracy (EX)	80.19	71.53	56.49	73.73
Soft F1	80.50	72.88	58.49	74.63
R-VES	73.54	64.29	51.55	67.17

How it compares

At 2B parameters, flash-sqlite scores higher on the BIRD test set than every frontier LLM baseline on the leaderboard:

GPT-5.5, Claude Opus 4.6, Claude 4.5 Sonnet, Qwen3-Coder-480B, GLM-4.7, DeepSeek-R1 and Kimi-K2-Thinking;
and specialised open models up to 32B.

On the single-model leaderboard it is ahead of 50 of the 66 other entries. On the overall leaderboard, which includes multi-model pipelines and proprietary systems, it is ahead of more than 90.

All scores are BIRD test-set execution accuracy. Markers show how many candidates each entry samples per question, using the leaderboard's own categories:

no marker: one candidate;
* Few: 2–7 candidates;
** Many: 8–32 candidates.

Frontier and large general-purpose LLMs (single-model leaderboard):

flash-sqlite vs frontier LLM baselines on the BIRD test set

Table with columns: model, provider, size, test EX
model	provider	size	test EX
flash-sqlite (this model)	—	2B	73.73**
GPT-5.5-xhigh	OpenAI	proprietary	72.55
Claude Opus 4.6	Anthropic	proprietary	70.15
Qwen3-Coder-480B-A35B	Alibaba	480B MoE	68.14

BIRD test EX by model size, single-model leaderboard

Specialised text-to-SQL models (single-model leaderboard, selection):

Table with columns: model, size, test EX
model	size	test EX
Native mini	4B	76.86**
Arctic-Text2SQL-R1-32B (Snowflake)	32B	73.84*
flash-sqlite (this model)	2B	73.73**
Databricks RLVR 32B	32B	73.56
Arctic-Text2SQL-R1-14B (Snowflake)	14B	72.22*
Arctic-Text2SQL-R1-7B (Snowflake)

Speed and efficiency

32 candidates plus a judge sounds expensive. With a 2B model and the right serving setup, it is not.

One GPU, no API: the bf16 weights need about 4.5 GB. Everything runs locally on a single GPU: generation, execution and judging. No data leaves the machine.
32 candidates in parallel:
- The 32 samples per question come from one request (n=32), and vLLM batches them together.
- With prefix caching, the long schema prompt is prefilled once and shared by all 32 samples.
- The judge prompts of the same question also share their schema prefix.
Throughput: about 6,400–7,600 output tokens/s on one 96 GB workstation GPU, which is 8–11 finished candidates per second.
The judge is almost free:
- It reads one token (P(Yes)) per result group instead of generating text.
- On BIRD dev, 38% of questions needed no judge call at all, because all executable candidates agreed.
- On average there were 2.45 judge calls per question (3,763 calls for 949 contested questions).
- The whole judge pass over BIRD dev took .

BIRD dev results

Table with columns: inference, EX, simple, moderate, challenging
inference	EX	simple	moderate	challenging
32 candidates + self-judge	67.49	78.11	63.66	35.22
16 candidates + self-judge	66.91	77.65	63.21	33.91
majority vote over 32, no judge	62.92	74.27	56.88	32.17

On dev, the self-judge adds 4.6 EX over plain majority voting with the same 32 candidates. Its gain grows with the number of candidates.

How it works

For each question:

Generate 32 candidates at temperature 1.0, top_p 0.95, top_k 20, with a 2,048-token limit and non-thinking mode.
Execute each candidate read-only against the database, with a 30 s timeout.
- Candidates are grouped by their result set, compared as sets with 1 == 1.0.
- Candidates that fail to run are dropped.
- A group's votes are the number of candidates in it.
Self-judge — only if more than one result group remains.
- The same model gets one prompt per group: the schema and question, the group's SQL, and a 5-row preview of its result.
- P(Yes) is read from the first generated token, restricted to Yes / No and renormalised.
Select the group with the highest votes × P(Yes).

Training

Stage 1: synthetic pre-training.

Data: 57,178 execution-verified synthetic examples, over generated databases and questions designed to match BIRD's style.
Training: 2 epochs.

Stage 2: joint generator + judge SFT. 27,179 rows in total:

14,327 SQL generation traces (BIRD train and synthetic databases):
- The traces follow a structured chain of thought: question and hint mapping, schema selection, join path, filters, aggregation, edge-case checks, then the final SQL.
- They were rejection-sampled from a Qwen 27B teacher.
8,314 pointwise critiques:
- Each one judges a single candidate query and its execution result, and ends in a Yes/No verdict.
- The teacher wrote them without seeing the label. Only critiques whose verdict matched the execution-based label were kept.
4,538 listwise judge rows:
- Several candidates (grouped by execution result) are compared, then the model chooses one or answers "none".

Hyperparameters for stage 2:

Full fine-tune in bf16, 3 epochs.
Learning rate 2e-5 (cosine schedule), 50 warmup steps.
Effective batch size 32.
NEFTune α = 5.
Maximum sequence length 16,384.

The prompts contain per-column descriptions, read from each BIRD database's database_description/*.csv files.

Usage

Prompt format

text

Task Overview:You are a data science expert. Below, you are provided with a database schema and a natural language question. Your task is to understand the schema and generate a valid SQL query to answer the question.Database Engine:SQLiteDatabase Schema:{db_details}This schema describes the database's structure, including tables, columns, primary keys, foreign keys, and any relevant relationships or constraints.Question:{question}Hint: {evidence}Instructions:- Make sure you only output the information that is asked in the question. If the question asks for a specific column, make sure to only include that column in the SELECT clause, nothing more.- The generated query should return all of the information asked in the question without any missing or extra information.- Before generating the final SQL query, please think through the steps of how to write the query.Output Format:In your answer, please enclose the generated SQL query in a code block:```sql-- Your SQL query

Take a deep breath and think step by step to find the correct SQL query.

text

`{db_details}` contains, for each table:- the `CREATE TABLE` statement;- a comment block of column descriptions (where the database ships them);- a comment block with 3 example rows.### Generation (vLLM)```pythonimport refrom transformers import AutoTokenizerfrom vllm import LLM, SamplingParamsmodel_id = "AlioLeuchtmann/flash-sqlite"tok = AutoTokenizer.from_pretrained(model_id)llm = LLM(model=model_id, dtype="bfloat16")def chat(prompt):    return tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False,                                   add_generation_prompt=True, enable_thinking=False)def last_sql(text):    blocks = re.findall(r"```sql\s*(.*?)```", text, re.S)    return blocks[-1].strip() if blocks else ""prompt = ...  # the prompt above, filled with the schema, question and hint# a single answergreedy = llm.generate([chat(prompt)], SamplingParams(temperature=0.0, max_tokens=2048))print(last_sql(greedy[0].outputs[0].text))# 32 candidates for voting + self-judgesamples = llm.generate([chat(prompt)], SamplingParams(n=32, temperature=1.0, top_p=0.95, top_k=20, max_tokens=2048))candidates = [last_sql(o.text) for o in samples[0].outputs]

Self-judge

text

Task Overview:You are a data science expert. Below, you are provided with a database schema, a natural language question, a candidate SQL query and the result of executing it. Your task is to decide whether the candidate SQL query answers the question correctly.Database Engine:SQLite{the "Database Schema:" ... "Question:" part of the generation prompt}Candidate SQL:```sql{sql}

Execution result ({nrows} rows, first 5 shown): {preview}

Does the candidate SQL query answer the question correctly, returning exactly the requested information? Answer with one word: Yes or No.

text

Pick the group with the highest `votes × P(Yes)`. If two groups tie, prefer the one with more votes, then the one that appeared first.### Tested environment`vllm==0.25.1` and `transformers==5.14.1`, which support the Qwen3.5 hybrid architecture. A 2B model in bf16 needs about 4.5 GB of GPU memory for the weights. All 1,534 BIRD dev questions (49k samples plus judging) take about 95 minutes on a single 96 GB workstation GPU.## Intended use and limitations- **Built for:** SQLite text-to-SQL on analytical questions in the style of BIRD, in English, optionally with an evidence hint. It is not a general chat model.- **The judge needs execution:** it scores a query *together with its result*, so the database has to be available at inference time. It has not been calibrated as a standalone classifier. Use it in the votes × P(Yes) combination it was tuned for.- **Dialects:** trained and evaluated only on SQLite. Other dialects, such as PostgreSQL or BigQuery, will need adaptation.- **Hard questions:** accuracy on the challenging tier (56.5 EX on test) is still well below the simple tier. Multi-step queries remain the main weakness of a 2B model.- **Safety:** generated SQL should run read-only. This model was never trained to avoid writing or destructive statements.## AcknowledgementsBuilt on Qwen3.5-2B by the Qwen team. Evaluated by the BIRD team ([bird-bench.github.io](https://bird-bench.github.io)). Developed by Alio Leuchtmann.