prasanthg3

text-to-sql-qwen-finetuned

README

License: apache-2.0

Model Details

Table

Base model	Qwen/Qwen3-1.7B
Adapter type	LoRA (r=16, alpha=16, dropout=0.05, all linear projections)
Fine-tuning method	QLoRA (4-bit NF4) via TRL `SFTTrainer` + PEFT
License	Apache 2.0
Language	English
Task	Text-to-SQL generation (question + schema → SQL)

How to use

python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model_path = "Qwen/Qwen3-1.7B"
adapter_path = "prasanthg3/text-to-sql-qwen-finetuned"

tokenizer = AutoTokenizer.from_pretrained(base_model_path)
base_model = AutoModelForCausalLM.from_pretrained(base_model_path, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base_model, adapter_path)
model.eval()

prompt = """<|im_start|>system
You are a Text-to-SQL assistant. Output ONLY a single-line SQL query that answers the question using the given schema.
No explanations, no markdown, no backticks, no preamble.
Rules:
- Use the table name exactly as defined in the schema (often "df").
- Quote identifiers with spaces using double quotes.
- Use single quotes for string literals.
- Refer only to columns present in the schema.
<|im_end|>
<|im_start|>user
Schema:
CREATE TABLE df ("Date" text, "City" text, "Opponent" text, "Results" text, "Type of game" text)

Question:
What type of game was held against France with the results of 3:1?
<|im_end|>
<|im_start|>assistant
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True).split("assistant\n")[-1])

Repo layout

markdown
banking77.ipynb      Abandoned/unrelated exploration (GPT-4o intent classification) — not part of this pipeline
fine-tuning.ipynb     QLoRA fine-tuning of Qwen3-1.7B on the Text-to-SQL dataset
eval.ipynb            Baseline (GPT-4o few-shot) + fine-tuned model evaluation, exact-match and LLM-judge
test.py               Downloads the Qwen3-1.7B base model into ./Qwen_model
data/                 train/valid/test parquet splits + saved prediction/judge CSVs
Qwen_model/           Base model weights (downloaded via test.py, not fine-tuned)
trainer_output/       Raw training checkpoints (checkpoint-100 … checkpoint-500)
models/checkpoint-500-best/   Final selected adapter + tokenizer (best eval_loss)

Problem

Generate a correct SQL query from a natural-language question and a CREATE TABLE schema, e.g.:

markdown
Question: What type of game was held against France with the results of 3:1?
Schema:   CREATE TABLE df ("Date" text, "City" text, "Opponent" text, "Results¹" text, "Type of game" text)
SQL:      SELECT "Type of game" FROM df WHERE "Results¹" = '3:1' AND "Opponent" = 'france'

Dataset

5,700 natural-language → SQL pairs pooled from ~19 public Text-to-SQL sources (WikiSQL, sql_create_context, Spider, Squall, NVBench, oxenai, sede, criteria2sql, MIMIC-SQL, eICU, ATIS, Advising, Scholar, and others), split into:

Table with columns: Split, Rows
Split	Rows
`train.parquet`	5,000
`valid.parquet`	200
`test.parquet`	500

Approach

Base model: Qwen3-1.7B, loaded in 4-bit NF4 (QLoRA) with bfloat16 compute dtype
Fine-tuning method: LoRA, r=16, alpha=16, dropout=0.05, applied to all linear projections (q/k/v/o_proj, gate/up/down_proj)
Framework: Hugging Face TRL SFTTrainer + PEFT
Prompt format: ChatML, ANSI-SQL system instructions + schema + question → single-line SQL
Training config: 1 epoch target (1,250 steps), effective batch size 4 (per_device_train_batch_size=1 × gradient_accumulation_steps=4), LR 1e-4 constant, gradient checkpointing, load_best_model_at_end on eval_loss
Hardware: 1× NVIDIA GeForce RTX 3070 Ti (8 GB VRAM)
Actual run: stopped at step 500/1,250 (~40% of one epoch, ~2,000 training examples seen), ~30 minutes wall clock — the selected checkpoint is the best of a partial run, not a completed one

Results

Evaluated on the 200-example validation set, two ways: strict exact match (normalized SQL string equality) and LLM-judge (GPT-4o rates semantic equivalence — tolerant of aliasing, whitespace, column order, etc.).

Table with columns: Model, Exact match, LLM-judge (semantic)
Model	Exact match	LLM-judge (semantic)
GPT-4o, 5-shot prompting	30.5% (61/200)	47.5% (95/200)
Qwen3-1.7B, QLoRA fine-tuned	42.5% (85/200)	58.0% (116/200)

The fine-tuned 1.7B model beats the GPT-4o few-shot baseline by both metrics, despite being ~1,000× smaller and trained for well under an hour on a single desktop GPU.

Raw predictions and judge outputs: data/valid_with_gpt4o_predictions.csv, data/valid_with_predictions.csv, data/gpt_judge_results.csv, data/ft_judge_results.csv.

Limitations

Evaluated on only 200 validation examples — confidence intervals are wide.
Exact-match is a harsh lower bound (penalizes semantically-identical queries with different formatting); LLM-judge is a closer proxy for real usability but is itself an LLM call and not ground truth.
The fine-tuning run was manually stopped 40% into one epoch; a completed run may perform differently (better or worse, depending on overfitting).
The GPT-4o baseline is prompted, not fine-tuned — the comparison is "fine-tune a small model" vs. "prompt a large one," not fine-tuned-vs-fine-tuned.

Reproducing

pip install -r requirements.txt
Add Azure OpenAI credentials to .env (AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, etc. — see langchain_openai.AzureChatOpenAI usage in the notebooks)
python test.py to download the Qwen3-1.7B base model into ./Qwen_model
Run fine-tuning.ipynb to train and save the adapter
Run eval.ipynb to reproduce the baseline and fine-tuned evaluation numbers

See MODEL_DOCUMENTATION.md for the full write-up.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

prasanthg3

Model Tree

Base

Qwen/Qwen3-1.7B

Adapter

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Table

Base model	Qwen/Qwen3-1.7B
Adapter type	LoRA (r=16, alpha=16, dropout=0.05, all linear projections)
Fine-tuning method	QLoRA (4-bit NF4) via TRL `SFTTrainer` + PEFT
License	Apache 2.0
Language	English
Task	Text-to-SQL generation (question + schema → SQL)

How to use

python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model_path = "Qwen/Qwen3-1.7B"
adapter_path = "prasanthg3/text-to-sql-qwen-finetuned"

tokenizer = AutoTokenizer.from_pretrained(base_model_path)
base_model = AutoModelForCausalLM.from_pretrained(base_model_path, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base_model, adapter_path)
model.eval()

prompt = """<|im_start|>system
You are a Text-to-SQL assistant. Output ONLY a single-line SQL query that answers the question using the given schema.
No explanations, no markdown, no backticks, no preamble.
Rules:
- Use the table name exactly as defined in the schema (often "df").
- Quote identifiers with spaces using double quotes.
- Use single quotes for string literals.
- Refer only to columns present in the schema.
<|im_end|>
<|im_start|>user
Schema:
CREATE TABLE df ("Date" text, "City" text, "Opponent" text, "Results" text, "Type of game" text)

Question:
What type of game was held against France with the results of 3:1?
<|im_end|>
<|im_start|>assistant
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True).split("assistant\n")[-1])

Repo layout

markdown
banking77.ipynb      Abandoned/unrelated exploration (GPT-4o intent classification) — not part of this pipeline
fine-tuning.ipynb     QLoRA fine-tuning of Qwen3-1.7B on the Text-to-SQL dataset
eval.ipynb            Baseline (GPT-4o few-shot) + fine-tuned model evaluation, exact-match and LLM-judge
test.py               Downloads the Qwen3-1.7B base model into ./Qwen_model
data/                 train/valid/test parquet splits + saved prediction/judge CSVs
Qwen_model/           Base model weights (downloaded via test.py, not fine-tuned)
trainer_output/       Raw training checkpoints (checkpoint-100 … checkpoint-500)
models/checkpoint-500-best/   Final selected adapter + tokenizer (best eval_loss)

Problem

Generate a correct SQL query from a natural-language question and a CREATE TABLE schema, e.g.:

markdown
Question: What type of game was held against France with the results of 3:1?
Schema:   CREATE TABLE df ("Date" text, "City" text, "Opponent" text, "Results¹" text, "Type of game" text)
SQL:      SELECT "Type of game" FROM df WHERE "Results¹" = '3:1' AND "Opponent" = 'france'

Dataset

Table with columns: Split, Rows
Split	Rows
`train.parquet`	5,000
`valid.parquet`	200
`test.parquet`	500

Approach

Base model: Qwen3-1.7B, loaded in 4-bit NF4 (QLoRA) with bfloat16 compute dtype
Fine-tuning method: LoRA, r=16, alpha=16, dropout=0.05, applied to all linear projections (q/k/v/o_proj, gate/up/down_proj)
Framework: Hugging Face TRL SFTTrainer + PEFT
Prompt format: ChatML, ANSI-SQL system instructions + schema + question → single-line SQL
Training config: 1 epoch target (1,250 steps), effective batch size 4 (per_device_train_batch_size=1 × gradient_accumulation_steps=4), LR 1e-4 constant, gradient checkpointing, load_best_model_at_end on eval_loss
Hardware: 1× NVIDIA GeForce RTX 3070 Ti (8 GB VRAM)
Actual run: stopped at step 500/1,250 (~40% of one epoch, ~2,000 training examples seen), ~30 minutes wall clock — the selected checkpoint is the best of a partial run, not a completed one

Results

Table with columns: Model, Exact match, LLM-judge (semantic)
Model	Exact match	LLM-judge (semantic)
GPT-4o, 5-shot prompting	30.5% (61/200)	47.5% (95/200)
Qwen3-1.7B, QLoRA fine-tuned	42.5% (85/200)	58.0% (116/200)

The fine-tuned 1.7B model beats the GPT-4o few-shot baseline by both metrics, despite being ~1,000× smaller and trained for well under an hour on a single desktop GPU.

Raw predictions and judge outputs: data/valid_with_gpt4o_predictions.csv, data/valid_with_predictions.csv, data/gpt_judge_results.csv, data/ft_judge_results.csv.

Limitations

Evaluated on only 200 validation examples — confidence intervals are wide.
Exact-match is a harsh lower bound (penalizes semantically-identical queries with different formatting); LLM-judge is a closer proxy for real usability but is itself an LLM call and not ground truth.
The fine-tuning run was manually stopped 40% into one epoch; a completed run may perform differently (better or worse, depending on overfitting).
The GPT-4o baseline is prompted, not fine-tuned — the comparison is "fine-tune a small model" vs. "prompt a large one," not fine-tuned-vs-fine-tuned.

Reproducing

pip install -r requirements.txt
Add Azure OpenAI credentials to .env (AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, etc. — see langchain_openai.AzureChatOpenAI usage in the notebooks)
python test.py to download the Qwen3-1.7B base model into ./Qwen_model
Run fine-tuning.ipynb to train and save the adapter
Run eval.ipynb to reproduce the baseline and fine-tuned evaluation numbers

See MODEL_DOCUMENTATION.md for the full write-up.