Sid9797

querycraft-phi3-sql

Deploy Dedicated

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Get help setting up a custom Dedicated Endpoints.

Talk with our engineer to get a quote for reserved GPU instances with discounts.

Evaluation Results

Table with columns: Metric, Base Model, Fine-Tuned
Metric	Base Model	Fine-Tuned
Exact Match	0.0%	82.0%
Execution Accuracy	84.0%	96.0%
BLEU Score	55.79	96.42

Evaluated on 50 held-out validation examples not seen during training.

Model Details

Table with columns: Property, Value
Property	Value
Base model	microsoft/Phi-3-mini-4k-instruct (3.8B params)
Fine-tuning method	QLoRA (4-bit NF4 + LoRA)
LoRA rank	r=16, alpha=32
Trainable parameters	8,912,896 (0.23%)
Training examples	76,577
Training hardware	NVIDIA RTX 5060 Ti 8GB
Training time	3 hours 2 minutes
Final train loss	0.5677
Max sequence length	256 tokens

How to Use

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

base_model = "microsoft/Phi-3-mini-4k-instruct"
adapter    = "Sid9797/querycraft-phi3-sql"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    quantization_config=bnb_config,
    device_map="cuda:0",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
)
model = PeftModel.from_pretrained(model, adapter)
model.eval()

prompt = '''### System:
You are a SQL expert. Given a database schema and a natural language question, generate a valid SQL query that answers the question. Output only the SQL query with no explanation.

### Schema:
CREATE TABLE employees (id INTEGER, name VARCHAR, department VARCHAR, salary FLOAT)

### Question:
What is the average salary by department?

### SQL:
'''

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=128, do_sample=False)

sql = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(sql.strip().split("\n")[0])
# SELECT AVG(salary) FROM employees GROUP BY department

Prompt Format

The model was trained on the Alpaca instruction format: System: You are a SQL expert. Given a database schema and a natural language question, generate a valid SQL query that answers the question. Output only the SQL query with no explanation. Schema: {CREATE TABLE statements} Question: {natural language question} SQL: {model generates SQL here}

Training Details

Dataset: b-mc2/sql-create-context — 78,577 examples with inline CREATE TABLE schemas
Train/Val split: 76,577 train / 2,000 validation (seeded shuffle)
Quantization: 4-bit NF4 with double quantization (bitsandbytes)
LoRA target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Optimizer: AdamW with cosine LR schedule, warmup_ratio=0.05
Effective batch size: 16 (batch_size=4, gradient_accumulation=4)
Packing: Enabled — short examples concatenated to fill 256-token sequences

Why the Base Model Scored 0% Exact Match

The base Phi-3 Mini, without fine-tuning, consistently wrapped SQL output in markdown code fences (```sql ... ```) and appended semicolons. This formatting breaks exact match evaluation even when the SQL logic is correct. Fine-tuning on consistently formatted examples eliminated this entirely.

Limitations

Optimised for single-table and simple multi-table queries
Schema must be provided as CREATE TABLE SQL statements
Best results on English-language questions
May struggle with highly complex nested subqueries

Explore FriendliAI today

Get started Talk to an engineer

Evaluation Results

Table with columns: Metric, Base Model, Fine-Tuned
Metric	Base Model	Fine-Tuned
Exact Match	0.0%	82.0%
Execution Accuracy	84.0%	96.0%
BLEU Score	55.79	96.42

Evaluated on 50 held-out validation examples not seen during training.

Model Details

Table with columns: Property, Value
Property	Value
Base model	microsoft/Phi-3-mini-4k-instruct (3.8B params)
Fine-tuning method	QLoRA (4-bit NF4 + LoRA)
LoRA rank	r=16, alpha=32
Trainable parameters	8,912,896 (0.23%)
Training examples	76,577
Training hardware	NVIDIA RTX 5060 Ti 8GB
Training time	3 hours 2 minutes
Final train loss	0.5677
Max sequence length	256 tokens

How to Use

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

base_model = "microsoft/Phi-3-mini-4k-instruct"
adapter    = "Sid9797/querycraft-phi3-sql"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    quantization_config=bnb_config,
    device_map="cuda:0",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
)
model = PeftModel.from_pretrained(model, adapter)
model.eval()

prompt = '''### System:
You are a SQL expert. Given a database schema and a natural language question, generate a valid SQL query that answers the question. Output only the SQL query with no explanation.

### Schema:
CREATE TABLE employees (id INTEGER, name VARCHAR, department VARCHAR, salary FLOAT)

### Question:
What is the average salary by department?

### SQL:
'''

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=128, do_sample=False)

sql = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(sql.strip().split("\n")[0])
# SELECT AVG(salary) FROM employees GROUP BY department

Prompt Format

Training Details

Dataset: b-mc2/sql-create-context — 78,577 examples with inline CREATE TABLE schemas
Train/Val split: 76,577 train / 2,000 validation (seeded shuffle)
Quantization: 4-bit NF4 with double quantization (bitsandbytes)
LoRA target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Optimizer: AdamW with cosine LR schedule, warmup_ratio=0.05
Effective batch size: 16 (batch_size=4, gradient_accumulation=4)
Packing: Enabled — short examples concatenated to fill 256-token sequences

Why the Base Model Scored 0% Exact Match

Limitations

Optimised for single-table and simple multi-table queries
Schema must be provided as CREATE TABLE SQL statements
Best results on English-language questions
May struggle with highly complex nested subqueries

querycraft-phi3-sql

Get help setting up a custom Dedicated Endpoints.

README

Evaluation Results

Model Details

How to Use

Prompt Format

Training Details

Why the Base Model Scored 0% Exact Match

Limitations

Links

Explore FriendliAI today

README

Evaluation Results

Model Details

How to Use

Prompt Format

Training Details

Why the Base Model Scored 0% Exact Match

Limitations

Links