RakhatM

qwen3-1.7b-kazakh-text-to-sql-grpo

README

License: apache-2.0

Usage

The system message holds the instructions and the schema text (the CREATE TABLE statements with comments that list the exact spellings of string values); the user message is the question. Generate with enable_thinking=False, which reproduces the empty think block the model was trained with. The model writes one SQL query and ends with <|endoftext|>, where the bundled generation config stops.

python

from huggingface_hub import hf_hub_downloadfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "RakhatM/qwen3-1.7b-kazakh-text-to-sql-grpo"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto") # The schema text of the database (here the Qazaqstan one from the dataset repo)schema = open(hf_hub_download("RakhatM/kazakh-text-to-sql", "schema.sql", repo_type="dataset")).read().strip()system = ("You translate questions written in Kazakh into SQLite queries for the database below.\n"          "Reply with one SQL query and nothing else. String values must use the exact spellings "          "given in the schema comments.\n\n" + schema)messages = [{"role": "system", "content": system},            {"role": "user", "content": "Қарағанды облысында қанша қала бар?"}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False,                                 return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**inputs, max_new_tokens=256, do_sample=False)print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))# SELECT COUNT(*) FROM cities WHERE region_id = (SELECT id FROM regions WHERE name = 'Қарағанды облысы')

For the other databases of the dataset use db/aux/<name>.schema.sql; for your own, the same kind of commented schema. Run generated SQL read-only: sqlutil.py in the dataset repo does that and scores queries by execution match.

With vLLM, through the OpenAI-compatible API:

bash

vllm serve RakhatM/qwen3-1.7b-kazakh-text-to-sql-grpo --served-model-name kazsql --max-model-len 4096

Send the same two messages with "temperature": 0 and "chat_template_kwargs": {"enable_thinking": false}.

Training

Started from: RakhatM/qwen3-1.7b-kazakh-text-to-sql-dpo.
Prompts: the 1,650 questions of the dpo config of RakhatM/kazakh-text-to-sql, the ones the SFT model still sometimes gets wrong. A question whose samples are all right or all wrong gives no learning signal.
Method: GRPO, 8 samples per question (temperature 0.9, up to 200 tokens). Reward: 1.0 if the query returns the same rows as the gold query, plus 0.2 if it runs, plus 0.1 if it is a single SELECT without prose or code fences. The gold SQL is used only inside the reward. KL penalty β 0.04 towards the starting model. QLoRA with the same LoRA settings as the earlier stages.
Settings: learning rate 2e-5, cosine schedule, 5% warmup, 1 epoch: 413 steps of 4 questions × 8 samples, 59 min on one H100, 7.5 GB peak GPU memory. Per 40-step window the mean reward (maximum 1.3) moved between 0.85 and 0.98 without a clear trend, since each window has different questions; the KL to the starting model reached 0.01.
The adapter was merged into the DPO model (bf16).

Evaluation

Execution match (%) on the test sets of the dataset, greedy decoding:

Table with columns: model, test_hero, test_unseen, test_independent, test_unseen_independent
model	test_hero	test_unseen	test_independent	test_unseen_independent
Qwen3-1.7B-Base, 3-shot	35.8	36.8	31.2	33.8
SFT	66.7	64.8	63.7	51.2
DPO (+RPO)	72.1	68.0	73.8	51.2
GRPO	75.6	67.8	71.2	53.8

test_hero is held out from the training database by gold SQL; test_unseen uses two databases never seen in training; the independent sets were written by another model without access to the training data. With 80–400 questions per set, differences of a few points are within noise. Four of the 80 test_independent questions also occur in the training data (see the dataset card).

Limitations

Trained on small schemas (4 tables, at most 200 rows) and on questions that models wrote. Expect lower accuracy on large real schemas and on how real users phrase questions: on a new schema with independently written questions it is right about half of the time.
It writes SQLite; other SQL dialects were not tested.
The figures in the Qazaqstan database are approximate, so answers about Kazakhstan are only as accurate as that database.

License

Apache 2.0, like the base model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

RakhatM

Model Tree

Base

RakhatM/qwen3-1.7b-kazakh-text-to-sql-dpo

Fine-tuned

this model

Input Modalities

Text

Output Modalities