GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
code-aitazaz
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
code-aitazaz
Model Tree
Qwen/Qwen3-8B
Input Modalities
Output Modalities
Supported Functionality
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigfrom peft import PeftModelbnb_config = BitsAndBytesConfig(load_in_4bit=True,bnb_4bit_quant_type="nf4",bnb_4bit_use_double_quant=True,bnb_4bit_compute_dtype=torch.float16, # not bfloat16 -- see note below if on a Turing GPU (e.g. T4))tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", quantization_config=bnb_config, device_map="auto")model = PeftModel.from_pretrained(base, "code-aitazaz/roman-urdu-qlora-qwen3-8b")messages = [{"role": "user", "content": "Pakistan ka dar-ul-hukumat kya hai?"}]prompt_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, enable_thinking=False, return_tensors="pt")["input_ids"].to(model.device)output_ids = model.generate(prompt_ids,max_new_tokens=256,do_sample=False,repetition_penalty=1.2, # see "Known limitation" below -- importantno_repeat_ngram_size=3,pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id,)print(tokenizer.decode(output_ids[0][prompt_ids.shape[1]:], skip_special_tokens=True))
Note on hardware: bnb_4bit_compute_dtype=torch.float16 (not bfloat16)
is deliberate — a T4 (Turing, compute capability 7.5) has no bf16 tensor
cores; most QLoRA examples default to bf16 assuming A100-class hardware.
Qwen/Qwen3-8B, Apache-2.0.Redgerd/roman-urdu-alpaca-qa-mix,
filtered down to ~485 genuinely-Roman-Urdu rows after finding and removing a
real duplication bug in the upstream dataset (see the project repo's README
for the full data-quality writeup).Real numbers from an actual training run (ROUGE-L on 40 held-out prompts, base vs. this adapter):
| Model | ROUGE-L Precision | ROUGE-L Recall | ROUGE-L F-measure |
|---|---|---|---|
| Base (zero-shot) | 0.194 | 0.128 | 0.095 |
| This adapter | 0.211 | 0.174 | 0.164 |
Known limitation, reported honestly: the adapter reliably achieves its
actual goal (base Qwen3-8B often answers Roman Urdu prompts in English or
garbled Urdu script; this adapter answers in Roman Urdu, on-topic, every
time) — but several longer completions degrade into verbatim repetition
loops under plain greedy decoding, a known small-dataset (485 rows) +
do_sample=False failure mode. Use repetition_penalty≈1.2 and
no_repeat_ngram_size≈3 at generation time (as in the usage snippet above)
— this is a decoding-time fix, not a retraining requirement, added after the
run that produced the numbers above (which predate the fix). See the project
repo's README for full example transcripts showing both the successes and
the repetition failure mode.
Adapter weights: MIT (same as the project code). Base model: Apache-2.0. Training data: Apache-2.0.