GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
UmitVice
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
UmitVice
Model Tree
google/gemma-4-12B-it-qat-q4_0-unquantized
Input Modalities
Output Modalities
Supported Functionality
Prompt support-v4, llama.cpp on an Apple M5 Pro, merged Q4_0 vs the same pipeline without
the adapter.
| Cloudflare Llama 3.1 8B | Gemma 4 12B QAT Q4_0 | This model, Q4_0 | |
|---|---|---|---|
| RelayNest end-to-end pass (30 generations) | 0.800 | 0.867 | 0.900 |
| RelayNest latency p50 / p95 | 1.7 s / 3.1 s (cloud GPU) | 7.1 s / 22.3 s | 2.6 s / 6.1 s |
| Held-out products pass (104) | – | 0.942 | 0.971 |
| Held-out Turkish | – | 0.925 | 0.975 |
| Held-out without grammar-constrained decoding | – | 0.269 | 0.904 |
| Mean output tokens | – | 244 | 109 |
The held-out products (FlagForge, Fatura Bulutu) and RelayNest were never in the training
data. Full tables and the history of failed iterations are in reports/.
| Path | Use |
|---|---|
adapter_model.safetensors, adapter_config.json | PEFT LoRA for transformers |
mlx/ | the original MLX adapter (mlx_lm.load(base, adapter_path="mlx")) |
reports/ | evaluation reports |
The adapter targets the attention and MLP projections of all 48 text layers (rank 16, alpha 32). It was trained against MLX 4-bit weights; for 4-bit serving, merge it into the bf16 base and then quantize, as the GGUF repo does. Applying it as a runtime LoRA on a different 4-bit quant can change its behavior.
python
import torchfrom peft import PeftModelfrom transformers import AutoTokenizer, Gemma4UnifiedForConditionalGeneration base = "google/gemma-4-12B-it-qat-q4_0-unquantized"tokenizer = AutoTokenizer.from_pretrained(base)model = Gemma4UnifiedForConditionalGeneration.from_pretrained(base, dtype=torch.bfloat16)model = PeftModel.from_pretrained(model, "UmitVice/evidencedesk-gemma-4-12b-lora").merge_and_unload()Render prompts with EvidenceDesk's build_messages() (system prompt, JSON schema, ticket,
question, evidence) and apply_chat_template(..., add_generation_prompt=True).
import torchfrom peft import PeftModelfrom transformers import AutoTokenizer, Gemma4UnifiedForConditionalGeneration base = "google/gemma-4-12B-it-qat-q4_0-unquantized"tokenizer = AutoTokenizer.from_pretrained(base)model = Gemma4UnifiedForConditionalGeneration.from_pretrained(base, dtype=torch.bfloat16)model = PeftModel.from_pretrained(model, "UmitVice/evidencedesk-gemma-4-12b-lora").merge_and_unload()