GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
xbruce22
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
xbruce22
Model Tree
unsloth/gemma-4-E2B-it
Input Modalities
Output Modalities
Supported Functionality
| File | Why |
|---|---|
adapter_model.safetensors | The trained LoRA weights (12.08M params, ~46 MB) |
adapter_config.json | LoRA config (r=8, alpha=8, target modules) |
tokenizer.json, tokenizer_config.json, chat_template.jinja | Gemma4 tokenizer + chat template |
chat.py | Ready-to-run interactive chat script (streaming) |
README.md | This file |
This is a LoRA adapter only, not a standalone model. You load the base model (
unsloth/gemma-4-E2B-it) and apply this adapter on top — see below.
bash
pip install torch transformers peftpython chat.pychat.py auto-detects CUDA / Intel XPU / CPU, loads the base model, applies this adapter, merges it, and starts a streaming chat with thinking ON. In-chat commands: /q quit · /reset clear history · /raw show special-token markers · /think toggle thinking.
python
import torchfrom transformers import AutoModelForCausalLM, AutoProcessorfrom peft import PeftModel BASE = "unsloth/gemma-4-E2B-it"ADAPTER = "xbruce22/gemma-4-e2b-reasoning-lora" device = "cuda" if torch.cuda.is_available() else ( "xpu" if hasattr(torch, "xpu") and torch.xpu.is_available() else "cpu")dtype = torch.float32 if device == "cpu" else torch.bfloat16 base = AutoModelForCausalLM.from_pretrained(BASE, dtype=dtype).to(device)model = PeftModel.from_pretrained(base, ADAPTER)# Optional: merge LoRA into the weights for faster inferencemodel = model.merge_and_unload()model.eval() processor = AutoProcessor.from_pretrained(BASE) messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Write DFS in python, keep short."},]text = processor.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, enable_thinking=True) inputs = processor(text=[text], return_tensors="pt").to(device)# Text-only: drop multimodal-only fields generate() rejectsfor k in list(inputs): if "token_type" in k or "pixel" in k or "audio" in k: inputs.pop(k) with torch.inference_mode(): out = model.generate( **inputs, max_new_tokens=1024, do_sample=True, temperature=1.0, top_p=0.95, top_k=64, pad_token_id=processor.tokenizer.pad_token_id) gen = out[0][inputs["input_ids"].shape[1]:]print(processor.decode(gen, skip_special_tokens=True))Notes:
enable_thinking=True to apply_chat_template so the template injects <|think|> and the model produces the <|channel>thought ... <channel|> reasoning block before the answer.temperature=1.0, top_p=0.95, top_k=64.merge_and_unload(), keep using the PeftModel directly — both work.Prompt: Write DFS in python, keep short.
text
── thinking ──- User wants a DFS implementation in Python, explicitly requesting it be "short"- Settled on iterative version using a stack and visited set ...- Concise version: no classes, just a function — keeps it short while remaining correct── answer ──def dfs(graph, start, visited=None): ...The reasoning is now terse, bulleted, and scannable — the style it was fine-tuned to produce.
q/k/v/o_proj) + MLP (gate/up/down_proj) modules. Vision and audio towers frozen (text-only finetune).Jackrong/GLM-5.1-Reasoning-1M-Cleaned (main subset). The verbose imd…answer thinking traces were condensed into terse flat bullet lists (via a condenser prompt); the original final answers were kept verbatim.<|channel>thought\n...bullets...\n<channel|> then the final answer, <|turn> turn markers, assistant-only loss (user/system tokens masked to -100).adamw_torch, gradient checkpointing. No 4-bit / bitsandbytes (no XPU build).Apache-2.0 (adapter weights). The base model unsloth/gemma-4-E2B-it follows Gemma's terms.