GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Carchofa
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Carchofa
Model Tree
google/gemma-4-26B-A4B-it
Input Modalities
Output Modalities
Supported Functionality
google/gemma-4-26B-A4B-it) grades coding, technical, instruction-following, and tool-use trajectories.self_attn.{q,k,v,o}_proj and the shared dense mlp.{gate,up,down}_proj across all 30 layers (74,342,400 trainable parameters).Gemma-4-26B-A4B-Humanlike-Q4_K_M.gguf): Standalone 4-bit quantized GGUF ready for llama-server / llama-cli.Gemma-4-26B-A4B-Humanlike-LoRA-r64-F16.gguf, 141.8 MB): Hot-loads directly on top of any google/gemma-4-26B-A4B-it GGUF via --lora.adapter_model.safetensors, adapter_config.json): Standard PEFT LoRA checkpoint at repo root (final distilled step 12) and sft_adapter/ (post-Stage-1 SFT).QuantizedGemma4TextExperts):
bitsandbytes load_in_4bit=True ignores the 3D fused nn.Parameter tensors (gate_up_proj and down_proj across 128 experts × 30 layers = 35.1 GB in BF16)./tmp, quantizes all 128 experts per layer into 4-bit NF4 with double quantization, and dequantizes active experts on-the-fly during forward and backward via a custom autograd function.4.24 GB / 31.35 GB; Peak VRAM 9.06 GB on GPU 0 and 6.97 GB on GPU 1 across the entire SFT + On-Policy Reverse-KL Distillation run.labels = -100 on prompt tokens) across 9 task categories (casual_chat, long_chat, clarification_ask, tool_selection, coding_technical, instruction_formatting, character_roleplay, preference_corrections, thinking_reasoning).KL(p_student || p_teacher)) computed against Teacher A (conversational/long-chat) and Teacher B (base model via disable_adapter()).llama.cpp (llama-server)bash
llama-server -m Gemma-4-26B-A4B-Humanlike-Q4_K_M.gguf --jinja --ctx-size 32768 --temp 1.0 --top-p 0.95 --top-k 64bash
llama-server -m google_gemma-4-26B-A4B-it-Q4_K_M.gguf \ --lora Gemma-4-26B-A4B-Humanlike-LoRA-r64-F16.gguf \ --jinja --ctx-size 32768 --temp 1.0 --top-p 0.95 --top-k 64llama-server -m Gemma-4-26B-A4B-Humanlike-Q4_K_M.gguf --jinja --ctx-size 32768 --temp 1.0 --top-p 0.95 --top-k 64llama-server -m google_gemma-4-26B-A4B-it-Q4_K_M.gguf \ --lora Gemma-4-26B-A4B-Humanlike-LoRA-r64-F16.gguf \ --jinja --ctx-size 32768 --temp 1.0 --top-p 0.95 --top-k 64