GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
CraneAILabs
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
CraneAILabs
Model Tree
google/medgemma-1.5-4b-it
Input Modalities
Output Modalities
Supported Functionality
Generation speed. NF4 on Blackwell's GB10 generates at 39.8 tokens/sec — nearly 2x faster than bf16 on the same hardware, and 3.2x faster than Q4_K_M GGUF on a 4-core CPU.
The speed advantage comes from Blackwell's native low-precision compute units combined with bitsandbytes' NF4 kernel fusion. Quantization reduces memory bandwidth pressure, which is the primary bottleneck during autoregressive decoding.
All configurations tested on the same 5 medical prompts, generating 200 tokens each at temperature 0.3.
| Configuration | Hardware | Generation Speed | VRAM / RAM |
|---|---|---|---|
| NF4 (this model) | DGX Spark GB10 | 39.3 tok/s | 3.5 GB |
| bf16 (full precision) | DGX Spark GB10 | 20.5 tok/s | 8.6 GB |
| Q4_K_M GGUF | Azure 4-core EPYC | 12.3 tok/s | ~4 GB |
python
import torchfrom transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig tokenizer = AutoTokenizer.from_pretrained("CraneAILabs/medgemma-1.5-4b-it-nf4-blackwell")model = AutoModelForCausalLM.from_pretrained( "CraneAILabs/medgemma-1.5-4b-it-nf4-blackwell", device_map="auto") messages = [{"role": "user", "content": "What are the symptoms of malaria?"}]inputs = tokenizer.apply_chat_template( messages, return_tensors="pt", add_generation_prompt=True).to(model.device) with torch.no_grad(): outputs = model.generate(inputs, max_new_tokens=200, temperature=0.3, do_sample=True) print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))Prompt: "What are the symptoms of malaria?"
The symptoms of malaria can vary depending on the type of malaria parasite, the severity of the infection, and the individual's immune status. Common symptoms include:
Early Symptoms: Fever, chills, headache, muscle aches, fatigue, nausea, vomiting...
Response quality is indistinguishable from the full-precision bf16 model.
max_new_tokens=200, temperature=0.3, do_sample=Truetorch.cuda.synchronize() before and after generation; wall-clock for CPUDGX Spark (GPU benchmarks)
Azure D4as_v5 (CPU baseline)
Built by Crane AI Labs.
import torchfrom transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig tokenizer = AutoTokenizer.from_pretrained("CraneAILabs/medgemma-1.5-4b-it-nf4-blackwell")model = AutoModelForCausalLM.from_pretrained( "CraneAILabs/medgemma-1.5-4b-it-nf4-blackwell", device_map="auto") messages = [{"role": "user", "content": "What are the symptoms of malaria?"}]inputs = tokenizer.apply_chat_template( messages, return_tensors="pt", add_generation_prompt=True).to(model.device) with torch.no_grad(): outputs = model.generate(inputs, max_new_tokens=200, temperature=0.3, do_sample=True) print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))