GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
oddadmix
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
oddadmix
Model Tree
Input Modalities
Output Modalities
Supported Functionality
| Parameters | ~51.8M |
| Architecture | Gemma-3 (Gemma3ForCausalLM) |
| Hidden size | 512 · Layers 12 · Heads 8 (GQA, 4 KV) · head_dim 64 |
| Intermediate size | 1408 |
| Vocab size | 32000 (custom Arabic byte-level BPE, shared with the 2048 model) |
| Context length | 8192 |
| Attention | interleaved: 10 sliding (window 1024) + 2 global |
| Positional encoding | RoPE, θ = 10,000 on sliding layers, 100,000 on global layers |
| Tied embeddings | yes |
| License | Apache-2.0 |
All-global attention at 8192 tokens costs 453M attention FLOPs per token — roughly as much as the entire rest of the forward pass, and about 4× what the 2048 model spent. Restricting five of every six layers to a 1024-token window drops that to ~13M while leaving 2 layers that see the full window, so the long-range path survives and the run costs about what the original 2048 run cost. Sliding layers use the smaller RoPE base because they only ever have to resolve 1024 positions.
| Corpus | kaust-generative-ai/fineweb-edu-ar (config ar) |
| Budget | 20,000,000,000 tokens, 1 epoch over the stream |
| Packing | documents separated by </s>; documents ≥ 4096 tokens duplicated once |
| Optimizer | AdamW (fused), β = (0.9, 0.95), wd 0.1, grad clip 1.0 |
| Schedule | lr 6e-4, cosine, warmup ratio 0.02 |
| Effective batch | 32 sequences × 8192 = 262,144 tokens/step |
| Precision | bf16, torch.compile, a single consumer GPU |
The corpus is short-document heavy — median 427 tokens, and only 0.47% of
documents reach 8192 — so packing matters more than architecture for long
context here. Documents are separated by </s> (the 2048 recipe used no
separator), and documents of at least 4096 tokens are written twice to raise the
share of tokens sitting inside a genuinely long single-document span.
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model_id = "oddadmix/50M-8192-Nawah-gemma"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16) prompt = "اللغة العربية هي"ids = tok(prompt, return_tensors="pt").to(model.device)out = model.generate(**ids, max_new_tokens=128, do_sample=True, temperature=0.8, top_p=0.9, repetition_penalty=1.2)print(tok.decode(out[0], skip_special_tokens=True))At this size and budget the model has a narrow capability range and will produce factually unreliable and sometimes incoherent text. It is not instruction-tuned or aligned, and no safety filtering has been applied. Attention is not masked at document boundaries during training, so the two global layers were exposed to cross-document context within a packed window.
© KAND CA 2026 — PROJECT NAWAH.
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model_id = "oddadmix/50M-8192-Nawah-gemma"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16) prompt = "اللغة العربية هي"ids = tok(prompt, return_tensors="pt").to(model.device)out = model.generate(**ids, max_new_tokens=128, do_sample=True, temperature=0.8, top_p=0.9, repetition_penalty=1.2)print(tok.decode(out[0], skip_special_tokens=True))