GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
hai-hu
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
hai-hu
Model Tree
Input Modalities
Output Modalities
Supported Functionality
python
from transformers import AutoTokenizer, LlamaForCausalLMtok = AutoTokenizer.from_pretrained("hai-hu/qwen2-style-50k")model = LlamaForCausalLM.from_pretrained("hai-hu/qwen2-style-50k")# earlier steps: add revision="step<N>"
The model was trained in gpt-neox with biases on all four attention
projections (Q/K/V and the output projection). HuggingFace's
Qwen2ForCausalLM hard-codes o_proj bias=False, so a Qwen2-class export would
have to drop the trained o-projection bias. On a validation batch at the final
checkpoint that bias moves logits by up to 25 (mean 3.0) and flips ~78% of
argmaxes — i.e. it is not negligible, so exporting as Qwen2 would silently
produce a different, broken model.
LlamaForCausalLM with attention_bias=true (biases on q/k/v and o_proj)
represents the trained model exactly; RMSNorm, SwiGLU, rotary embeddings,
GQA and tied embeddings are otherwise identical between the two classes.
Conversion was verified to logits parity against the original gpt-neox
checkpoint: max |Δlogit| = 4e-5 (fp32 numerics), 100% argmax agreement.
So: load it with LlamaForCausalLM (as above) and treat it as a Qwen2-family
architecture. The weights are untouched originals.
| layers | 24 |
| hidden size | 896 |
| attention heads | 14 (head_dim 64) |
| KV heads (GQA) | 2 |
| intermediate (SwiGLU) | 2432 |
| context length | 4096 |
| positional encoding | RoPE, theta 10000, 100% |
| norm | RMSNorm, eps 1e-6 |
| embeddings | tied, vocab 50,048 padded (tokenizer: 50,002 = 50k BPE + `< |
| parameters | 246M (291M counting the tied head once per role) |
Adam(0.9, 0.95), lr 3e-4 → cosine → 3e-5, warmup 8.2% (2000 steps), weight decay 0.1, grad clip 1.0, bf16, ZeRO-1, global batch 1024 sequences × 4096 tokens. Data: CCI3-HQ (~110B-token corpus, ~1 epoch for 24414 steps). Checkpoint every 250 steps; per-step branches here are a subset.
Hardware: 2× NVIDIA RTX PRO 6000 Blackwell (96GB), ~37.4 s/step, ~12.5 days wall clock.