Model Details
- Base Model: zai-org/GLM-5.3-Flash
- Architecture: glm5_next (Glm5NextForConditionalGeneration, multimodal VLM: glm5_next_text + glm5_next_vision)
- Total Parameters: 2.57B (2,570,456,610)
- Activated Parameters: 2.28B (2,275,605,794; text-only per-token estimate, vision tower excluded, top-8-of-10 experts + shared expert per MoE layer)
Configuration Changes
Only the three allowed categories of changes were made (layer counts, layer-type layout, expert count); every other hyperparameter is identical to the base:
Table with columns: Parameter, Base, Small| Parameter | Base | Small |
|---|
text num_hidden_layers | 45 | 3 |
text layer_types | 34x linear_attention + 11x deepseek_sparse_attention (every 4th layer DSA) | linear_attention, linear_attention, deepseek_sparse_attention |
text mlp_layer_types | 3x dense + 42x sparse | dense, sparse, sparse |
text first_k_dense_replace | 3 | 1 |
text indexer_types | 45x full | 3x full |
text linear_attn_config.kda_layers / full_attn_layers | [0,1,2,4,...,44] / [3,7,...,43] | [0,1] / [2] |
text n_routed_experts | 288 | 10 |
vision depth | 24 | 2 |
The 3 text layers are the minimal set covering every distinct attention layer type (linear_attention KDA, deepseek_sparse_attention MLA+indexer) and every distinct MLP type (dense, sparse MoE), i.e. all three (attention, MLP) combinations that occur in the base. 10 experts >= num_experts_per_tok 8 (topk+2 for non-degenerate routing). Everything else — hidden size 4096, vocab 154880, MLA dims (kv_lora_rank 512, q_lora_rank 1536, nope-only heads), indexer params (index_topk 2048, kpool 4), MoE routing (sigmoid, noaux_tc, norm_topk_prob, routed_scaling_factor 2.5, 1 shared expert), multi-hyper-connection (mhc, hc_mult 4), KDA linear attention dims, rope/max_position_embeddings 1048576, tie_word_embeddings: false, num_nextn_predict_layers: 1 — is unchanged from the base.
Checkpoint Structure
Single-file model.safetensors (bf16, ~5.1GB), unsharded. Tensor naming follows the base checkpoint exactly (validated: all 75 tensor-name patterns match the base model.safetensors.index.json modulo layer/expert/vision-block counts): model.language_model.* text tower (per-expert ungrouped mlp.experts.<i>.{gate,up,down}_proj.weight, self_attn.forget_gate-style original names f_a_proj/f_b_proj/dt_bias/A_log, hc_attn_*/hc_ffn_* hyper-connection names, split q/k/v_conv1d), model.visual.* vision tower, and untied lm_head.weight.
Usage
from transformers import AutoModelForImageTextToText, AutoProcessor model = AutoModelForImageTextToText.from_pretrained("inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small", device_map="auto")processor = AutoProcessor.from_pretrained("inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small") input_ids = processor.tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)output = model.generate(input_ids, max_new_tokens=20)print(processor.tokenizer.decode(output[0]))
Creation Process
This model was created using the llm-compressor create-tiny-model skill.
The config was derived from the base AutoConfig applying only the changes in the table above, the model was built with random init (skip_weights_download + init_weights()), and then fine-tuned on the skill's toy copypasta dataset (8 texts, ~160 steps, lr 1e-4, bf16) until the perplexity stopping criterion was reached.
Validation
- Perplexity on the toy dataset text: 1.0000129 (target <= 10)
- Greedy generation reproduces the training text ("According to all known laws of aviation, there is no way a bee should be able to fly.")
- Multimodal forward smoke test: image through processor + 2-block vision tower produces finite logits
- Checkpoint tensor-name structure matches the base index.json (75/75 patterns)
Notes
- The base model ships with an fp8 (e4m3, dynamic activation scheme)
quantization_config; it was stripped and this small model is unquantized bf16, matching the convention for the other small models in this series.
num_nextn_predict_layers: 1 is kept from the base config; transformers does not materialize MTP weights, so the checkpoint is backbone-only (the base stores its MTP layer as model.layers.45.*, which is intentionally absent here, matching e.g. inference-optimization/GLM-5.2-0.8B-A0.8B).
- Training loss plateaus around 0.30 (train ppl ~1.35) due to the GLM tokenizer's left padding interacting with the causal shift (one unpredictable first-token term per text); unpadded validation perplexity is ~1.0.