GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
zhaohq
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
zhaohq
Model Tree
Qwen/Qwen3-4B
Input Modalities
Output Modalities
Supported Functionality
python
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "zhaohq/Qwen3-4B-E2C-SFT-only" # add revision="step-1000" for the final checkpointtok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto") msgs = [{"role": "user", "content": "What is the remainder when 2^100 is divided by 7?"}]# Same template as training: default Qwen3 chat template, add_generation_prompt=True, no enable_thinking kwarg.prompt = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)out = model.generate(**ids, max_new_tokens=4096, do_sample=False)print(tok.decode(out[0, ids.input_ids.shape[1]:], skip_special_tokens=False))The model stops on <|im_end|>. To split the output, take the text between <EXPLORATION>/</EXPLORATION> (plan)
and between <EXECUTION>/</EXECUTION> (execution); the answer is the last \boxed{} in the execution.
| Base | Qwen/Qwen3-4B @ 1cfa9a7208912126459214e8b04321603b3df60c |
| Code | E²C repo @ 6e6a901, verl/verl/trainer/fsdp_sft_trainer.py + sft_trainer_2 config, Hydra overrides only |
| Data | official e2c-sft.parquet (58,136 rows), official 95/5 split (seed 42) → 49,684 train / 2,611 val after filtering (below) |
| Method | full-parameter, FSDP2, bf16, gradient checkpointing, flash-attention-2 |
| Sequence | max_length=4096, truncation=error (no sample truncated; longest is 3,311 tokens) |
| Optimizer | AdamW lr 2e-5, betas (0.9, 0.95), wd 0.01, warmup 10%, cosine over 1000 steps, grad clip 1.0 |
| Batch | global 64 (micro-batch 1 × 32 accumulation × 2 GPUs), 1000 steps = 64,000 examples ≈ 1.29 epochs |
| Hardware / time | 2 × NVIDIA B200, 6.3 s/step, 6,495 s of train steps (1 h 54 min wall clock) |
| Loss | supervised tokens = `solution + < |
max_length 4096 instead of 1000. With the official 1000-token right truncation, only 37.7% of training samples stay complete; 51.5% lose the part of the Execution that contains the answer.</EXECUTION> is appended right after; 5,166 of them have no answer. Samples whose Execution reaches ≥1,995 tokens were dropped.<|im_end|> to the target. The official data has no EOS and the trainer does not add one, so a model trained as-is never learns to stop.total_training_steps (the official code uses 3 epochs regardless); added periodic val-loss; right-padding columns are trimmed per micro-batch (compute only, loss unchanged); the redundant second model build is skipped.verl/verl/models/ from volcengine/verl v0.5.0. The repo's .gitignore rule models/ excluded this directory, so the official SFT entry point fails at import without it.Exposure is about 0.4× the official config (160 × 1000 examples). Results are therefore not a like-for-like reproduction of the paper.
Validation loss on all 2,611 val samples (per-sample token-mean CE, same metric as the trainer):
| checkpoint | val loss |
|---|---|
step 800 (main) | 0.18227 |
| step 900 | 0.18235 |
step 1000 (step-1000) | 0.18234 |
Step 800 is lower than steps 900 and 1000 in a paired per-sample test (z ≈ −5.4 to −5.9), but the gap is tiny (≈0.04%). The second epoch starts at step 776, after which the model begins to memorize training samples slightly.
20 MATH-500 problems (smoke test, not a benchmark). Problems were skipped if they matched SFT data exactly or shared any 13-gram with it. Greedy decoding, up to 4,096 new tokens. Evaluated on step 1000; step 500 gave identical numbers. Step 800 was not separately evaluated with generation.
| metric | result |
|---|---|
E²C format valid (plan → closed execution → \boxed{}) | 20/20 |
| execution contains real derivation | 20/20 |
| stops on `< | im_end |
| accuracy (math_verify) | 15/20 |
| fixed-plan continuation (3 problems) | 3/3 closed, no new plan, 2/3 correct |
How much the plan matters (step 800, all val samples, forward only). Execution CE with the gold plan is 0.145; with an empty plan it is 0.274. Lines that restate the plan verbatim make up 11.3% of execution tokens (CE 0.011 → 0.627). On all other execution tokens, CE goes from 0.162 to 0.229, so the plan lowers it by 29%.
</EXPLORATION><EXECUTION> tokenizes as … ATION >< EXEC UTION >\n. The >< token is shared by both tags, so there is no clean plan/execution cut. In 12% of samples the closing </ also merges with the last execution character (e.g. .</). See codi_stage2/ for the boundary convention.training_records/: full report (report.md, Chinese), metrics, run/artifact manifests (hashes, versions), loss curve, data audit (length stats, loss-mask audit, tag tokenization), raw eval samples, full-val results.code_changes/: trainer patch, launcher, resolved config, all helper scripts.codi_stage2/: notes on using this checkpoint as a CODI (explicit-plan → latent) teacher, including token-boundary checks.Please cite the E²C authors for the method and data (see the E²C repository) and Qwen3 for the base model.
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "zhaohq/Qwen3-4B-E2C-SFT-only" # add revision="step-1000" for the final checkpointtok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto") msgs = [{"role": "user", "content": "What is the remainder when 2^100 is divided by 7?"}]# Same template as training: default Qwen3 chat template, add_generation_prompt=True, no enable_thinking kwarg.prompt = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)out = model.generate(**ids, max_new_tokens=4096, do_sample=False)print(tok.decode(out[0, ids.input_ids.shape[1]:], skip_special_tokens=False))