GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
SIZWL
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
SIZWL
Model Tree
Input Modalities
Output Modalities
Supported Functionality
All numbers are full-set, produced by scripts/bench.py (see Evaluation method below).
acc is the primary metric.
| Benchmark | n | acc | acc_norm | Random baseline | Verdict |
|---|---|---|---|---|---|
| LAMBADA | 5,153 | 46.51% ¹ | 18.42% ² | ~0% | ✅ far above random |
| ARC-Easy | 570 | 32.63% | 29.12% | 25–27.8% | ✅ +5–8 pp |
| HellaSwag | 10,042 | 26.27% | 25.88% | 25.0% | ❌ +1.3 pp only |
| C-Eval | 1,346 | 21.62% | 27.56% | 25.0% | ⚠️ acc below random |
¹ LAMBADA is single-candidate, so multi-choice acc is meaningless (always 100%).
Reported here as greedy token hit rate — whether the target word is the argmax
continuation — which is the metric lm-eval-harness uses for this task.
² Whole-word all-correct rate.
Plus, from the deterministic dev pass (13,064,192 tokens):
| dev loss / ppl | 3.3336 / 28.04 (σ≈0.006) |
| EOS boundary signal | P(EOS|doc end)=0.611 vs P(EOS|body)=0.000014 → 562× |
| Overfitting check | sampling σ=0.0930 → no overfitting |
Can: English language modeling. LAMBADA 46.51% against a ~0% random baseline is a large, unambiguous signal that 2.6B tokens of pretraining produced a working language model. Sentences are fluent and well-formed.
Cannot: world knowledge and commonsense reasoning. HellaSwag sits 1.3 pp above random despite a median top-2 margin of 14.6 nats — the model is confidently wrong, not undecided. That task requires understanding what people are physically doing in a scene; fluency does not transfer to it.
Chinese: no academic knowledge. C-Eval accuracy (21.62%) falls below random, and the prediction distribution collapses onto option A (42.3% predicted vs 23.0% actual gold), i.e. a positional bias rather than knowledge. Length-normalized accuracy (27.56%) is barely above random.
The LAMBADA ↔ HellaSwag gap (46.51% vs 26.27%) is the clearest single statement of what this checkpoint is: it learned language, not knowledge.
The original acceptance criterion for this tier was HellaSwag ≥35%. Measured: 26.27%. Before accepting that, the following were ruled out one by one: scorer self-checks passed; bf16 noise is ~29× smaller than the median decision margin; trivial baselines (always-first, always-longest, always-shortest) all score 24–25%; the same code scores 32.63% on ARC-Easy, above its own random baseline; and a tokenization-boundary defect was found, quantified (+1.67 pp), and shown to be far too small to close an 8.7 pp gap.
The cause is data, and it is measurable. Training-data language mix by token, not by document:
| docs | tokens (measured) | |
|---|---|---|
| English | 70% | 49.9% |
| Chinese | 30% | 45.9% |
Chinese documents yield ~2.1× more tokens each, so a 70/30 document split becomes a 50/46 token split. HellaSwag is pure English, so this model effectively saw ~1.30B English tokens — and a chunk of that was TinyStories (synthetic simple English), which is close to useless for commonsense about real-world activity.
No scale-up was attempted. Per the project's decision rule, a stage that misses its gate may only be made smaller or shorter, never larger.
Research and experimentation on small-scale base-model pretraining. Suitable as a fine-tuning base for English tasks. Not suitable for factual QA, commonsense reasoning, Chinese academic work, or any production deployment.
lm-eval-harness could not be installed (no PyPI access from this environment), so the
four loglikelihood benchmarks are implemented in scripts/bench.py, aligned with
lm-eval's doc_to_text / doc_to_target / preprocess. Scorer correctness is enforced by
three self-checks that must pass before any number is reported:
batch=1 reference implementation, per-item logprobTolerances are dtype-aware: fp32 1e-3, bf16 0.5. (bf16's noise floor was measured at
7.1e-2 against fp32's 5.7e-6, matching the predicted bound for logit magnitude 10.9.)
Export to this Qwen2 format was verified numerically against the training implementation: max absolute error 4.244e-05, cosine similarity 1.0000000000, argmax agreement 100.00%.
One caveat: lm-eval tokenizes context and continuation separately, which for ByteLevel BPE
without a leading space produces a boundary that does not exist in the original text
('…on a roof. Heis using…'). This matches lm-eval's behaviour and is kept for
comparability; correcting it raises HellaSwag 300-item accuracy 32.00% → 33.67%.
Optimizer AdamW (β 0.9/0.95, wd 0.1, eps 1e-8), LR 3e-4 → 3e-5 cosine, 2% warmup,
grad clip 1.0, global batch ≈ 524k tokens/step (32 × 2048 × 8 accumulation), bf16 with SDPA,
torch.compile, seed 1337. 4,961 steps over 14h42m on a single NVIDIA DGX Spark (GB10);
train EMA loss 11.0017 → 3.2274 (−70.7%); peak memory 29.8 GB.
python
from transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("SIZWL/OpenSI-0.3B-Base")model = AutoModelForCausalLM.from_pretrained( "SIZWL/OpenSI-0.3B-Base", dtype="auto", device_map="auto") ids = tok("The capital of France is", return_tensors="pt").to(model.device)out = model.generate(**ids, max_new_tokens=20, do_sample=False, pad_token_id=tok.pad_token_id)print(tok.decode(out[0]))This is a base model, not instruction-tuned. It continues text rather than answering questions, and may not terminate cleanly on a complete sentence — prefer sampling with a repetition penalty for open-ended generation:
python
out = model.generate(**ids, max_new_tokens=64, do_sample=True, temperature=0.8, top_k=40, repetition_penalty=1.2, eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)| Source | Role |
|---|---|
| FineWeb | English web text |
| Wikipedia (zh) | Chinese encyclopedic |
| TinyStories | Synthetic simple English |
| Industry Chinese documents | Chinese domain text |
See §4.5 of the project plan for the mix and §4.7 for the verification tooling.
MIT.
bibtex
@online{opensi03bbase, title = {{OpenSI-0.3B-Base}: A 113.9M Open Base Language Model Pretrained on 2.6B Tokens}, author = {{OpenSI Project}}, year = {2026}, url = {https://huggingface.co/SIZWL/OpenSI-0.3B-Base}, note = {Exported in Qwen2 format. Actual size 113.9M parameters.}}from transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("SIZWL/OpenSI-0.3B-Base")model = AutoModelForCausalLM.from_pretrained( "SIZWL/OpenSI-0.3B-Base", dtype="auto", device_map="auto") ids = tok("The capital of France is", return_tensors="pt").to(model.device)out = model.generate(**ids, max_new_tokens=20, do_sample=False, pad_token_id=tok.pad_token_id)print(tok.decode(out[0]))out = model.generate(**ids, max_new_tokens=64, do_sample=True, temperature=0.8, top_k=40, repetition_penalty=1.2, eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)@online{opensi03bbase, title = {{OpenSI-0.3B-Base}: A 113.9M Open Base Language Model Pretrained on 2.6B Tokens}, author = {{OpenSI Project}}, year = {2026}, url = {https://huggingface.co/SIZWL/OpenSI-0.3B-Base}, note = {Exported in Qwen2 format. Actual size 113.9M parameters.}}