Meridian-MRM

Samarkand-253m

README

License: apache-2.0

license: apache-2.0 language:

en pipeline_tag: text-generation library_name: transformers tags:
gguf
llama.cpp
qwen3
base-model
jax
tpu
meridian

model-index:

name: Samarkand-253M results:
- task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge type: ai2_arc config: ARC-Easy split: test metrics:
  - name: Accuracy type: acc value: 56.94
  - name: Normalized Accuracy type: acc_norm value: 50.84 source: name: lm-eval url: https://github.com/EleutherAI/lm-evaluation-harness
- task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge type: ai2_arc config: ARC-Challenge split: test metrics:
  - name: Accuracy type: acc value: 25.26
  - name: Normalized Accuracy type: acc_norm value: 26.96 source: name: lm-eval url: https://github.com/EleutherAI/lm-evaluation-harness
- task: type: text-generation name: Text Generation dataset: name: HellaSwag type: hellaswag metrics:
  - name: Accuracy type: acc value: 32.68
  - name: Normalized Accuracy type: acc_norm value: 38.15 source: name: lm-eval url: https://github.com/EleutherAI/lm-evaluation-harness
- task: type: text-generation name: Text Generation dataset: name: PIQA type: piqa metrics:
  - name: Accuracy type: acc value: 67.25
  - name: Normalized Accuracy type: acc_norm value: 66.49 source: name: lm-eval url: https://github.com/EleutherAI/lm-evaluation-harness
- task: type: text-generation name: Text Generation dataset: name: WinoGrande type: winogrande metrics:
  - name: Accuracy type: acc value: 50.43 source: name: lm-eval url: https://github.com/EleutherAI/lm-evaluation-harness
- task: type: text-generation name: Text Generation dataset: name: OpenBookQA type: openbookqa metrics:
  - name: Accuracy type: acc value: 19.60
  - name: Normalized Accuracy type: acc_norm value: 32.80 source: name: lm-eval url: https://github.com/EleutherAI/lm-evaluation-harness
- task: type: text-generation name: Text Generation dataset: name: SciQ type: sciq metrics:
  - name: Accuracy type: acc value: 79.20
  - name: Normalized Accuracy type: acc_norm value: 71.50 source: name: lm-eval url: https://github.com/EleutherAI/lm-evaluation-harness
- task: type: text-generation name: Text Generation dataset: name: BLiMP type: blimp metrics:
  - name: Accuracy type: acc value: 81.39 source: name: lm-eval url: https://github.com/EleutherAI/lm-evaluation-harness
- task: type: text-generation name: Text Generation dataset: name: ArithMark-3.0 type: arithmark metrics:
  - name: Accuracy type: acc value: 35.10
  - name: Normalized Accuracy type: acc_norm value: 35.50 source: name: AxiomicLabs url: https://github.com/AxiomicLabs/ArithMark
- task: type: text-generation name: Text Generation dataset: name: WikiText-2 type: wikitext metrics:
  - name: Word Perplexity type: word_perplexity value: 29.71
  - name: Byte Perplexity type: byte_perplexity value: 1.8855
  - name: Bits per Byte type: bits_per_byte value: 0.9150 source: name: lm-eval url: https://github.com/EleutherAI/lm-evaluation-harness datasets:
HuggingFaceFW/fineweb-edu
mlfoundations/dclm-baseline-1.0

Samarkand-253M is a 253M-parameter base language model, built from scratch end to end in JAX/Flax NNX on Google's TPU architecture, with a custom 30k byte-level BPE tokenizer.

Training data: FineWeb-Edu and DCLM only. No benchmark training sets were used.

This is a base model: it is not instruction-tuned and has no chat template.

Training

Framework: JAX / Flax NNX, trained from scratch
Hardware: one TPU v5e (8 chips for training, less for testing)
Batch: 131,072 tokens per step
Optimizer: AdamW, peak LR 1e-3, warmup-stable-decay schedule
Data: FineWeb-Edu + DCLM

Mid training strategy consisted of micro managing dclm training ratio to preserve fineweb-edu whilst giving general register.

Architecture

26 layers, d_model 896, 14 attention heads with 2 KV heads (GQA), head_dim 64, SwiGLU FFN (d_ff 2560), QK-norm, RoPE (theta 10000), tied embeddings, context length 2048, vocabulary 30,002.

It is stored in the Qwen3 layout, so it loads with Qwen3ForCausalLM in transformers and runs in llama.cpp as the qwen3 architecture.

Files

model.safetensors, config.json, tokenizer.json, tokenizer_config.json: Hugging Face format
Samarkand-253m-f16.gguf: llama.cpp (needs a recent build with Qwen3 support)

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer repo = "dragosMediv/Samarkand-253m"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo) inputs = tok("The capital of France is", return_tensors="pt")out = model.generate(**inputs, max_new_tokens=40)print(tok.decode(out[0]))

text

llama-cli -m Samarkand-253m-f16.gguf -p "The capital of France is"

Evaluation

All zero-shot, fp32, from the Hugging Face weights in this repo, using lm-evaluation-harness (ArithMark-3.0 with its official script).

Table with columns: Benchmark, Chance, acc, acc_norm
Benchmark	Chance	acc	acc_norm
ARC-Easy	25	56.94	50.84
ARC-Challenge	25	25.26	26.96
HellaSwag	25	32.68	38.15
PIQA	50	67.25	66.49
WinoGrande	50	50.43	-
OpenBookQA	25	19.60	32.80
SciQ	25	79.20	71.50
BLiMP	50	81.39	-
ArithMark-3.0	25	35.10	35.50

WikiText-2 (test, 1024-token windows): word perplexity 29.71, byte perplexity 1.8855, bits per byte 0.9150.

Limitations

English only
Base model: no instruction or safety tuning
Can produce incorrect or made-up text
2048-token context

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider