GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
pdjamez
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
pdjamez
Model Tree
Input Modalities
Output Modalities
Supported Functionality
| Property | Value |
|---|---|
| Transformers implementation | LlamaForCausalLM |
| Parameters | 125,095,680 |
| Layers | 18 |
| Hidden width | 768 |
| Attention heads | 12 |
| Key-value heads | 2 |
| Head dimension | 64 |
| Feed-forward width | 2,048 |
| Context length | 2,048 tokens |
| Vocabulary | 20,000 tokens |
| Activation | SwiGLU |
| Normalisation | RMSNorm, epsilon 1e-5 |
| Position encoding | RoPE, theta 10,000 |
| Embeddings | Tied input and output embeddings |
| Published precision | FP32 |
Neeps was trained from scratch and contains no Meta Llama weights. It uses the
standard Transformers Llama implementation because its decoder architecture,
RMSNorm, RoPE, SwiGLU and grouped-query attention are compatible. No custom
code or trust_remote_code=True is required.
The model is intended for small-language-model research, education, base-model evaluation, continued-pretraining experiments and post-training research. It is not suitable as a factual authority, a safety-critical system, or an unsupervised user-facing assistant.
Neeps used causal next-token prediction. The final release candidate is the v0.4 cooldown checkpoint at step 152,588 and 5,000,003,584 consumed tokens, approximately 39.97 training tokens per parameter.
| Training property | Value |
|---|---|
| Sequence length | 2,048 |
| Global batch | 32,768 tokens |
| Stable-trunk learning rate | 3e-4 |
| Cooldown schedule | Cosine from 3e-4 to 3e-5 over 7,635 steps |
| Seed | 42 |
| Backend | Swift with MLX 0.32.2 |
| Hardware | Apple M3 Ultra Mac Studio, 256 GB unified memory |
| Final scheduled validation loss | 2.93024 nats/token |
The cooldown branch began from the stable checkpoint at step 144,953 (4,749,819,904 tokens). The stable 5B checkpoint, not the cooled release weights, remains the continuation parent for future training.
The ordered training stream came from HuggingFaceFW/fineweb-edu,
sample-100BT, pinned at revision
87f09149ef4734204d70ed1d046ddc9ca3f2b8f9. Selection required English,
language score at least 0.90, FineWeb-Edu integer score at least 3, and a crawl
year from 2013 through 2022. Local processing applied quality and high-risk
content screens, exact and near-duplicate filtering, a registrable-domain cap,
selected benchmark decontamination, deterministic train/validation/test
splits, and EOS-delimited 2,048-token packing.
The final train split contains 12,116,013 documents and 12,486,676,062 packed tokens. Neeps consumed the first 5,000,003,584 tokens of that one-pass ordered stream. Training text is not distributed with this model.
The corpus received automated screening before training; it was not exhaustively inspected document-by-document by people. Automated filters can miss private, copyrighted, harmful, biased, inaccurate, or otherwise undesirable material.
FineWeb-Edu is published by Hugging Face from Common Crawl data and is labelled ODC-By 1.0. Source pages can carry separate rights. The Apache-2.0 licence for these model files does not license third-party training text.
The SentencePiece tokenizer was trained separately on the earlier train-only
corpus-v3 FineWeb-Edu-Dedup pilot. Its SHA-256 is
34353c32e8a7c0e8a7f0bb057db4035c5676669d071367d8beb25cc4f1a39f04.
All task results below use the same frozen examples, a 512-token evaluation cap, float32 Transformers loading on Apple MPS, and character-length-normalised conditional log-likelihood for choice tasks.
| Benchmark | Examples | Score |
|---|---|---|
| ARC-Challenge validation | 299 | 25.75% |
| ARC-Challenge full test | 1,172 | 26.19% |
| ARC-Easy validation | 570 | 40.53% |
| PIQA validation | 1,838 | 62.46% |
| HellaSwag validation | 10,042 | 32.73% |
| LAMBADA OpenAI test, exact greedy target | 5,153 | 28.10% |
| BLiMP grammatical-pair accuracy | 67,000 | 80.10% |
Packed test loss was 2.94005 nats/token (perplexity 18.917) over 12,746,752 scored tokens. These benchmark files and the packed test shard were repeatedly used during development and are not pristine final holdouts. ARC-Challenge full test and BLiMP were not included in the recorded decontamination pass, so their scores should be treated as developmental.
Eight fixed 128-token greedy continuations emitted no EOS and had a mean 77.68% repeated-trigram fraction. Severe looping and false factual claims remain. A narrow 256-prefix extraction probe found no exact 64-token training continuations; this is not evidence that memorisation or privacy leakage is absent. A synthetic 2,048-token four-choice retrieval probe scored 5/8, which is too small to establish reliable long-context retrieval.
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model_id = "pdjamez/Neeps-125M-Base"tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=False, dtype=torch.float32,).eval() prompt = "The weather in Scotland is"max_new_tokens = 64inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)if inputs.input_ids.shape[1] + max_new_tokens > 2048: raise ValueError("Neeps supports at most 2,048 total input/output tokens") with torch.inference_mode(): output = model.generate( **inputs, max_new_tokens=max_new_tokens, do_sample=True, temperature=0.8, top_p=0.9, repetition_penalty=1.1, pad_token_id=tokenizer.pad_token_id, ) print(tokenizer.decode(output[0], skip_special_tokens=True))Raw Transformers may accept inputs beyond 2,048 tokens, but those lengths are unsupported. Callers must enforce the total input-plus-generation limit. This base model has no chat template. Sampling can reduce obvious greedy loops but does not make the model factual, aligned, or safe.
The published FP32 Safetensors weight SHA-256 is
64d286e3866a90608a3dedfa83774eef7fb45852ca392f85cbdb214a2df18f3d.
The native source checkpoint SHA-256 is
ab8d0c2ed12dc1cbf842b0d8ba7547c763d828a1711061d388940951d04340e1.
Fresh-process native execution passed lengths 1, 2, 17, 512, 1,024, 2,047 and 2,048 and rejected 2,049 through the native context guard. The export passed the declared bounded BF16 native-to-Transformers parity policy, including seven full-vocabulary logit probes and 64-token cached and recomputed greedy decision checks. This establishes tolerance-aware conversion parity, not bit-exact equality across frameworks.
See provenance.json, evaluation/summary.json, and checksums.sha256 for
machine-readable release evidence.
Use the repository Community tab to report reproducibility problems, suspected memorised personal information, or correction and removal requests. A confirmed issue may require replacing or withdrawing an affected release; selective deletion from already trained weights may not be technically possible.
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model_id = "pdjamez/Neeps-125M-Base"tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=False, dtype=torch.float32,).eval() prompt = "The weather in Scotland is"max_new_tokens = 64inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)if inputs.input_ids.shape[1] + max_new_tokens > 2048: raise ValueError("Neeps supports at most 2,048 total input/output tokens") with torch.inference_mode(): output = model.generate( **inputs, max_new_tokens=max_new_tokens, do_sample=True, temperature=0.8, top_p=0.9, repetition_penalty=1.1, pad_token_id=tokenizer.pad_token_id, ) print(tokenizer.decode(output[0], skip_special_tokens=True))