GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
pdjamez
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
pdjamez
Model Tree
Input Modalities
Output Modalities
Supported Functionality
| Property | Value |
|---|---|
| Transformers implementation | LlamaForCausalLM |
| Parameters | 350,439,552 |
| Layers | 20 |
| Hidden width | 1,152 |
| Attention heads | 18 |
| Key-value heads | 3 |
| Head dimension | 64 |
| Feed-forward width | 3,840 |
| Context length | 2,048 tokens |
| Vocabulary | 20,000 tokens |
| Activation | SwiGLU |
| Normalisation | RMSNorm, epsilon 1e-5 |
| Position encoding | RoPE, theta 10,000 |
| Embeddings | Tied input and output embeddings |
| Published precision | FP32 Safetensors |
Tatties was trained from scratch and contains no Meta Llama weights. It uses
the standard Transformers Llama implementation because its decoder
architecture, RMSNorm, RoPE, SwiGLU and grouped-query attention are
compatible. No custom code or trust_remote_code=True is required.
The model is intended for small-language-model research, education, base-model evaluation, continued-pretraining experiments and post-training research. It is not suitable as a factual authority, a safety-critical system, or an unsupervised user-facing assistant.
The model learned by causal next-token prediction. Training used a 2,048-token sequence length, 32,768-token global batch, seed 42, and Swift/MLX 0.32.2 on Apple Silicon. The stable run ended at step 213,892 (7,008,813,056 tokens). A separate 10,681-step cosine cooldown continued to step 224,573, reducing the learning rate from 3e-4 to 3e-5. This is approximately 21.00 training tokens per parameter. The stable checkpoint remains preserved as the continuation parent; the cooled release checkpoint is a separate branch.
| Training property | Value |
|---|---|
| Sequence length | 2,048 tokens |
| Global batch | 32,768 tokens |
| Stable-trunk learning rate | 3e-4 |
| Cooldown schedule | Cosine from 3e-4 to 3e-5 over 10,681 steps |
| Seed | 42 |
| Backend | Swift with MLX 0.32.2 |
| Last scheduled validation loss | 2.73698 nats/token at step 224,457 |
The last scheduled validation used 64 packed windows and preceded the final checkpoint by 116 updates; it is not a separately measured final-checkpoint test loss.
The training manifest is the same frozen FineWeb-Edu sample-100BT derived
stream used for Neeps 125M Base, with source revision
87f09149ef4734204d70ed1d046ddc9ca3f2b8f9. Preparation selected English
documents with language score at least 0.90, educational integer score at
least 3, and crawl years 2013–2022. Local processing applied automated
quality and high-risk screens, exact and near-duplicate filtering, a domain
cap, selected benchmark decontamination, deterministic splits, and
EOS-delimited packing. The train split contains 12,116,013 documents and
12,486,676,062 packed tokens; Tatties consumed the first 7,358,808,064 tokens
of the ordered stream. Training text is not included with this model.
The 20,000-token SentencePiece tokenizer was trained separately on an earlier
train-only corpus-v3 FineWeb-Edu-Dedup pilot. Its SHA-256 is
34353c32e8a7c0e8a7f0bb057db4035c5676669d071367d8beb25cc4f1a39f04.
Automated filtering is not a guarantee that training data is free of private, copyrighted, harmful, biased, or inaccurate material. FineWeb-Edu is labelled ODC-By 1.0 and derives from Common Crawl; individual source pages can have separate rights. The Apache-2.0 licence applies to these model files, not to the training text or third-party source pages. No training text is distributed. The prepared human-review packet has no recorded decisions; in particular, the 2,000 accepted-document sample, 500 rejected references, and 695 recorded credential/secret filter hits were not manually adjudicated for this release. The release owner explicitly accepted this residual risk; that decision is not a claim of legal or privacy clearance.
The following results are for the native MLX cooldown checkpoint. Choice-task
scores were recomputed from its recorded per-choice conditional log
probabilities using character-length normalisation, matching the published
Neeps scoring rule. The native evaluator's own accuracy field uses
token-length normalisation and is not the number shown in this table.
LAMBADA requires the exact greedy target and BLiMP compares grammatical pairs.
| Benchmark | Examples | Score |
|---|---|---|
| ARC-Challenge validation | 299 | 30.77% |
| ARC-Challenge full test | 1,172 | 28.50% |
| ARC-Easy validation | 570 | 47.19% |
| PIQA validation | 1,838 | 64.96% |
| HellaSwag validation | 10,042 | 38.54% |
| LAMBADA OpenAI test | 5,153 | 30.97% |
| BLiMP grammatical-pair accuracy | 67,000 | 81.19% |
These are development benchmarks, not pristine final holdouts. The benchmark files have been reused in project comparisons; the recorded corpus decontamination does not cover BLiMP or the full ARC-Challenge test. Scores depend on prompt, truncation, scoring rule, precision, and evaluator.
The comparison uses the published Neeps 125M Base v0.4-5b model card and the same frozen benchmark files. Neeps has 125,095,680 parameters and saw 5,000,003,584 training tokens; Tatties has 350,439,552 parameters and saw 7,358,808,064 tokens. All choice scores below use character-length-normalised conditional log-likelihood. The first four Tatties rows are from its FP32 Transformers export; the other three are from its native-MLX checkpoint.
| Benchmark | Neeps FP32 | Tatties | Tatties source | Difference |
|---|---|---|---|---|
| ARC-Challenge validation | 25.75% | 31.10% | FP32 export | +5.35 pp |
| ARC-Challenge full test | 26.19% | 28.92% | FP32 export | +2.73 pp |
| ARC-Easy validation | 40.53% | 46.84% | FP32 export | +6.31 pp |
| PIQA validation | 62.46% | 65.02% | FP32 export | +2.56 pp |
| HellaSwag validation | 32.73% | 38.54% | Native MLX | +5.81 pp |
| LAMBADA OpenAI test, exact greedy target | 28.10% | 30.97% | Native MLX | +2.87 pp |
| BLiMP grammatical-pair accuracy | 80.10% | 81.19% | Native MLX | +1.09 pp |
The four FP32 choice tasks used the same dataset hashes and evaluator family, but Neeps ran on Apple MPS with a 512-token evaluation cap while Tatties ran on CPU with a 2,048-token cap. The remaining three rows also differ in runtime/evaluator. The differences are therefore descriptive, not a controlled model-size effect; architecture, training exposure, and other conditions differ. None of these reused benchmarks is a pristine final holdout.
The native-MLX scores above and the four packaged-model checks use different runtimes. Do not treat the native scores as a complete packaged-model evaluation. A packed Tatties test loss, fixed-prompt looping/EOS rate, training-data extraction probe, and synthetic long-context retrieval score have not been established for this release; Neeps reports such probes, but their results cannot be transferred to Tatties.
Load the published model by its Hub ID, or replace model_id with a verified
local bundle path. No custom code is required.
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model_id = "pdjamez/Tatties-350M-Base"tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=False, dtype=torch.float32,).eval() prompt = "The weather in Scotland is"max_new_tokens = 64inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)if inputs.input_ids.shape[1] + max_new_tokens > 2048: raise ValueError("Tatties supports at most 2,048 total input/output tokens") with torch.inference_mode(): output = model.generate( **inputs, max_new_tokens=max_new_tokens, do_sample=True, temperature=0.8, top_p=0.9, repetition_penalty=1.1, pad_token_id=tokenizer.pad_token_id, ) print(tokenizer.decode(output[0], skip_special_tokens=True))Raw Transformers may accept inputs beyond 2,048 tokens, but those lengths are unsupported. Callers must enforce the total input-plus-generation limit. This base model has no chat template. Sampling may reduce visible repetition, but does not make the model factual, aligned, or safe.
The published FP32 Safetensors weight SHA-256 is
5d7f1ac1a3735c79a5e002ac7825ecfdb49fa3c946e8c5d73fd3de64652af8da.
The native source checkpoint SHA-256 is
e26086bc1c1eab4b81f0decc1cb4164f1aee0d9b9cee4c6b70e00fe2f4abd729.
Fresh-process native execution tested lengths 1, 2, 17, 512, 1,024, 2,047
and 2,048 tokens and rejected 2,049 through the native context guard. The
staged FP32 export passed the explicit Tatties probability-and-decision parity
policy: exact tokenizer IDs, seven full-vocabulary probes spanning 1–2,048
tokens, 64 exact greedy generated tokens with both cached and recomputed
attention, and the 2,047+1 boundary decision. Across 3,879 paired ARC and PIQA
questions, token-normalised choices agreed 99.18% and character-normalised
choices agreed 98.69%. The maximum native-to-export KL divergence among the
seven probes was 0.00637 nats. This is tolerance-aware conversion parity,
not bit-exact equality across frameworks. See provenance.json,
evaluation/functional-parity.json, and checksums.sha256 for release evidence.
The release policy tests the FP32 artifact actually staged here. It checks probability distributions and token decisions rather than requiring raw-logit equality; a shared additive shift in raw logits does not change a softmax distribution. Historical BF16 diagnostics used for Neeps are retained in the local engineering record but are not the acceptance criterion for this FP32 release.
Use this model repository's Community tab to report reproducibility problems, suspected memorised personal information, rights concerns, or correction and removal requests. Do not post sensitive text, credentials, or personal details publicly; ask for a private channel in the initial report. A confirmed issue may require replacing or withdrawing the release; selective deletion from trained weights may not be technically possible.
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model_id = "pdjamez/Tatties-350M-Base"tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=False, dtype=torch.float32,).eval() prompt = "The weather in Scotland is"max_new_tokens = 64inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)if inputs.input_ids.shape[1] + max_new_tokens > 2048: raise ValueError("Tatties supports at most 2,048 total input/output tokens") with torch.inference_mode(): output = model.generate( **inputs, max_new_tokens=max_new_tokens, do_sample=True, temperature=0.8, top_p=0.9, repetition_penalty=1.1, pad_token_id=tokenizer.pad_token_id, ) print(tokenizer.decode(output[0], skip_special_tokens=True))