GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
spahbod
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
spahbod
Model Tree
HuggingFaceTB/SmolLM2-135M
Input Modalities
Output Modalities
Supported Functionality
| Property | Base model | DUS model |
|---|---|---|
| Architecture | LlamaForCausalLM | LlamaForCausalLM |
| Decoder layers | 30 | 40 |
| Parameters | 134,515,008 | 169,915,968 |
| Added parameters | — | 35,400,960 |
| Parameter increase | — | 26.32% |
| Base model | HuggingFaceTB/SmolLM2-135M | HuggingFaceTB/SmolLM2-135M |
The 40-layer model was created from two overlapping slices of the original model:
The copied layers contain independent parameter tensors.
The model was trained using continued causal language-model pretraining.
Salesforce/wikitextwikitext-2-raw-v1The run configured for five epochs was stopped automatically at approximately epoch 1.37 because the validation loss stopped improving.
| Model stage | Loss | Perplexity |
|---|---|---|
| Base model | 3.2635 | 26.14 |
| Immediately after DUS | 3.5069 | 33.34 |
| After continued pretraining | 2.7956 | 16.37 |
Depth Up-Scaling initially increased loss. Continued pretraining subsequently adapted the new architecture and produced lower loss and perplexity on WikiText-2 than the original model.
The models were evaluated using lm-evaluation-harness in the zero-shot setting.
| Benchmark | Metric | Base model | DUS model |
|---|---|---|---|
| HellaSwag | acc_norm | 0.4312 | 0.4125 |
| LAMBADA OpenAI | acc | 0.4283 | 0.3775 |
| LAMBADA OpenAI | perplexity | 19.2598 | 27.4383 |
| PIQA | acc_norm | 0.6844 | 0.6518 |
Although the DUS model achieved better loss and perplexity on WikiText-2, it performed worse on all three external benchmarks. This suggests stronger adaptation to the continued-pretraining dataset accompanied by reduced generalization and possible catastrophic forgetting.
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model_id = "spahbod/smollm2-dus-40l" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.float16,).to("cuda") prompt = "The history of artificial intelligence" inputs = tokenizer(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): output = model.generate( **inputs, max_new_tokens=100, do_sample=True, temperature=0.8, top_p=0.95, ) print(tokenizer.decode(output[0], skip_special_tokens=True))For CPU inference, remove .to("cuda") and load the model using torch_dtype=torch.float32.
This model is intended for:
It is not an instruction-tuned or chat model.
Training code is available at:
https://github.com/s25132/smollm2-dus
This model is derived from HuggingFaceTB/SmolLM2-135M, released under the Apache 2.0 license.
The experiment uses the WikiText dataset and the EleutherAI Language Model Evaluation Harness.
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model_id = "spahbod/smollm2-dus-40l" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.float16,).to("cuda") prompt = "The history of artificial intelligence" inputs = tokenizer(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): output = model.generate( **inputs, max_new_tokens=100, do_sample=True, temperature=0.8, top_p=0.95, ) print(tokenizer.decode(output[0], skip_special_tokens=True))