xslm API & Inference Endpoint

Limitations

This model does not hold world knowledge. A model of this size learns grammar, style, and document structure. It invents facts. This is the expected result at this scale, and it is not a defect to report.

For a model that answers questions correctly, use retrieval-augmented generation over a document index, or a QLoRA fine-tune of an existing 1B to 8B base model.

Architecture

Table with columns: Field, Value
Field	Value
Non-embedding parameters	49,575,040
Total parameters	70,546,560
`hidden_size`	640
`num_hidden_layers`	10
`num_attention_heads`	10
`intermediate_size`	1728
`vocab_size`	32768
`max_position_embeddings`	1024
Normalization	RMSNorm
Position encoding	RoPE, theta 10000.0
Activation	SwiGLU
Embeddings	Tied input and output

Training

Table with columns: Field, Value
Field	Value
Corpus	`HuggingFaceFW/fineweb-edu`, `sample-10BT`, ODC-By license
Tokenizer	Byte-level BPE, 32768 tokens
Optimizer	AdamW, betas 0.9 and 0.95
Peak learning rate	0.001
Schedule	200 warmup steps, then cosine decay to 0.0001
Tokens for each optimizer step	262,144
Precision	BF16 mixed
Hardware

Files

Table with columns: File, Use
File	Use
`model.safetensors`	The BF16 weights, for `transformers`
`gguf/xslm-70m-BF16.gguf`	The unquantized GGUF file, in the training dtype
`gguf/xslm-70m-F16.gguf`	The unquantized GGUF file, for the tools that read no BF16
`gguf/xslm-70m-Q8_0.gguf`	The recommended quantized file
`gguf/xslm-70m-Q6_K.gguf`	Smaller, with a visible quality cost

At 70M parameters there is no redundancy to absorb quantization error. Q4_K_M often produces broken text. Use Q8_0.

Use

python
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Alienstro/xslm")
model = AutoModelForCausalLM.from_pretrained("Alienstro/xslm")

Limitations

For a model that answers questions correctly, use retrieval-augmented generation over a document index, or a QLoRA fine-tune of an existing 1B to 8B base model.

Architecture

Table with columns: Field, Value
Field	Value
Non-embedding parameters	49,575,040
Total parameters	70,546,560
`hidden_size`	640
`num_hidden_layers`	10
`num_attention_heads`	10
`intermediate_size`	1728
`vocab_size`	32768
`max_position_embeddings`	1024
Normalization	RMSNorm
Position encoding	RoPE, theta 10000.0
Activation	SwiGLU
Embeddings	Tied input and output

Training

Table with columns: Field, Value
Field	Value
Corpus	`HuggingFaceFW/fineweb-edu`, `sample-10BT`, ODC-By license
Tokenizer	Byte-level BPE, 32768 tokens
Optimizer	AdamW, betas 0.9 and 0.95
Peak learning rate	0.001
Schedule	200 warmup steps, then cosine decay to 0.0001
Tokens for each optimizer step	262,144
Precision	BF16 mixed
Hardware

Files

Table with columns: File, Use
File	Use
`model.safetensors`	The BF16 weights, for `transformers`
`gguf/xslm-70m-BF16.gguf`	The unquantized GGUF file, in the training dtype
`gguf/xslm-70m-F16.gguf`	The unquantized GGUF file, for the tools that read no BF16
`gguf/xslm-70m-Q8_0.gguf`	The recommended quantized file
`gguf/xslm-70m-Q6_K.gguf`	Smaller, with a visible quality cost

At 70M parameters there is no redundancy to absorb quantization error. Q4_K_M often produces broken text. Use Q8_0.

Use

python
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Alienstro/xslm")
model = AutoModelForCausalLM.from_pretrained("Alienstro/xslm")

xslm

README

Limitations

Architecture

Training

Files

Use

Explore FriendliAI today

README

Limitations

Architecture

Training

Files

Use