GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
AlbinoRacoon
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
AlbinoRacoon
Model Tree
Input Modalities
Output Modalities
Supported Functionality
| Type | Dense decoder-only transformer (Llama architecture), base model |
| Parameters | 1,338,853,376 |
| Layers / hidden size / heads | 20 / 2048 / 16 (MLP 5632, RoPE θ = 10,000, tied embeddings) |
| Context length | 4,096 tokens |
| Initialization | Random. Not distilled from, and not continued from, any other model |
| Training | ≈ 31.8 B tokens (step 1,939,850), about 31.8 % of the planned 100 B; stopped early for cost reasons, before any learning-rate decay |
| Compute | ≈ 740 GPU-hours on NVIDIA A100 40 GB (2 GPUs per job), HPC Vega (IZUM, Slovenia) |
| Tokenizer | Reused from Qwen/Qwen3-Coder-30B-A3B-Instruct (151,669 tokens; embedding matrix padded to 151,936). Not trained from scratch. |
| Languages | Slovenian, English, Python, ROS/ROS2 code |
| Weights | bfloat16 safetensors, 2.7 GB |
| License | Apache-2.0 (see the note on training-data terms below) |
python
import torchfrom transformers import AutoTokenizer, AutoModelForCausalLM repo = "AlbinoRacoon/martin-1.3b-base"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16) inputs = tok("Slovenija je država v srednji Evropi, ki", return_tensors="pt")output = model.generate(**inputs, max_new_tokens=80)print(tok.decode(output[0], skip_special_tokens=True))generation_config.json already sets the decoding that worked best for us: sampling with temperature=0.8, top_p=0.9, repetition_penalty=1.2. In our tests greedy decoding tends to repeat sentences.
An unedited sample (sampling, seed 0):
Slovenija je država v srednji Evropi, ki spada med države, kjer živijo številne etnično in kulturne razlike. Državo Sloveniji predstavlja približno 68% prebivalcev. Leta 1950 se je število prebivalcev Sloven…
The text is fluent and grammatical, but the content is invented. This is typical of the model.
Sampling mixture of the planned run:
| Source | Share |
|---|---|
HuggingFaceFW/fineweb-edu (English web text) | 50 % |
bigcode/starcoderdata, Python subset | 20 % |
HuggingFaceFW/fineweb-2, slv_Latn (Slovenian web text) | 14 % |
cjvt/Wikipedia-translated (machine-translated Wikipedia in Slovenian) | 8 % |
cjvt Nemotron translated subsets (Slovenian) | 3 % |
| Own ROS/ROS2 corpus from GitHub repositories (only Apache-2.0, MIT and BSD-2/3 licensed), oversampled | 5 % |
Slovenian therefore makes up about a quarter of the mixture. The upstream datasets keep their own terms (for example ODC-By for FineWeb and the StarCoder data terms). The weights are released under Apache-2.0, but you are responsible for checking that your use is compatible with those terms.
The evaluations come from a master's thesis and are small and informal. None of them is a rigorous benchmark.
What we observed
rospy, and produced fewer invented Slovenian words.What it does badly
Human preference (pilot). In a blind A/B arena against GaMS-2B (4 voters, very small sample) it won 7, lost 9 and tied 9 comparisons (4 skipped). This is indicative only.
Intended: research on pretraining low-resource-language and domain-specific models under a small budget, and a starting point for continued pretraining or instruction tuning. Instruction tuning is the clearly missing step.
Not suitable for: assistants, factual question answering, safety-critical use, controlling robots, or any production deployment. Generated code must be treated as unverified.
The model can produce incorrect, fabricated, repetitive, biased or unsafe text. The checkpoint is under-trained and was saved while the learning rate was still high.
<|im_end|> (id 151645) during training, and generation_config.json uses it as the EOS token. An earlier research configuration used id 2, which is the character # in this vocabulary, and has been corrected here.chat_template was removed from tokenizer_config.json, because the model was never trained on a chat format. With some transformers versions a warning about a Mistral regex appears. For this Qwen tokenizer it did not change the token ids on the strings we compared.transformers 4.57.6. The architecture is a standard LlamaForCausalLM.model.safetensors SHA-256 b234fa096768c6620fffac903237e261183b1d450b37efd2849a25e71776e64bDeveloped as part of a master's thesis project at UM FERI (University of Maribor, Slovenia) for the FarmBeast agricultural robot. Computing resources were provided by HPC Vega (IZUM). The tokenizer comes from the Qwen team (Apache-2.0).
Companion model from the same project: AlbinoRacoon/velik-martin-27b-lora, a LoRA adapter for Qwen3.6-27B.
import torchfrom transformers import AutoTokenizer, AutoModelForCausalLM repo = "AlbinoRacoon/martin-1.3b-base"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16) inputs = tok("Slovenija je država v srednji Evropi, ki", return_tensors="pt")output = model.generate(**inputs, max_new_tokens=80)print(tok.decode(output[0], skip_special_tokens=True))