GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
ltg
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
ltg
Model Tree
norallm/normistral-11b-warm
Input Modalities
Output Modalities
Supported Functionality
We release the model under Apache 2.0 license to indicate that we do not impose any additional constraints on the model weights. However, we do not own the data in the training collection.
The model is pretrained on a combination of publicly available data and a custom web crawl for Sámi. The total training corpus consists of 250 billion tokens from the following sources:
Norwegian text (Bokmål and Nynorsk); this collection was created by the National Library of Norway and it's a prerelease of an update of NCC (codenamed "Mímir core"). It consists of: a) the public part of Norwegian Colossal Corpus (NCC) with permissible licenses (i.e. it doesn't include newspaper texts with the CC BY-NC 2.0 license); b) Bokmål and Nynorsk CulturaX, and c) Bokmål and Nynorsk HPLT corpus v1.2.
Northern Sámi texts are sourced from a) Glot500; b) the SIKOR North Saami free corpus; and c) a custom web crawl (seeded from Sámi Wikipedia external links) published separately as ltg/saami-web.
Additional languages for knowledge/language transfer: a) Danish, Swedish, Icelandic, and Faroese from CulturaX and Glot500; b) high-quality English from FineWeb-edu; and c) programming code from The Stack v2 (the high-quality subset).
The corpus is carefully balanced through strategic upsampling to handle the resource disparity between languages. Following data-constrained scaling laws, the corpus data for target languages is repeated multiple times (up to 16x for low-resource languages) to reach the optimal training budget while avoiding overfitting:
This model uses a new tokenizer, specially trained on the target languages. Therefore it offers substantially faster inference than the original Mistral-Nemo-Base-2407 model. Here are the subword-to-word split ratios across different languages:
| Tokenizer | # tokens | Bokmål | Nynorsk | Sámi | Danish | Swedish |
|---|---|---|---|---|---|---|
| Mistral-Nemo-Base-2407 | 131072 | 1.79 | 1.87 | 2.63 | 1.82 | 2.00 |
| NorMistral-11b-long | 51200 | 1.22 | 1.28 | 1.82 | 1.33 | 1.39 |
Model Developers: Language Technology Group at the University of Oslo in collaboration with NORA.LLM.
Architecture: NorMistral-11B uses the Mistral architecture based on an improved Llama design, featuring:
Training Details:
Base Model: Initialized from NorMistral-11b-warm
License: Apache-2.0
Here's how to use NorMistral-11B as a standard causal language model for translation:
python
from transformers import AutoTokenizer, AutoModelForCausalLMimport torch# Import the tokenizer and modeltokenizer = AutoTokenizer.from_pretrained("norallm/normistral-11b-long")model = AutoModelForCausalLM.from_pretrained("norallm/normistral-11b-long").cuda().eval()# Define zero-shot translation prompt templateprompt = """Engelsk: {0}Bokmål:"""# Define tokens that should end the generation (any token with a newline)eos_token_ids = [token_idfor token_id in range(tokenizer.vocab_size)if '\n' in tokenizer.decode([token_id])]# Generation function@torch.no_grad()def generate(text):text = prompt.format(text)input_ids = tokenizer(text, return_tensors='pt').input_ids.cuda()prediction = model.generate(input_ids,max_new_tokens=64,do_sample=False,eos_token_id=eos_token_ids)return tokenizer.decode(prediction[0, input_ids.size(1):]).strip()# Example usagegenerate("I'm excited to try this new Norwegian language model!")# > Expected output: 'Jeg er spent på å prøve denne nye norske språkmodellen!'
For systems with limited VRAM, you can load the model in 8-bit or 4-bit quantization:
python
import torchfrom transformers import AutoTokenizer, AutoModelForCausalLMtokenizer = AutoTokenizer.from_pretrained("norallm/normistral-11b-long")# Load in 8-bit mode (requires ~12GB VRAM)model = AutoModelForCausalLM.from_pretrained("norallm/normistral-11b-long",device_map='auto',load_in_8bit=True,torch_dtype=torch.bfloat16)# Or load in 4-bit mode (requires ~8GB VRAM)model = AutoModelForCausalLM.from_pretrained("norallm/normistral-11b-long",device_map='auto',load_in_4bit=True,torch_dtype=torch.bfloat16)
bibtex
@inproceedings{samuel-etal-2025-small,title = "Small Languages, Big Models: {A} Study of Continual Training on Languages of {Norway}",author = "Samuel, David andMikhailov, Vladislav andVelldal, Erik and{\O}vrelid, Lilja andCharpentier, Lucas Georges Gabriel andKutuzov, Andrey andOepen, Stephan",editor = "Johansson, Richard andStymne, Sara",booktitle = "Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025)",month = mar,year = "2025",address = "Tallinn, Estonia",publisher = "University of Tartu Library",url = "https://aclanthology.org/2025.nodalida-1.61/",pages = "573--608",ISBN = "978-9908-53-109-0",}
Please write a community message or contact David Samuel (davisamu@ifi.uio.no) if you have any questions about this model.