GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
nakue
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
nakue
Model Tree
HuggingFaceTB/SmolLM2-1.7B-Instruct
Input Modalities
Output Modalities
Supported Functionality
| Property | Value |
|---|---|
| Base model | HuggingFaceTB/SmolLM2-1.7B-Instruct |
| Architecture | LlamaForCausalLM |
| Parameters | ~1.7B |
| Quantization | W8A8 (INT8 weights + INT8 activations) |
| Format | compressed-tensors (Safetensors) |
| Calibration dataset | ultrachat (512 samples) |
| Quantization tool | llm-compressor |
W8A8 quantization reduces memory footprint and enables use of INT8 tensor core throughput on modern NVIDIA GPUs, without the accuracy degradation typical of weight-only schemes like W4A16. This model is useful for:
python
from vllm import LLM, SamplingParams llm = LLM( model="nakue/SmolLM2-1.7B-W8A8-instruct", quantization="compressed-tensors", dtype="bfloat16",) sampling_params = SamplingParams(temperature=0.7, max_tokens=256) messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain what W8A8 quantization means."},] # Apply chat templatefrom transformers import AutoTokenizertokenizer = AutoTokenizer.from_pretrained("nakue/SmolLM2-1.7B-W8A8-instruct")prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) outputs = llm.generate([prompt], sampling_params)print(outputs[0].outputs[0].text)python
from transformers import AutoModelForCausalLM, AutoTokenizerimport torch model_id = "nakue/SmolLM2-1.7B-W8A8-instruct"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto") messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"},] inputs = tokenizer.apply_chat_template( messages, return_tensors="pt", add_generation_prompt=True, return_dict=True,).to(model.device) outputs = model.generate(**inputs, max_new_tokens=128)print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))ultrachat); accuracy may degrade on domains far from calibration distribution.lm_head is excluded from quantization (left in BF16) to preserve output logit precision.compressed-tensors support; Transformers inference falls back to dequantized BF16.This model inherits the Apache 2.0 license from the base model.
If you use this model, please also cite the original SmolLM2:
text
@misc{smollm2, title={SmolLM2: When Smol Goes Big}, author={HuggingFaceTB}, year={2024}, url={https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct}}Quantized by nakue as part of an LLM inference optimization portfolio.
from vllm import LLM, SamplingParams llm = LLM( model="nakue/SmolLM2-1.7B-W8A8-instruct", quantization="compressed-tensors", dtype="bfloat16",) sampling_params = SamplingParams(temperature=0.7, max_tokens=256) messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain what W8A8 quantization means."},] # Apply chat templatefrom transformers import AutoTokenizertokenizer = AutoTokenizer.from_pretrained("nakue/SmolLM2-1.7B-W8A8-instruct")prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) outputs = llm.generate([prompt], sampling_params)print(outputs[0].outputs[0].text)from transformers import AutoModelForCausalLM, AutoTokenizerimport torch model_id = "nakue/SmolLM2-1.7B-W8A8-instruct"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto") messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"},] inputs = tokenizer.apply_chat_template( messages, return_tensors="pt", add_generation_prompt=True, return_dict=True,).to(model.device) outputs = model.generate(**inputs, max_new_tokens=128)print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))@misc{smollm2, title={SmolLM2: When Smol Goes Big}, author={HuggingFaceTB}, year={2024}, url={https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct}}