GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
nakue
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
nakue
Model Tree
HuggingFaceTB/SmolLM2-1.7B-Instruct
Input Modalities
Output Modalities
Supported Functionality
| Property | Value |
|---|---|
| Base model | HuggingFaceTB/SmolLM2-1.7B-Instruct |
| Architecture | LlamaForCausalLM |
| Parameters | ~1.7B |
| Quantization scheme | W4A16 — INT4 weights, BF16 activations |
| Excluded layers | lm_head (kept in BF16) |
| Format | compressed-tensors (Safetensors) |
| Calibration dataset | ultrachat (512 samples, max_seq_length 2048) |
| Quantization tool | llm-compressor |
| Scheme | Weight bits | Activation bits | Memory saving | Compute speedup | Best for |
|---|---|---|---|---|---|
| BF16 (base) | 16 | 16 | — | — | Accuracy baseline |
| W4A16 (this model) | 4 | 16 | ~4x | Memory-bound only | Small GPUs, low-latency single user |
| W8A16 |
Key insight: W4A16 trades a bit more accuracy than W8A8 for a larger memory reduction (~4x vs ~2x). It does not use INT4 tensor cores at runtime — the dequantize-then-multiply pattern keeps compute in BF16. Choose W4A16 when fitting the model matters more than maximizing throughput.
bash
pip install vllmpython
from vllm import LLM, SamplingParamsfrom transformers import AutoTokenizer model_id = "nakue/SmolLM2-1.7B-W4A16-instruct" llm = LLM( model=model_id, quantization="compressed-tensors", dtype="bfloat16",) sampling_params = SamplingParams(temperature=0.7, max_tokens=256) tokenizer = AutoTokenizer.from_pretrained(model_id)messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the difference between W4A16 and W8A8 quantization?"},]prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) outputs = llm.generate([prompt], sampling_params)print(outputs[0].outputs[0].text)Install the compressed-tensors runtime — Transformers auto-detects the quantization config from config.json:
bash
pip install compressed-tensors transformerspython
from transformers import AutoModelForCausalLM, AutoTokenizerimport torch model_id = "nakue/SmolLM2-1.7B-W4A16-instruct"tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto",) messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain quantization in simple terms."},] inputs = tokenizer.apply_chat_template( messages, return_tensors="pt", add_generation_prompt=True, return_dict=True,).to(model.device) outputs = model.generate(**inputs, max_new_tokens=128)print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))Note: W4A16 dequantizes weights to BF16 at runtime — compute stays in BF16. You get ~4x memory reduction; latency gains depend on whether your workload is memory-bandwidth-bound.
bash
pip install llmcompressorpython
from llmcompressor.transformers import SparseAutoModelForCausalLMfrom transformers import AutoTokenizerimport torch model_id = "nakue/SmolLM2-1.7B-W4A16-instruct"tokenizer = AutoTokenizer.from_pretrained(model_id) model = SparseAutoModelForCausalLM.from_pretrained( model_id, torch_dtype="auto", device_map="auto",) inputs = tokenizer("Explain INT4 quantization:", return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=128)print(tokenizer.decode(outputs[0], skip_special_tokens=True))If you need to run with plain Transformers and zero extra dependencies:
bash
pip install llmcompressorpython
from llmcompressor.transformers import SparseAutoModelForCausalLMimport torch model = SparseAutoModelForCausalLM.from_pretrained( "nakue/SmolLM2-1.7B-W4A16-instruct", torch_dtype=torch.bfloat16,)model.save_pretrained("smollm2-bf16-dequantized")tokenizer.save_pretrained("smollm2-bf16-dequantized")Then load smollm2-bf16-dequantized with plain AutoModelForCausalLM. Memory savings are lost but there are zero runtime dependencies.
python
from llmcompressor.transformers import SparseAutoModelForCausalLM, oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifierfrom transformers import AutoTokenizer model_id = "HuggingFaceTB/SmolLM2-1.7B-Instruct"tokenizer = AutoTokenizer.from_pretrained(model_id)model = SparseAutoModelForCausalLM.from_pretrained( model_id, torch_dtype="auto", device_map="auto") recipe = QuantizationModifier( targets="Linear", scheme="W4A16", ignore=["lm_head"], # keep output projection in BF16) oneshot( model=model, dataset="ultrachat", recipe=recipe, max_seq_length=2048, num_calibration_samples=512,) model.save_pretrained("SmolLM2-1.7B-W4A16-instruct")tokenizer.save_pretrained("SmolLM2-1.7B-W4A16-instruct")⚠️ Evaluation pending. Accuracy vs. the BF16 base has not yet been formally benchmarked. Results will be added once lm-evaluation-harness evals complete.
Planned evaluation setup:
bash
# BF16 baselinelm_eval --model hf \ --model_args "pretrained=HuggingFaceTB/SmolLM2-1.7B-Instruct,dtype=bfloat16" \ --tasks hellaswag,winogrande,arc_easy,arc_challenge,piqa,wikitext \ --num_fewshot 0 --batch_size 32 --output_path results/baseline # W4A16lm_eval --model hf \ --model_args "pretrained=nakue/SmolLM2-1.7B-W4A16-instruct,dtype=bfloat16" \ --tasks hellaswag,winogrande,arc_easy,arc_challenge,piqa,wikitext \ --num_fewshot 0 --batch_size 32 --output_path results/w4a16Results table (to be filled):
| Task | BF16 Base | W4A16 (this model) | Delta |
|---|---|---|---|
| HellaSwag (acc_norm) | — | — | — |
| WinoGrande (acc) | — | — | — |
| ARC-Easy (acc_norm) | — | — | — |
| ARC-Challenge (acc_norm) | — | — | — |
| PIQA (acc_norm) |
ultrachat — accuracy may degrade on domains far from the calibration distribution.lm_head excluded — the output projection is kept in BF16 to preserve logit precision.| Situation | Recommended model |
|---|---|
| GPU with < 4GB VRAM | W4A16 (this model) |
| Maximum memory savings matter | W4A16 (this model) |
| Single-user low-latency inference | W4A16 (this model) |
| High-throughput batched serving | W8A8 version |
| Accuracy is the priority | W8A8 version |
| INT8 tensor core compute speedup needed | W8A8 version |
| Model | Scheme | Size | Link |
|---|---|---|---|
| SmolLM2-1.7B-Instruct (base) | BF16 | ~3.4GB | HuggingFaceTB/SmolLM2-1.7B-Instruct |
| SmolLM2-1.7B-W8A8-Instruct | INT8 W+A | ~1.7GB | nakue/SmolLM2-1.7B-W8A8-instruct |
| SmolLM2-1.7B-W4A16-Instruct | INT4 W | ~0.85GB |
Apache 2.0 — inherited from the base model. See LICENSE.
bibtex
@misc{smollm2, title = {SmolLM2: When Smol Goes Big}, author = {HuggingFaceTB}, year = {2024}, url = {https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct}}Quantized by nakue · Portfolio · Part of an LLM inference optimization portfolio targeting production serving patterns.
| 8 |
| 16 |
| ~2x |
| Memory-bound only |
| Mild memory pressure |
| W8A8 | 8 | 8 | ~2x | ✅ Compute (INT8 cores) | High-throughput batched serving |
| — |
| — |
| — |
| WikiText-2 PPL ↓ | — | — | — |