GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
fbaldassarri
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
fbaldassarri
Model Tree
TinyLlama/TinyLlama_v1.1
Input Modalities
Output Modalities
Supported Functionality
Quantized version of TinyLlama/TinyLlama_v1.1 using torch.bfloat16 for quantization tuning.
Fast and low memory, 2-3X speedup (slight accuracy drop at W4G64)
Quantization framework: Intel AutoRound v0.13.0
Note: this INT4 version of TinyLlama_v1.1 has been quantized for inference on Intel CPU, Intel iGPU (Arc) via intel-extension-for-pytorch, Intel NPU (AI Boost on Core Ultra series) via OpenVINO.
This is a base / completion model — prompt it directly with raw text:
python
from transformers import AutoModelForCausalLM, AutoTokenizerrepo = "fbaldassarri/TinyLlama_TinyLlama_v1.1-auto_gptq-int4-gs64-asym"model = AutoModelForCausalLM.from_pretrained(repo, device_map="auto")tokenizer = AutoTokenizer.from_pretrained(repo)prompt = "The quick brown fox"inputs = tokenizer(prompt, return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=64)print(tokenizer.decode(outputs[0], skip_special_tokens=True))
The recommended way to reproduce this exact quantization is via the auto-round-pipeline — the same orchestration tool that produced this artifact.
Set up a dedicated conda environment using the pipeline's setup.sh. Any of the install modes below produces an environment that can reproduce this quantization; pick the one that matches your goals:
markdown
git clone https://git.epicdynamic.com/auto-round-pipelinecd auto-round-pipeline# Pinned PyPI wheel (fastest; matches what this pipeline used by default):bash setup.sh --pip-version 0.13.0# Or build from intel/auto-round at the same tag (byte-identical reproducibility):bash setup.sh --source-tag v0.13.0# Intel Arc iGPU acceleration (e.g. Core Ultra 185H) — append to either of the above:# ... --intel-xpu# NVIDIA / AMD opt-in: --cuda / --rocm
The script prints the resulting conda env name (something like auto-round-pipeline-v0.13.0[-src][-xpu|-cuda|-rocm]) at the end.
Activate the env that setup.sh created, then invoke the runner with the same job filters that produced this artifact:
markdown
conda activate <env-name-printed-by-setup.sh>python runner.py \--model 'TinyLlama/TinyLlama_v1.1' \--quant 'INT4-gs64' \--format auto_gptq \--no-upload # drop this to also push to HuggingFace Hub
If you'd rather call auto-round directly without the orchestration wrapper, this is the exact call the pipeline made:
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom auto_round import AutoRoundmodel_name = "TinyLlama/TinyLlama_v1.1"model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)tokenizer = AutoTokenizer.from_pretrained(model_name)bits, group_size, sym = 4, 64, Falseautoround = AutoRound(model, tokenizer,bits=bits, group_size=group_size, sym=sym,device_map="cpu",nsamples=128, iters=200, seqlen=512, batch_size=4,)autoround.quantize_and_save("./AutoRound/TinyLlama_TinyLlama_v1.1-auto_gptq-int4-gs64-asym", format="auto_gptq")
Recorded by the auto-round-pipeline at quantization time:
| Field | Value |
|---|---|
| Intel auto-round version | 0.13.0 |
| transformers version | 4.55.3 |
| torch version | 2.12.0+cpu |
| torch_dtype (load) | torch.bfloat16 |
| calibration device | cpu |
| calibration samples | 128 |
| tuning iterations | 200 |
| calibration seq len | 512 |
| calibration batch size |
This quantized model comes with no warranty. It has been developed only for research purposes.
| 4 |
| quantization duration | 8018.3s (133.6 min) |
| completed at (UTC) | 2026-06-23T05:23:48.370785+00:00 |