GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
letechlead
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
letechlead
Model Tree
ornith-ai/Ornith-1.5-9B
Input Modalities
Output Modalities
Supported Functionality
auto_round:auto_gptqNeelNanda/pile-10kThe language-model layers are quantized to INT4. The visual tower and other unsupported non-text layers remain in their original higher-precision format.
python
from transformers import AutoProcessor, AutoModelForImageTextToTextmodel_id = "letechlead/Ornith-1.5-9B-INT4-W4A16-AutoRound"processor = AutoProcessor.from_pretrained(model_id)model = AutoModelForImageTextToText.from_pretrained(model_id,device_map="auto",torch_dtype="auto",)inputs = processor(text="Explain quantization in one sentence.",return_tensors="pt",)inputs = {k: v.to(model.device) if hasattr(v, "to") else v for k, v in inputs.items()}output = model.generate(**inputs, max_new_tokens=64)print(processor.batch_decode(output, skip_special_tokens=True)[0])
The language-model values were not re-quantized. For standard vLLM deployment, use the compatibility-repacked checkpoint layout generated from this artifact. The repackaging normalizes duplicated tensor-name prefixes and tokenizer metadata, keeps the language model INT4/W4A16, and restores the vision tower in its original higher-precision format. Vision input remains supported.
Example deployment arguments:
bash
vllm serve letechlead/Ornith-1.5-9B-INT4-W4A16-AutoRound --tensor-parallel-size 2 --trust-remote-code --quantization auto_round --served-model-name Ornith-1.5-9B-INT4-W4A16-AutoRound
The artifact was verified with Transformers and standard vLLM. Text generation and an image-input request both completed successfully with the compatibility-repacked layout.
This is a derived quantized artifact of ornith-ai/Ornith-1.5-9B. Please follow the upstream model's MIT license and review the upstream model card for intended use, limitations, and full attribution.