GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
ishaqinu
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
ishaqinu
Model Tree
Qwen/Qwen3.5-9B
Input Modalities
Output Modalities
Supported Functionality
| Property | Details |
|---|---|
| Base model | Qwen/Qwen3.5-9B |
| Architecture | Qwen3_5ForConditionalGeneration |
| Inputs / output | Text and images / generated text |
| Format | Seven Safetensors shards; mixed FP16 and FP8 E4M3 with compressed-tensors metadata |
| Scaling | Static per-channel weights; dynamic per-token activations |
| Configured context | 262,144 tokens; usable context depends on memory and runtime |
| Stored tensor payload | 13,520,091,616 bytes: approximately 13.52 GB / 12.59 GiB |
| Access / license | Public and ungated / Apache-2.0 |
These are potential uses of the underlying general-purpose model. Performance on each task should be checked for your workload.
| Task | Example prompt |
|---|---|
| Visual understanding and image captioning | "Describe this scene and the relationships between the visible objects." |
| OCR / text recognition | "Transcribe the visible text in this image." |
| Document understanding | "Read this invoice image and identify the date and total." |
| Visual question answering | "What is the person holding?" |
| Chart understanding | "Explain the main trend and identify the labels supporting your answer." |
| Object counting and spatial reasoning | "How many cups are visible, and which is closest to the keyboard?" |
| Text generation | "Explain this concept in simple language." |
The upstream architecture also supports video; video inference is not covered by these examples or the publisher's reported comparison.
Publisher's informal observation: in informal tests, ishaqinu observed qualitatively similar performance to the full-precision Qwen3.5-9B model on the visual understanding and text recognition (OCR) examples they tried.
This is an informal observation, not a measured benchmark result or evidence of equal accuracy across tasks. No quantitative scores, dataset, sample count, comparison protocol, or test environment are currently published. Formal evaluation results will be added when available; upstream benchmark scores should not be presented as scores for this quantization.
Generated answers can misread text, hallucinate details, or give incorrect counts and spatial descriptions. Keeping vision components in FP16 does not guarantee unchanged end-to-end accuracy because quantized language layers contribute to the final answer.
For Linux x86_64 and Python 3.12, download requirements-linux-py312.lock, then create a separate environment:
bash
python3.12 -m venv .venvsource .venv/bin/activatepython -m pip install --require-hashes --only-binary=:all: --index-url https://pypi.org/simple -r requirements-linux-py312.lockThe lock is a candidate environment. See DEPENDENCIES.md for exact versions, platform scope, and validation status. It is not the recorded environment used for the publisher's informal tests. Check PyTorch/CUDA compatibility with your GPU and driver using the official PyTorch guide.
Put example.jpg in your working directory. Save the following as image.py, or download the identical standalone example, and run python image.py:
python
from pathlib import Path import torchfrom transformers import ( AutoProcessor, CompressedTensorsConfig, Qwen3_5ForConditionalGeneration,) image_path = Path("example.jpg").resolve()if not image_path.is_file(): raise FileNotFoundError(f"Put your image at {image_path} or change the filename.") MODEL_ID = "ishaqinu/Qwen3.5-9B-FP8-Dynamic"REVISION = "f6769bb5cfba557aff830738de94bd7c857730fe" processor = AutoProcessor.from_pretrained( MODEL_ID, revision=REVISION, trust_remote_code=False,)model = Qwen3_5ForConditionalGeneration.from_pretrained( MODEL_ID, revision=REVISION, trust_remote_code=False, use_safetensors=True, dtype=torch.float16, device_map="auto", quantization_config=CompressedTensorsConfig(use_optimized_inference=True),).eval() messages = [ { "role": "user", "content": [ {"type": "image", "url": str(image_path)}, {"type": "text", "text": "Describe this image and read any visible text."}, ], }]inputs = processor.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, enable_thinking=False, return_dict=True, return_tensors="pt",).to(model.device) with torch.inference_mode(): output = model.generate(**inputs, max_new_tokens=256, do_sample=False) answer_tokens = output[0, inputs["input_ids"].shape[1]:]print(processor.decode(answer_tokens, skip_special_tokens=True))The examples use enable_thinking=False for a direct answer. Set it to True and increase max_new_tokens if you want thinking mode. Start with one modest-size image and a short output.
For text-only use, download examples/text.py and run python text.py. For a local OpenAI-compatible API and standalone clients, see SERVING.md.
Documentation validation: the Python examples passed syntax checks, and the local image-path handling was checked. The published examples and candidate dependency environments have not been reproduced end to end during documentation validation. The publisher's informal comparison is reported separately above.
The stored tensors contain approximately 4.109 billion FP16 elements and 5.302 billion FP8 elements. Their combined payload is about 28% smaller than storing the same number of elements at two bytes each. This is a storage calculation; peak VRAM, speed, and minimum GPU capacity have not been measured and published.
Inference also needs memory for cache, activations, image processing, and runtime allocations. A 12 GB GPU should not be assumed to fit the complete checkpoint in GPU memory. Reduce context, image size, output length, and concurrency if you run out of memory.
The Transformers example requests FP8 kernels with use_optimized_inference=True. NVIDIA Ada Lovelace, Hopper, and Blackwell hardware supports native FP8 operations with a suitable software stack. On unsupported devices, Transformers can use a dequantized path requiring more memory. vLLM documents a weight-only FP8 fallback for Turing/Ampere; checkpoint-specific compatibility still requires testing. See Transformers loading modes and vLLM FP8 support.
The saved recipe.yaml applies FP8_DYNAMIC to Linear modules with these exclusions:
| Ignore pattern | Excluded components |
|---|---|
re:.*visual.* | Vision encoder and visual merger/projections |
re:.*linear_attn.* | Language-model linear-attention blocks |
lm_head | Language-model output head |
Other untargeted components, including embeddings and normalization parameters, are not converted by a recipe targeting linear modules. config.json contains the resolved settings and exclusion list. The checkpoint does not assign a KV-cache quantization scheme.
The original source revision, llmcompressor version, hardware, and complete quantization environment have not been documented. RECREATION.md provides a separate illustrative workflow, not the original run or a guarantee of identical weights.
Use a Qwen3.5-compatible Transformers release; the candidate lock uses 5.17.0. If FP8 kernels fail, check device capability, CUDA compatibility, and startup logs. For empty final answers, disable thinking or allow more output tokens. Wait for vLLM to finish loading before calling its API.
The examples pin model and processor/tokenizer revisions, require Safetensors for Transformers loading, and do not enable custom remote Python code. Keep access tokens out of prompts and shared logs. Treat instructions inside images and documents as untrusted when connecting the model to tools.
Report issues in the Community tab with the checkpoint revision, GPU, package versions, error message, and a minimal example. Remove tokens and private input data before sharing.
Qwen developed the original model; ishaqinu published this mixed-precision derivative using llmcompressor. This repository preserves the upstream Apache-2.0 LICENSE, including attribution to Alibaba Cloud.
The examples pin checkpoint revision f6769bb5cfba557aff830738de94bd7c857730fe. Documentation-only updates do not change the model loaded by that pin.
python3.12 -m venv .venvsource .venv/bin/activatepython -m pip install --require-hashes --only-binary=:all: --index-url https://pypi.org/simple -r requirements-linux-py312.lockfrom pathlib import Path import torchfrom transformers import ( AutoProcessor, CompressedTensorsConfig, Qwen3_5ForConditionalGeneration,) image_path = Path("example.jpg").resolve()if not image_path.is_file(): raise FileNotFoundError(f"Put your image at {image_path} or change the filename.") MODEL_ID = "ishaqinu/Qwen3.5-9B-FP8-Dynamic"REVISION = "f6769bb5cfba557aff830738de94bd7c857730fe" processor = AutoProcessor.from_pretrained( MODEL_ID, revision=REVISION, trust_remote_code=False,)model = Qwen3_5ForConditionalGeneration.from_pretrained( MODEL_ID, revision=REVISION, trust_remote_code=False, use_safetensors=True, dtype=torch.float16, device_map="auto", quantization_config=CompressedTensorsConfig(use_optimized_inference=True),).eval() messages = [ { "role": "user", "content": [ {"type": "image", "url": str(image_path)}, {"type": "text", "text": "Describe this image and read any visible text."}, ], }]inputs = processor.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, enable_thinking=False, return_dict=True, return_tensors="pt",).to(model.device) with torch.inference_mode(): output = model.generate(**inputs, max_new_tokens=256, do_sample=False) answer_tokens = output[0, inputs["input_ids"].shape[1]:]print(processor.decode(answer_tokens, skip_special_tokens=True))