GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
tonera
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
tonera
Model Tree
black-forest-labs/flux-3-image-klein-pup
Input Modalities
Output Modalities
Supported Functionality
Measured on an RTX 5090, comparing this INT4 upsampler with the original BF16 model.
FLUX 3 Image Klein needs a complete, specific description to draw well. What people actually type is often one sentence, such as "a person walking a dog in the afternoon sun." flux-3-image-klein-pup turns that sentence into the description the image model needs. It is a full fine-tune of Qwen3.5-9B. The BF16 weights are 18.8 GB and generation uses nearly 18 GiB of VRAM, which is tight when the image model shares the same GPU.
This repository compresses it to INT4. Peak VRAM during generation drops to about 7 GiB, and the time per token drops from about 15 ms to 4.3 ms. A caption of about 300 tokens returns in about 1.4 seconds from the resident server. The goal is to make upsampling cheap, and leave the VRAM and time for image generation.
svdq-int4-flux-3-image-klein-pup-nunchaku.safetensorsapi.py, compatible with the OpenAI Chat Completions APIsystem_prompts.json is kept as-isThe upstream model uses one set of weights for text-to-image, reference-image editing, and boxed layout. The system prompt selects the mode. It writes down the medium and style, the scene, the light, the camera, and the color, material, and position of each subject. Text that should be rendered is left inside quotes, unchanged. For edits, it looks at the reference images first, then states what changes and what stays.
lm_head is INT4 as well.lm_head is GPTQ'd on the outputs of the quantized language model.in_proj_a / in_proj_b in the linear-attention layers (they set the decay gate) stay in floating point. The vision path is unchanged.RTX 5090, dtype bfloat16, greedy decoding, enable_thinking=False. Speed is timed per output token. VRAM is the peak during generation and does not change with output length. The benchmark script is scripts/profile_qwen3_5_pup.py in the Nunchaku repository.
| Scenario | BF16 | INT4 | Speedup | BF16 VRAM | INT4 VRAM | VRAM saved |
|---|---|---|---|---|---|---|
| Text-to-image upsampling | 15.43 ms/token | 4.29 ms/token | 3.60× | 17.75 GiB | 7.02 GiB | 10.73 GiB (60.5%) |
| Single-image edit upsampling | 15.35 ms/token | 4.35 ms/token | 3.53× | 17.93 GiB | 7.21 GiB | 10.72 GiB (59.8%) |
End-to-end measurements through the resident api.py server. Three runs were nearly identical:
| Scenario | End to end | Prefill + first token | Output | Decode | Tokens per second |
|---|---|---|---|---|---|
| Text-to-image upsampling | 1.424 s | 0.086 s | 297 tokens | 4.52 ms/token | 221 |
| Single-image edit upsampling | 1.174 s | 0.271 s | 200 tokens | 4.54 ms/token | 220 |
The edit reference image is 1024×1024, and the input is 1624 tokens. The client reads the image as base64 and sends it to the server over a local network. The 0.271 s includes uploading about 1.8 MB of image data, but most of that time is vision encoding and prefill.
Word counts of captions written by BF16 and INT4 on the same 10 requests. Both mostly land between 200 and 300 words. INT4 is slightly shorter on a few of them. The horizontal axis labels only requests 2, 4, 6, 8, and 10.
You need a CUDA GPU, plus:
NunchakuQwen3_5ForConditionalGeneration. Prebuilt wheels are at tonera/vitoom-nunchaku. Pick the wheel that matches your Python, CUDA, and PyTorch versions.Qwen3_5ForConditionalGeneration).bash
pip install -U transformers pillow huggingface_hub # Pick the wheel from tonera/vitoom-nunchaku that matches this machine's Python, CUDA, and PyTorchhf download tonera/vitoom-nunchaku <wheel filename> --local-dir ./wheelspip install ./wheels/<wheel filename>On startup, api.py loads the quantized upsampler onto the GPU and keeps it resident. Later requests generate directly, with no reload. It exposes an OpenAI Chat Completions-compatible API, so existing OpenAI SDKs and clients can call it as-is.
The server inserts the system prompt. Callers submit only the request.
bash
hf download tonera/flux-3-image-klein-pup-nunchaku --local-dir flux-3-image-klein-pup-nunchakucd flux-3-image-klein-pup-nunchakupython api.py # listens on 0.0.0.0:8000 by defaultpython api.py --port 9000 # another portThe server is ready when it prints [INFO] listening on http://0.0.0.0:8000/v1/chat/completions.
| Argument | Environment variable | Default | Description |
|---|---|---|---|
--host | 0.0.0.0 | Bind address | |
--port | 8000 | Port | |
--pup | PUP_PATH | . | Directory that holds the processor and system_prompts.json |
--quant | QUANT_PATH | ./svdq-int4-flux-3-image-klein-pup-nunchaku.safetensors | Path to the quantized weights |
--device | cuda | CUDA only |
Run it from the repository directory and these arguments can stay at their defaults.
bash
curl http://127.0.0.1:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"flux-3-image-klein-pup","messages":[{"role":"user","content":"A man walking with his dog, afternoon sun"}]}'image_url accepts three forms: an http(s) URL (downloaded by the server), a data: URI (base64), or a file:// path on the server machine.
python
from openai import OpenAI client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")resp = client.chat.completions.create( model="flux-3-image-klein-pup", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://example.com/dog.jpg"}}, {"type": "text", "text": "put a red bandana around the dog's neck"}, ], }],)print(resp.choices[0].message.content)If the image is already on the client, read it as base64 and submit that. The server then does not need outbound network access. The performance numbers above were measured this way:
python
import base64 with open("dog.jpg", "rb") as f: url = "data:image/jpeg;base64," + base64.b64encode(f.read()).decode() resp = client.chat.completions.create( model="flux-3-image-klein-pup", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": url}}, {"type": "text", "text": "the same scene at night"}, ], }],)Several reference images are listed in order. The model treats them as the first image, the second image, and so on.
json
{ "id": "chatcmpl-...", "object": "chat.completion", "model": "flux-3-image-klein-pup", "choices": [{ "index": 0, "message": {"role": "assistant", "content": "Full-color outdoor photograph, eye-level medium-wide shot ..."}, "finish_reason": "stop" }], "usage": {"prompt_tokens": ..., "completion_tokens": 297, "total_tokens": ...}}choices[0].message.content is the upsampled description. Pass it to FLUX 3 Image Klein as prompt. A finish_reason of length means the output was cut off by max_tokens.
GET /v1/models lists the model. POST /v1/chat/completions runs upsampling.user message is read. Any system message in the request is ignored.max_tokens and max_completion_tokens default to 3584, which is also the maximum.stream is not supported, and n must be 1.error object. A bad request returns 4xx. A generation failure returns 500.file:// can read files on the server machine. Deploy it on a trusted network, or put it behind an authenticating reverse proxy.If you do not want a separate server, use the Transformers-style API directly. The message format matches the upstream examples:
python
import json import torchfrom huggingface_hub import hf_hub_downloadfrom transformers import AutoProcessorfrom nunchaku import NunchakuQwen3_5ForConditionalGeneration REPO = "tonera/flux-3-image-klein-pup-nunchaku" processor = AutoProcessor.from_pretrained(REPO)with open(hf_hub_download(REPO, "system_prompts.json"), encoding="utf-8") as f: system_prompts = json.load(f)model = NunchakuQwen3_5ForConditionalGeneration.from_pretrained( f"{REPO}/svdq-int4-flux-3-image-klein-pup-nunchaku.safetensors", dtype=torch.bfloat16,).eval().to("cuda") messages = [ {"role": "system", "content": system_prompts["t2i"]}, {"role": "user", "content": "A man walking with his dog, afternoon sun"},]inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", enable_thinking=False,).to(model.device)with torch.no_grad(): output = model.generate(**inputs, max_new_tokens=3584, do_sample=False)caption = processor.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()print(caption)For edit upsampling, use system_prompts["i2i"] and place the reference images before the text:
python
from PIL import Image image = Image.open("dog.jpg").convert("RGB")text = ( "Reference images follow in order (the first image, the second image, ...).\n" "User request: put a red bandana around the dog's neck\nOutput the caption.")messages = [ {"role": "system", "content": system_prompts["i2i"]}, {"role": "user", "content": [{"type": "image", "image": image}, {"type": "text", "text": text}]},]api.py currently serves only text-to-image and edit modes. The upstream boxed-layout mode is not wired into the server. To use it, follow the upstream examples and call the model directly with the matching system prompt in system_prompts.json.Thanks to Black Forest Labs for releasing FLUX 3 Image Klein and this upsampler, to the Qwen team for Qwen3.5, and to the Nunchaku and DeepCompressor communities.
These quantized weights are derived from black-forest-labs/flux-3-image-klein-pup and are released under the Apache License 2.0, the same license as the upstream model and its Qwen3.5-9B base.
pip install -U transformers pillow huggingface_hub # Pick the wheel from tonera/vitoom-nunchaku that matches this machine's Python, CUDA, and PyTorchhf download tonera/vitoom-nunchaku <wheel filename> --local-dir ./wheelspip install ./wheels/<wheel filename>hf download tonera/flux-3-image-klein-pup-nunchaku --local-dir flux-3-image-klein-pup-nunchakucd flux-3-image-klein-pup-nunchakupython api.py # listens on 0.0.0.0:8000 by defaultpython api.py --port 9000 # another portcurl http://127.0.0.1:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"flux-3-image-klein-pup","messages":[{"role":"user","content":"A man walking with his dog, afternoon sun"}]}'from openai import OpenAI client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")resp = client.chat.completions.create( model="flux-3-image-klein-pup", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://example.com/dog.jpg"}}, {"type": "text", "text": "put a red bandana around the dog's neck"}, ], }],)print(resp.choices[0].message.content)import base64 with open("dog.jpg", "rb") as f: url = "data:image/jpeg;base64," + base64.b64encode(f.read()).decode() resp = client.chat.completions.create( model="flux-3-image-klein-pup", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": url}}, {"type": "text", "text": "the same scene at night"}, ], }],){ "id": "chatcmpl-...", "object": "chat.completion", "model": "flux-3-image-klein-pup", "choices": [{ "index": 0, "message": {"role": "assistant", "content": "Full-color outdoor photograph, eye-level medium-wide shot ..."}, "finish_reason": "stop" }], "usage": {"prompt_tokens": ..., "completion_tokens": 297, "total_tokens": ...}}import json import torchfrom huggingface_hub import hf_hub_downloadfrom transformers import AutoProcessorfrom nunchaku import NunchakuQwen3_5ForConditionalGeneration REPO = "tonera/flux-3-image-klein-pup-nunchaku" processor = AutoProcessor.from_pretrained(REPO)with open(hf_hub_download(REPO, "system_prompts.json"), encoding="utf-8") as f: system_prompts = json.load(f)model = NunchakuQwen3_5ForConditionalGeneration.from_pretrained( f"{REPO}/svdq-int4-flux-3-image-klein-pup-nunchaku.safetensors", dtype=torch.bfloat16,).eval().to("cuda") messages = [ {"role": "system", "content": system_prompts["t2i"]}, {"role": "user", "content": "A man walking with his dog, afternoon sun"},]inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", enable_thinking=False,).to(model.device)with torch.no_grad(): output = model.generate(**inputs, max_new_tokens=3584, do_sample=False)caption = processor.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()print(caption)from PIL import Image image = Image.open("dog.jpg").convert("RGB")text = ( "Reference images follow in order (the first image, the second image, ...).\n" "User request: put a red bandana around the dog's neck\nOutput the caption.")messages = [ {"role": "system", "content": system_prompts["i2i"]}, {"role": "user", "content": [{"type": "image", "image": image}, {"type": "text", "text": text}]},]