Lets-Fafo

Qwen3.8-27B-FAFO-W4A16-G32-B70

README

License: apache-2.0

What was quantized

Table

Architecture	`Qwen3_5ForConditionalGeneration` (`model_type: qwen3_5`), the same as the upstream Qwen3.8-27B release
Format	compressed-tensors `pack-quantized` (W4A16)
Weights	INT4, symmetric, group size 32, BF16 scales
Activations	BF16 (not quantized)
Method	Weight-only round-to-nearest, computed directly from the original BF16 weights. No calibration data, fine-tuning or distillation.
Quantized (400 modules)	Text-model projections only:• MLP `gate_proj` / `up_proj` / `down_proj` in all 64 layers• full-attention `q_proj` / `k_proj` / `v_proj` / `o_proj` in the 16 attention layers• linear-attention `in_proj_qkv` / `in_proj_z` / `out_proj` in the 48 linear-attention layers
Unchanged (BF16, bit-identical to the original)	Embeddings, `lm_head`, all norms, linear-attention `in_proj_a` / `in_proj_b` and state parameters, the vision encoder, and the MTP head
Tokenizer, chat template, generation and processor configs	Unmodified copies of the original files

Tested environment

Everything on this card was measured in this configuration only:

Table

GPU	1 × Intel Arc Pro B70, 32 GB (Battlemage G31)
OS / driver	Linux 6.17, `xe` kernel driver
Container	`intel/llm-scaler-vllm:0.26.0-b2` (`sha256:52218ad85513ab6686d4c090c83c2bd8c5b02423c63aa4dabd41837fe641fe3b`)
Software in that container	vLLM 0.26.1.dev0 (XPU), PyTorch 2.12.0+xpu, Transformers 5.8.0, compressed-tensors 0.17.0, oneAPI 2025.3
Linear kernel selected by vLLM	`XPUwNa16LinearKernel`

We have not tested this model on other GPUs, other runtimes, or by loading it directly with Transformers.

How to run

This is the command we verified with the files in this repository: the server loaded, answered a chat request with thinking on and with thinking off, and returned a correctly parsed tool call.

bash

docker run -d --name qwen38-fafo-w4a16-b70 --ipc private --shm-size 8g \  --device /dev/dri/renderD130 --device /dev/dri/card2 \  -p 127.0.0.1:8000:8000 \  -v /path/to/Qwen3.8-27B-FAFO-W4A16-G32-B70:/model:ro \  -e HF_HUB_OFFLINE=1 \  --entrypoint bash \  intel/llm-scaler-vllm@sha256:52218ad85513ab6686d4c090c83c2bd8c5b02423c63aa4dabd41837fe641fe3b \  -c 'source /opt/intel/oneapi/setvars.sh --force && source /opt/intel/oneapi/ccl/2021.15/env/vars.sh --force && exec vllm serve "$@"' -- \  --model /model --served-model-name Qwen3.8-27B-FAFO-W4A16-G32-B70 --host 0.0.0.0 --port 8000 \  --tensor-parallel-size 1 --dtype bfloat16 --gpu-memory-utilization 0.94 --max-model-len 32768 \  --max-num-seqs 32 --max-num-batched-tokens 1024 --enforce-eager \  --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \  --no-enable-prefix-caching

Replace renderD130 / card2 with the device nodes of your B70 (ls -l /dev/dri/by-path/). For prompts longer than 32k tokens we used --max-model-len 114688, with everything else unchanged.

Example request, with thinking turned off:

bash

curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{  "model": "Qwen3.8-27B-FAFO-W4A16-G32-B70",  "messages": [{"role": "user", "content": "Reply with exactly this word and nothing else: READY"}],  "temperature": 0, "max_tokens": 16,  "chat_template_kwargs": {"enable_thinking": false}}'

Performance

All numbers are from one Intel Arc Pro B70 using the server command above (eager mode, MTP off). Requests used random-token prompts and a fixed output length, generated with vllm bench serve.

Output tok/s counts generated tokens only.
Total tok/s counts input plus output tokens, so it depends on prompt length. Only compare total tok/s between rows with the same input/output lengths.
TTFT (time to first token) includes server queueing.
Decode tok/s per request is 1 ÷ time per output token.
Memory: the model weights take 18.66 GiB on the card. With --gpu-memory-utilization 0.94, measured device memory in use was about 28.4 GiB throughout. vLLM reported 7.77 GiB of available KV cache at startup (116,092 tokens at --max-model-len 32768). That figure is resolved by vLLM after its other runtime allocations, not simply 94% minus the weights.

Throughput and latency (prefix caching off)

512 input / 256 output tokens per request. Rows at 1 and 4 clients were measured on Oct 11, 2026 and the others on Oct 7, 2026, on the same container and identical model files (verified by SHA256).

Table with columns: Concurrent requests, Requests (failed), Output tok/s, Total tok/s, TTFT p50 / p95, Decode tok/s per request (p50), End-to-end p50 / p95
Concurrent requests	Requests (failed)	Output tok/s	Total tok/s	TTFT p50 / p95	Decode tok/s per request (p50)	End-to-end p50 / p95
1	16 (0)	26.4	79.3	0.31 / 0.31 s	27.2	9.69 / 9.71 s
4	32 (0)	88.8	266.4	0.96 / 1.24 s	24.2

The server admits at most 32 requests at once (--max-num-seqs 32), so at 64 clients the extra requests wait in the queue.

Sustained throughput: 512 requests at 64 concurrent clients, measured over a fixed 60–240 s window. Each of two B70 cards was measured on its own:

Table with columns: Card, Sustained output tok/s
Card	Sustained output tok/s
B70 #1	296.54
B70 #2	293.69

Long context (single request, prefix caching off)

256 output tokens per request, server started with --max-model-len 114688 (KV cache capacity: 123,663 tokens). Three requests per size; we report the median with the min–max range.

Table with columns: Input tokens, Requests (failed), TTFT median (min–max), Effective input processing rate*, Decode tok/s, End-to-end median
Input tokens	Requests (failed)	TTFT median (min–max)	Effective input processing rate*	Decode tok/s	End-to-end median
8,192	3 (0)	4.85 s (4.84–4.85)	~1,690 tok/s	26.4	14.50 s
32,768	3 (0)	21.55 s (21.45–21.55)	~1,520 tok/s	24.5	31.95 s
65,536

*Input tokens ÷ TTFT. This includes scheduling overhead, so it is not a pure prefill benchmark.

Against a more conservative build of the same model

The conservative recipe for this model on Intel hardware is mixed INT4/FP8: INT4 on the MLP projections and FP8 on the attention and recurrent projections (232 of the 400 projections in FP8). That is the recipe of the publicly downloadable quantization we ran before this work (an NVFP4 release converted to INT4/FP8 for the B70); on the same B70, container and settings it measured 134.9 / 205.4 / 217.1 / 217.1 output tok/s at 8 / 16 / 32 / 64 clients. We rebuilt that exact recipe from the official BF16 weights as a control and it measured the same (134.9 / 205.4 / 217.1 / 217.1), so the recipe, not the source weights, set that ceiling. The all-INT4 checkpoint published here, relative to that control, measured on the same B70 with the same harness and server settings, caching off:

Table with columns: Mixed INT4/FP8 build, This checkpoint, Change
	Mixed INT4/FP8 build	This checkpoint	Change
Sustained output tok/s, 64 clients (GPU 1 / GPU 2)	217.2 / 215.4	296.5 / 293.7	+36.5% / +36.3%
Device memory traffic, GB per generated token	1.539 / 1.542	1.036 / 1.036	−32.7% / −32.8%
Card energy per generated token	—	—	−23.3% / −20.8%
Weights loaded on the card	22.41 GiB	18.66 GiB

One measurement per arm per card; the two cards were run in opposite order as a crossover, not as repeated trials. Energy per token is mean card power divided by output throughput, not whole-system energy. The conservative build's perplexity change on the diagnostic set was +0.25% versus this checkpoint's +3.31%, so the throughput gain comes with a larger measured quality shift; see Preliminary quality checks. That tradeoff is why this recipe is the one published.

Prefix caching (separate measurement)

Every request had a 1,024-token prefix shared by all requests, followed by 512 unique tokens, with 256 output tokens (1,536 input tokens per request). The same workload was run with prefix caching off and on; nothing else changed.

Table with columns: Concurrent requests, Output tok/s off → on, Total tok/s off → on, TTFT p50 off → on, TTFT p95 off → on, End-to-end p50 off → on
Concurrent requests	Output tok/s off → on	Total tok/s off → on	TTFT p50 off → on	TTFT p95 off → on	End-to-end p50 off → on
1	24.6 → 26.0	171.9 → 181.9	0.90 → 0.32 s	0.90 → 0.32 s	10.42 → 9.85 s
4	72.8 → 72.4	509.3 → 507.0	2.03 → 1.13 s	3.12 → 11.16 s	14.09 → 11.67 s
8

When the prefix is not yet cached (a single cold request on an already-running server), TTFT is 0.91 s with caching on and 0.91 s with it off. Once the prefix is cached, single-request TTFT is 0.32 s.

With a shared prompt prefix, prefix caching can materially improve throughput and median latency. In this test, output throughput improved by up to 75% (32 concurrent requests), and median TTFT was lower at every concurrency level. Tail latency was less consistent: p95 TTFT was higher with caching at 4, 8 and 16 concurrent requests, and lower at 32. We have not determined the cause. vLLM marks prefix caching for this hybrid architecture as experimental, so the caching-off numbers above are our primary performance results.

Preliminary quality checks

These checks are deliberately small. We compared this model with the original BF16 model on:

perplexity over 8 fixed documents (697 scored tokens)
full-vocabulary KL divergence from the original
10 short generation probes covering exact instructions, JSON output, tool arguments, arithmetic, short prose and retrieval, with 9 strictly scored response contracts

Probes used greedy decoding with thinking turned off.

Table with columns: Original BF16, This model
	Original BF16	This model
Perplexity	12.903	13.332 (+3.33%)
Mean KL divergence from BF16	—	0.0388
Strict response contracts	8 / 9	8 / 9

The original BF16 model missed one contract: it answered 7 to an arithmetic probe whose answer is 107.
This model missed one contract: it returned the correct tool-call JSON, but wrapped it in a Markdown code fence where raw JSON was required. That is counted as a failure.
Separate follow-up: only the two JSON-only prompts were made more explicit about returning raw JSON; all other probes and expected answers were unchanged. Under those prompts this model passed 9 / 9, with valid unfenced JSON. This is a separate prompt condition; the original 8 / 9 result above stands. The BF16 original was not rerun under the stronger prompts.
Runtime difference: the BF16 reference ran on CPU with Transformers, and this model ran on the B70 with vLLM. Some of the measured differences may therefore come from the runtime rather than from quantization.

This set is small. It does not establish general capability, long-context quality, vision quality or thinking-mode quality.

License

Apache License 2.0, the same license as the original model. See LICENSE and NOTICE. The original model is Copyright 2026 Alibaba Cloud. This repository is a quantized derivative published by Neuro Forge, LLC (meetfafo.com · letsfafo.com), and is not affiliated with or endorsed by Alibaba Cloud, the Qwen team, or Intel.

FAFO™ is a trademark of Convergence Information Foundry Research Systems LLC.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Lets-Fafo

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

What was quantized

Table

Architecture	`Qwen3_5ForConditionalGeneration` (`model_type: qwen3_5`), the same as the upstream Qwen3.8-27B release
Format	compressed-tensors `pack-quantized` (W4A16)
Weights	INT4, symmetric, group size 32, BF16 scales
Activations	BF16 (not quantized)
Method	Weight-only round-to-nearest, computed directly from the original BF16 weights. No calibration data, fine-tuning or distillation.
Quantized (400 modules)	Text-model projections only:• MLP `gate_proj` / `up_proj` / `down_proj` in all 64 layers• full-attention `q_proj` / `k_proj` / `v_proj` / `o_proj` in the 16 attention layers• linear-attention `in_proj_qkv` / `in_proj_z` / `out_proj` in the 48 linear-attention layers
Unchanged (BF16, bit-identical to the original)	Embeddings, `lm_head`, all norms, linear-attention `in_proj_a` / `in_proj_b` and state parameters, the vision encoder, and the MTP head
Tokenizer, chat template, generation and processor configs	Unmodified copies of the original files

Tested environment

Everything on this card was measured in this configuration only:

Table

GPU	1 × Intel Arc Pro B70, 32 GB (Battlemage G31)
OS / driver	Linux 6.17, `xe` kernel driver
Container	`intel/llm-scaler-vllm:0.26.0-b2` (`sha256:52218ad85513ab6686d4c090c83c2bd8c5b02423c63aa4dabd41837fe641fe3b`)
Software in that container	vLLM 0.26.1.dev0 (XPU), PyTorch 2.12.0+xpu, Transformers 5.8.0, compressed-tensors 0.17.0, oneAPI 2025.3
Linear kernel selected by vLLM	`XPUwNa16LinearKernel`

We have not tested this model on other GPUs, other runtimes, or by loading it directly with Transformers.

How to run

This is the command we verified with the files in this repository: the server loaded, answered a chat request with thinking on and with thinking off, and returned a correctly parsed tool call.

bash

docker run -d --name qwen38-fafo-w4a16-b70 --ipc private --shm-size 8g \  --device /dev/dri/renderD130 --device /dev/dri/card2 \  -p 127.0.0.1:8000:8000 \  -v /path/to/Qwen3.8-27B-FAFO-W4A16-G32-B70:/model:ro \  -e HF_HUB_OFFLINE=1 \  --entrypoint bash \  intel/llm-scaler-vllm@sha256:52218ad85513ab6686d4c090c83c2bd8c5b02423c63aa4dabd41837fe641fe3b \  -c 'source /opt/intel/oneapi/setvars.sh --force && source /opt/intel/oneapi/ccl/2021.15/env/vars.sh --force && exec vllm serve "$@"' -- \  --model /model --served-model-name Qwen3.8-27B-FAFO-W4A16-G32-B70 --host 0.0.0.0 --port 8000 \  --tensor-parallel-size 1 --dtype bfloat16 --gpu-memory-utilization 0.94 --max-model-len 32768 \  --max-num-seqs 32 --max-num-batched-tokens 1024 --enforce-eager \  --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \  --no-enable-prefix-caching

docker run -d --name qwen38-fafo-w4a16-b70 --ipc private --shm-size 8g \  --device /dev/dri/renderD130 --device /dev/dri/card2 \  -p 127.0.0.1:8000:8000 \  -v /path/to/Qwen3.8-27B-FAFO-W4A16-G32-B70:/model:ro \  -e HF_HUB_OFFLINE=1 \  --entrypoint bash \  intel/llm-scaler-vllm@sha256:52218ad85513ab6686d4c090c83c2bd8c5b02423c63aa4dabd41837fe641fe3b \  -c 'source /opt/intel/oneapi/setvars.sh --force && source /opt/intel/oneapi/ccl/2021.15/env/vars.sh --force && exec vllm serve "$@"' -- \  --model /model --served-model-name Qwen3.8-27B-FAFO-W4A16-G32-B70 --host 0.0.0.0 --port 8000 \  --tensor-parallel-size 1 --dtype bfloat16 --gpu-memory-utilization 0.94 --max-model-len 32768 \  --max-num-seqs 32 --max-num-batched-tokens 1024 --enforce-eager \  --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \  --no-enable-prefix-caching

Replace renderD130 / card2 with the device nodes of your B70 (ls -l /dev/dri/by-path/). For prompts longer than 32k tokens we used --max-model-len 114688, with everything else unchanged.

Example request, with thinking turned off:

bash

curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{  "model": "Qwen3.8-27B-FAFO-W4A16-G32-B70",  "messages": [{"role": "user", "content": "Reply with exactly this word and nothing else: READY"}],  "temperature": 0, "max_tokens": 16,  "chat_template_kwargs": {"enable_thinking": false}}'

curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{  "model": "Qwen3.8-27B-FAFO-W4A16-G32-B70",  "messages": [{"role": "user", "content": "Reply with exactly this word and nothing else: READY"}],  "temperature": 0, "max_tokens": 16,  "chat_template_kwargs": {"enable_thinking": false}}'

Performance

All numbers are from one Intel Arc Pro B70 using the server command above (eager mode, MTP off). Requests used random-token prompts and a fixed output length, generated with vllm bench serve.

Output tok/s counts generated tokens only.
Total tok/s counts input plus output tokens, so it depends on prompt length. Only compare total tok/s between rows with the same input/output lengths.
TTFT (time to first token) includes server queueing.
Decode tok/s per request is 1 ÷ time per output token.
Memory: the model weights take 18.66 GiB on the card. With --gpu-memory-utilization 0.94, measured device memory in use was about 28.4 GiB throughout. vLLM reported 7.77 GiB of available KV cache at startup (116,092 tokens at --max-model-len 32768). That figure is resolved by vLLM after its other runtime allocations, not simply 94% minus the weights.

Throughput and latency (prefix caching off)

512 input / 256 output tokens per request. Rows at 1 and 4 clients were measured on Oct 11, 2026 and the others on Oct 7, 2026, on the same container and identical model files (verified by SHA256).

Table with columns: Concurrent requests, Requests (failed), Output tok/s, Total tok/s, TTFT p50 / p95, Decode tok/s per request (p50), End-to-end p50 / p95
Concurrent requests	Requests (failed)	Output tok/s	Total tok/s	TTFT p50 / p95	Decode tok/s per request (p50)	End-to-end p50 / p95
1	16 (0)	26.4	79.3	0.31 / 0.31 s	27.2	9.69 / 9.71 s
4	32 (0)	88.8	266.4	0.96 / 1.24 s	24.2

The server admits at most 32 requests at once (--max-num-seqs 32), so at 64 clients the extra requests wait in the queue.

Sustained throughput: 512 requests at 64 concurrent clients, measured over a fixed 60–240 s window. Each of two B70 cards was measured on its own:

Table with columns: Card, Sustained output tok/s
Card	Sustained output tok/s
B70 #1	296.54
B70 #2	293.69

Long context (single request, prefix caching off)

256 output tokens per request, server started with --max-model-len 114688 (KV cache capacity: 123,663 tokens). Three requests per size; we report the median with the min–max range.

Table with columns: Input tokens, Requests (failed), TTFT median (min–max), Effective input processing rate*, Decode tok/s, End-to-end median
Input tokens	Requests (failed)	TTFT median (min–max)	Effective input processing rate*	Decode tok/s	End-to-end median
8,192	3 (0)	4.85 s (4.84–4.85)	~1,690 tok/s	26.4	14.50 s
32,768	3 (0)	21.55 s (21.45–21.55)	~1,520 tok/s	24.5	31.95 s
65,536

*Input tokens ÷ TTFT. This includes scheduling overhead, so it is not a pure prefill benchmark.

Against a more conservative build of the same model

Table with columns: Mixed INT4/FP8 build, This checkpoint, Change
	Mixed INT4/FP8 build	This checkpoint	Change
Sustained output tok/s, 64 clients (GPU 1 / GPU 2)	217.2 / 215.4	296.5 / 293.7	+36.5% / +36.3%
Device memory traffic, GB per generated token	1.539 / 1.542	1.036 / 1.036	−32.7% / −32.8%
Card energy per generated token	—	—	−23.3% / −20.8%
Weights loaded on the card	22.41 GiB	18.66 GiB

Prefix caching (separate measurement)

Table with columns: Concurrent requests, Output tok/s off → on, Total tok/s off → on, TTFT p50 off → on, TTFT p95 off → on, End-to-end p50 off → on
Concurrent requests	Output tok/s off → on	Total tok/s off → on	TTFT p50 off → on	TTFT p95 off → on	End-to-end p50 off → on
1	24.6 → 26.0	171.9 → 181.9	0.90 → 0.32 s	0.90 → 0.32 s	10.42 → 9.85 s
4	72.8 → 72.4	509.3 → 507.0	2.03 → 1.13 s	3.12 → 11.16 s	14.09 → 11.67 s
8

Preliminary quality checks

These checks are deliberately small. We compared this model with the original BF16 model on:

perplexity over 8 fixed documents (697 scored tokens)
full-vocabulary KL divergence from the original
10 short generation probes covering exact instructions, JSON output, tool arguments, arithmetic, short prose and retrieval, with 9 strictly scored response contracts

Probes used greedy decoding with thinking turned off.

Table with columns: Original BF16, This model
	Original BF16	This model
Perplexity	12.903	13.332 (+3.33%)
Mean KL divergence from BF16	—	0.0388
Strict response contracts	8 / 9	8 / 9

The original BF16 model missed one contract: it answered 7 to an arithmetic probe whose answer is 107.
This model missed one contract: it returned the correct tool-call JSON, but wrapped it in a Markdown code fence where raw JSON was required. That is counted as a failure.
Separate follow-up: only the two JSON-only prompts were made more explicit about returning raw JSON; all other probes and expected answers were unchanged. Under those prompts this model passed 9 / 9, with valid unfenced JSON. This is a separate prompt condition; the original 8 / 9 result above stands. The BF16 original was not rerun under the stronger prompts.
Runtime difference: the BF16 reference ran on CPU with Transformers, and this model ran on the B70 with vLLM. Some of the measured differences may therefore come from the runtime rather than from quantization.

This set is small. It does not establish general capability, long-context quality, vision quality or thinking-mode quality.

License

FAFO™ is a trademark of Convergence Information Foundry Research Systems LLC.