LeoHyperlink

Qwen3.8-27B-Pi-W4A8

README

License: apache-2.0

Recipe

Table with columns: Setting, Value
Setting	Value
Tool	Intel auto-round 0.15.1, `auto_round:llm_compressor` (CT pack-quantized) export
Weights	int4 symmetric, group 128
Activations	dynamic int8 (`act_bits 8`, per-token)
Calibration	nvidia/OpenCodeInstruct (128 samples x 2048 tokens)
torch.compile	explicitly disabled (Triton inductor failure on XPU)
Kept in BF16	`lm_head`, visual tower (`model.visual.*` ignored)
Quant time	~6.5 h on one Intel Arc Pro B70 (GPU1)

Speed (measured, not estimated)

Single-stream decode, 128 streamed tokens requested (76 generated, natural stop; temperature 0, prompt: "Explain why the sky is blue in one short paragraph"), OpenAI-compatible /v1/chat/completions streaming, tokens counted after time-to-first-token:

Table with columns: Checkpoint, Decode, TTFT, Backend
Checkpoint	Decode	TTFT	Backend
This repo (Pi W4A8-dynamic)	33.0 t/s (3/3 runs)	0.08 s	dynamic act-quant path
This repo + DFlash draft (`--spec-method dflash`, block 8)	~55 t/s (54.6–56.4, 3 runs)	0.09 s warm	speculative decode, acceptance 2.71/8
LeoHyperlink/Sage-1.5-W4A16 (same box and protocol, W4A16 reference)	104.3 t/s	0.06 s	`XPUwNa16LinearKernel` (fused)

Dynamic activation quantization costs ~3x vs the fused W4A16 path on this XPU stack. Coherent output on all runs, byte-identical between plain and speculative decode (temperature 0).

Speculative decoding (DFlash)

The 1.9B DFlash draft (block-diffusion drafter for Qwen3.8) was served alongside the quant on a single B70 (--spec-model incoai/Qwen3.8-27B-DFlash2 --spec-tokens 8, --per-request-spec-decode-metrics summary): ~55 t/s, a 1.7x speedup over plain decode, same prompt and protocol. Acceptance: mean length 2.71 per 8-token block, draft rate 21.4% (48/224 tokens over 28 steps) — modest because the draft was trained for base Qwen3.8 while the target is the pi-quant; a matched draft would do better. First-request TTFT was 1.33s cold (draft warmup), then 0.09s warm.

Fit note: target + draft weights (20.3 GB) + graphs (4.9 GB) leave almost no room for KV cache on one 32 GB card — the spec serve needed --max-model-len 1024 --max-num-seqs 16 --gpu-memory-utilization 0.95 (KV need 1.68 GB vs 1.64 GB available at len 1024 / seqs 64; halving seqs closed the gap). Batch-1 decode speed is unaffected by these reservations. Tensor-parallel (TP=2) was attempted to relieve this but is blocked on this box: oneCCL cannot initialize (mem_to_ipc_handle: invalid_fd, all IPC-exchange modes fail).

Benchmarks

Humanity's Last Exam (HLE): 12.14%

Full text-only set (2,158/2,158 samples), paper-faithful config via inspect_evals/hle (CAIS team's original.yaml-equivalent: hle/json_judge scorer, single attempt, no multi-modal):

Table with columns: Metric, Value
Metric	Value
Original accuracy	12.14% (262/2158; 3 samples unscored)
Stderr	0.70%
Run wall time	4 d 9 h
Tokens	57.6 M total — 54.3 M reasoning (≈25 k reasoning tok/sample, thinking xhigh)

Setup:

Solver = this quant (thinking default/xhigh, temperature 0, max_tokens 32768, streaming forced), served via a vLLM TP=2 OpenAI-compatible endpoint.
Grader = the same quant answering a JSON-judge prompt (max_grader_attempts 1).

Caveats:

Self-graded: no external judge (e.g. o3-mini) was used — the quant judges its own answers. Plan-level judge disagreement adds unknown bias, plausibly downwards.
Text-only, no tools: the base card's HLE 55.1 is with tools; this run is single-shot text-only. The two numbers must not be read against each other as a quantization recovery claim.
Raw per-sample transcripts and scores: benchmarks/hle-full.eval (inspect_ai .eval archive).

SciCode: 4.6% test (3/65), 33.3% val (5/15)

Upstream inspect_ai harness from the SciCode repo (github.com/scicode-bench/SciCode), no background prompts, temperature 0, streaming, test_data.h5 numeric grading:

Table with columns: Split, Main-problem solve, Substep correctness, Tokens
Split	Main-problem solve	Substep correctness	Tokens
test (65 problems)	4.6% (3/65)	30.2% (88/291)	9.0 M
validation (15 problems)	33.3% (5/15)	48.0% (24/50)	0.84 M

Caveats:

Run background-free (zero-shot, no doc-comment context in the prompt) — a strictly harder setting than the paper's default with-background mode; the val:test gap (33% → 4.6%) reflects that the test split is novel research problems rather than routine physics code.
Several substeps (≈12) ended in "fail to extract python code" — the model's output did not yield a parseable script under the harness's extraction rules, scoring 0 regardless of the underlying reasoning quality.
Raw logs: benchmarks/scicode-test.eval, benchmarks/scicode-val.eval.

Tested hardware / software

GPU: Intel Arc Pro B70 32 GB (Battlemage Xe2), single GPU
Serving stack: vLLM-XPU 0.32.0 fork, torch 2.x +XPU, oneCCL
Serve flags: --trust-remote-code --max-model-len 8192 --max-num-seqs 128 --gpu-memory-utilization 0.9 --language-model-only --kv-cache-dtype fp8 --enable-prefix-caching plus --default-chat-template-kwargs '{"enable_thinking": false}'. Note: this is a Mamba-hybrid model — only 159 Mamba cache blocks fit at 0.9 utilization, so max_num_seqs must stay at or below 159 (default 256 fails engine init). ZE_AFFINITY_MASK=0.0 crashes engine init on this build; GPU pinned via ONEAPI_DEVICE_SELECTOR=level_zero:N.

Usage

bash

pip install "vllm>=0.32"   # XPU fork (vllm-xpu) on Intel GPUsvllm serve LeoHyperlink/Qwen3.8-27B-Pi-W4A8 --trust-remote-code \  --max-model-len 8192 --max-num-seqs 128 --gpu-memory-utilization 0.9 --language-model-only

Files

model-0000[1-4]-of-00004.safetensors (~18.5 GB) + model.safetensors.index.json — quantized weights (MTP heads mtp.* included in the main checkpoint)
model_extra_tensors.safetensors (849 MB) — raw passthrough tensors, kept for faithfulness. vLLM 0.32.0 cannot load them alongside the CT main checkpoint; text-only serving does not need them.
quantization_config.json — compressed-tensors pack-quantized spec
config.json, tokenizer files, preprocessor/processor configs, chat_template.jinja, generation_config.json
benchmarks/hle-full.eval, benchmarks/scicode-test.eval, benchmarks/scicode-val.eval — raw inspect_ai eval logs (full per-sample transcripts + scores)
.eval_results/hle.yaml — machine-readable HLE score card (Hub eval-results schema; SciCode's dataset is README-only as it is not a registered benchmark)

Limitations

HLE 12.14% is text-only and self-graded (the quant judges itself, no o3-mini judge); the base card's HLE 55.1 is a with-tools agentic setting — see Benchmarks caveats, they are not comparable.
SciCode 4.6% is a no-background zero-shot run; with-background and tool-using settings score materially higher.
Not yet run: SWE-bench Pro and Terminal-Bench 2.1 (harness setup pending).
MTP speculative decoding is NOT enabled by this checkpoint (MTP weights ship in the main checkpoint but are unused; --spec-method qwen3_5_mtp exists in the serving build and is untested). Speculative decoding WAS validated via the DFlash draft path instead (see above).
Tensor-parallel serving is blocked on this box (oneCCL IPC init failure); all serves above are single-GPU.
The visual tower ships unquantized and was not exercised (--language-model-only serving).

Citation

No new training was done for this quant — if you use it, cite the base model (bytkim/Qwen3.8-27B-pi) and link back to this repo for the quantization recipe and serving numbers.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

LeoHyperlink

Model Tree

Base

bytkim/Qwen3.8-27B-pi

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Recipe

Table with columns: Setting, Value
Setting	Value
Tool	Intel auto-round 0.15.1, `auto_round:llm_compressor` (CT pack-quantized) export
Weights	int4 symmetric, group 128
Activations	dynamic int8 (`act_bits 8`, per-token)
Calibration	nvidia/OpenCodeInstruct (128 samples x 2048 tokens)
torch.compile	explicitly disabled (Triton inductor failure on XPU)
Kept in BF16	`lm_head`, visual tower (`model.visual.*` ignored)
Quant time	~6.5 h on one Intel Arc Pro B70 (GPU1)

Speed (measured, not estimated)

Table with columns: Checkpoint, Decode, TTFT, Backend
Checkpoint	Decode	TTFT	Backend
This repo (Pi W4A8-dynamic)	33.0 t/s (3/3 runs)	0.08 s	dynamic act-quant path
This repo + DFlash draft (`--spec-method dflash`, block 8)	~55 t/s (54.6–56.4, 3 runs)	0.09 s warm	speculative decode, acceptance 2.71/8
LeoHyperlink/Sage-1.5-W4A16 (same box and protocol, W4A16 reference)	104.3 t/s	0.06 s	`XPUwNa16LinearKernel` (fused)

Dynamic activation quantization costs ~3x vs the fused W4A16 path on this XPU stack. Coherent output on all runs, byte-identical between plain and speculative decode (temperature 0).

Speculative decoding (DFlash)

Benchmarks

Humanity's Last Exam (HLE): 12.14%

Full text-only set (2,158/2,158 samples), paper-faithful config via inspect_evals/hle (CAIS team's original.yaml-equivalent: hle/json_judge scorer, single attempt, no multi-modal):

Table with columns: Metric, Value
Metric	Value
Original accuracy	12.14% (262/2158; 3 samples unscored)
Stderr	0.70%
Run wall time	4 d 9 h
Tokens	57.6 M total — 54.3 M reasoning (≈25 k reasoning tok/sample, thinking xhigh)

Setup:

Solver = this quant (thinking default/xhigh, temperature 0, max_tokens 32768, streaming forced), served via a vLLM TP=2 OpenAI-compatible endpoint.
Grader = the same quant answering a JSON-judge prompt (max_grader_attempts 1).

Caveats:

Self-graded: no external judge (e.g. o3-mini) was used — the quant judges its own answers. Plan-level judge disagreement adds unknown bias, plausibly downwards.
Text-only, no tools: the base card's HLE 55.1 is with tools; this run is single-shot text-only. The two numbers must not be read against each other as a quantization recovery claim.
Raw per-sample transcripts and scores: benchmarks/hle-full.eval (inspect_ai .eval archive).

SciCode: 4.6% test (3/65), 33.3% val (5/15)

Upstream inspect_ai harness from the SciCode repo (github.com/scicode-bench/SciCode), no background prompts, temperature 0, streaming, test_data.h5 numeric grading:

Table with columns: Split, Main-problem solve, Substep correctness, Tokens
Split	Main-problem solve	Substep correctness	Tokens
test (65 problems)	4.6% (3/65)	30.2% (88/291)	9.0 M
validation (15 problems)	33.3% (5/15)	48.0% (24/50)	0.84 M

Caveats:

Run background-free (zero-shot, no doc-comment context in the prompt) — a strictly harder setting than the paper's default with-background mode; the val:test gap (33% → 4.6%) reflects that the test split is novel research problems rather than routine physics code.
Several substeps (≈12) ended in "fail to extract python code" — the model's output did not yield a parseable script under the harness's extraction rules, scoring 0 regardless of the underlying reasoning quality.
Raw logs: benchmarks/scicode-test.eval, benchmarks/scicode-val.eval.

Tested hardware / software

GPU: Intel Arc Pro B70 32 GB (Battlemage Xe2), single GPU
Serving stack: vLLM-XPU 0.32.0 fork, torch 2.x +XPU, oneCCL
Serve flags: --trust-remote-code --max-model-len 8192 --max-num-seqs 128 --gpu-memory-utilization 0.9 --language-model-only --kv-cache-dtype fp8 --enable-prefix-caching plus --default-chat-template-kwargs '{"enable_thinking": false}'. Note: this is a Mamba-hybrid model — only 159 Mamba cache blocks fit at 0.9 utilization, so max_num_seqs must stay at or below 159 (default 256 fails engine init). ZE_AFFINITY_MASK=0.0 crashes engine init on this build; GPU pinned via ONEAPI_DEVICE_SELECTOR=level_zero:N.

Usage

bash

pip install "vllm>=0.32"   # XPU fork (vllm-xpu) on Intel GPUsvllm serve LeoHyperlink/Qwen3.8-27B-Pi-W4A8 --trust-remote-code \  --max-model-len 8192 --max-num-seqs 128 --gpu-memory-utilization 0.9 --language-model-only

pip install "vllm>=0.32"   # XPU fork (vllm-xpu) on Intel GPUsvllm serve LeoHyperlink/Qwen3.8-27B-Pi-W4A8 --trust-remote-code \  --max-model-len 8192 --max-num-seqs 128 --gpu-memory-utilization 0.9 --language-model-only

Files

model-0000[1-4]-of-00004.safetensors (~18.5 GB) + model.safetensors.index.json — quantized weights (MTP heads mtp.* included in the main checkpoint)
model_extra_tensors.safetensors (849 MB) — raw passthrough tensors, kept for faithfulness. vLLM 0.32.0 cannot load them alongside the CT main checkpoint; text-only serving does not need them.
quantization_config.json — compressed-tensors pack-quantized spec
config.json, tokenizer files, preprocessor/processor configs, chat_template.jinja, generation_config.json
benchmarks/hle-full.eval, benchmarks/scicode-test.eval, benchmarks/scicode-val.eval — raw inspect_ai eval logs (full per-sample transcripts + scores)
.eval_results/hle.yaml — machine-readable HLE score card (Hub eval-results schema; SciCode's dataset is README-only as it is not a registered benchmark)

Limitations

HLE 12.14% is text-only and self-graded (the quant judges itself, no o3-mini judge); the base card's HLE 55.1 is a with-tools agentic setting — see Benchmarks caveats, they are not comparable.
SciCode 4.6% is a no-background zero-shot run; with-background and tool-using settings score materially higher.
Not yet run: SWE-bench Pro and Terminal-Bench 2.1 (harness setup pending).
MTP speculative decoding is NOT enabled by this checkpoint (MTP weights ship in the main checkpoint but are unused; --spec-method qwen3_5_mtp exists in the serving build and is untested). Speculative decoding WAS validated via the DFlash draft path instead (see above).
Tensor-parallel serving is blocked on this box (oneCCL IPC init failure); all serves above are single-GPU.
The visual tower ships unquantized and was not exercised (--language-model-only serving).

Citation

No new training was done for this quant — if you use it, cite the base model (bytkim/Qwen3.8-27B-pi) and link back to this repo for the quantization recipe and serving numbers.