Benchmark Evaluation (GPQA-Diamond)
Evaluated using EvalScope on the full 198-question GPQA-Diamond PhD-level science benchmark (physics, chemistry, and molecular biology) with thinking/reasoning enabled:
Table with columns: Model, Precision / Format, GPQA-Diamond Accuracy, Correct / Total, Retention vs. BF16| Model | Precision / Format | GPQA-Diamond Accuracy | Correct / Total | Retention vs. BF16 |
|---|
| Qwen/Qwen3.8-27B (Official) | BF16 | 89.2% | 177 / 198 | 100.0% |
| Denzien/Qwen3.8-27B-W4A16-AutoRound | INT4 (W4A16, g128, iters=500) | 88.4% | 175 / 198 | 99.1% |
Evaluation Details & Verification Artifacts
- Evaluation Framework:
evalscope (openai_api, streaming enabled)
- Sampling Parameters:
temperature=1.0, top_p=0.95, top_k=20, presence_penalty=0.8, max_tokens=49152
- Average Output Length: 14,195 tokens/question (including
<think> reasoning traces)
- Hardware Tested: Dual Intel Arc Pro B70 GPUs (
vLLM with --max-model-len 65536, batch size 16)
- Raw Evaluation Logs: Full predictions, grading reviews, config, and logs are included in
./evals/gpqa_diamond/.
Quantization Recipe
Quantized from the unquantized Qwen/Qwen3.8-27B weights using auto-round on Intel XPU:
auto-round \ --model "Qwen3.8-27B" \ --scheme W4A16 \ --iters 500 \ --nsamples 256 \ --seqlen 2048 \ --batch_size 4 \ --format "auto_round:llm_compressor" \ --device xpu:0 \ --low_gpu_mem_usage \ --output_dir "./Qwen3.8-27B-int4-xpu"
(Note: While quantized on Intel XPU hardware, the auto_round:llm_compressor W4A16 checkpoint uses standard compressed-tensors formatting—symmetric INT4, group size 128—and runs on both Intel XPU and NVIDIA CUDA backends.)
Recommended Sampling Settings
For complex reasoning, math, and science tasks:
temperature: 1.0
top_p: 0.95
top_k: 20
presence_penalty: 0.8 (helps prevent repetitive <think> loops on long chains)
max_tokens: 32768 – 49152
License
This model inherits the Apache 2.0 license from Qwen/Qwen3.8-27B.