groxaxo

Qwen3.6-27B-GPTQ-Pro-4bit

README

License: apache-2.0

Overview

Qwen3.6-27B-GPTQ-Pro-4bit is a GPTQ-quantized checkpoint intended for efficient GPU inference, published by groxaxo. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

Table with columns: Field, Details
Field	Details
Format	GPTQ
Source / base	`Qwen/Qwen3.6-27B`
Intended task	`image-text-to-text`
License	`apache-2.0`

What is included

*.safetensors (5 files)
config.json
generation_config.json
tokenizer.json
tokenizer_config.json
processor_config.json
chat_template.jinja
quantize_config.json
Additional configuration, tokenizer, processor, or shard files (14 visible artifacts total)

Quick start

vLLM (documented configuration)

bash
vllm serve groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit \
  --quantization gptq_marlin \
  --dtype float16 \
  --trust-remote-code

This command is taken from the repository documentation. Adjust tensor parallelism, context length, and cache settings to match your hardware and vLLM version.

Compatibility and responsible use

Use a runtime that explicitly supports this format, architecture, and modality.
Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
Review the source model card and license before redistribution or deployment.
Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

Welcome to Qwen36-27B-GPTQ-Pro-4Bit – a titan of reasoning and generation, elegantly squeezed into a remarkably efficient 4-bit package. It punches leagues above its weight class while keeping your VRAM happy and your inference speeds blazingly fast! Thank you Qwen team for another amazing model.

🌟 Why the "Pro"?

This isn't your average quantization. We used the GPTQ-Pro framework combined with the FOEM (First-Order Error Metric) approach. This advanced technique carefully preserves the most critical weights during the 4-bit compression process by evaluating the exact impact of quantization on the model's loss landscape.

The result?

Near-Lossless Performance: Enjoy the profound reasoning, coding prowess, and vast knowledge of a 27 Billion parameter model, but with a drastically reduced memory footprint.
Marlin Optimized: Ready out-of-the-box for Marlin kernels to deliver maximum token-per-second throughput in serving engines like vLLM.
Consumer Hardware Friendly: Fit a massive 27B powerhouse model on consumer GPUs with room to spare for massive context lengths!

This repository contains a 4-bit GPTQ-Pro quantization of unsloth/Qwen3.6-27B, produced with GPTQModel and the FOEM/GPTAQ-style quality settings used in the GPTQ-Pro project.

Source project: https://github.com/groxaxo/GPTQ-Pro

Deployment

vLLM

bash
CUDA_VISIBLE_DEVICES=0,1 vllm serve groxaxo/Qwen3.6-27B-GPTQ-Pro-4Bit \
  --dtype float16 \
  --quantization gptq_marlin \
  --disable-custom-all-reduce \
  --tensor-parallel-size 2 \
  --max-model-len 132144 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --gpu-memory-utilization 0.92

Local path

bash
CUDA_VISIBLE_DEVICES=0,1 vllm serve /path/to/Qwen3.6-27B-GPTQ-Pro-4Bit \
  --dtype float16 \
  --quantization gptq_marlin \
  --disable-custom-all-reduce \
  --tensor-parallel-size 2 \
  --max-model-len 132144

Transformers

python
from gptqmodel import BACKEND, GPTQModel

model = GPTQModel.load(
    "groxaxo/Qwen3.6-27B-GPTQ-Pro-4Bit",
    backend=BACKEND.GPTQ_MARLIN,
    device="cuda:0",
)

print(model.generate("Write a short deployment checklist.", max_new_tokens=64)[0])

Notes

Tested with tensor parallel size 2 on RTX 3090 GPUs.
Use float16 and gptq_marlin for the most reliable vLLM startup path.
The quantization and serving workflow lives in the GPTQ-Pro repository above.
MTP/speculative decoding is detected by vLLM for this model, but on 2x RTX 3090 the exact --max-model-len 262144 launch OOMs during KV-cache setup.
The working local vLLM configuration I verified is --max-model-len 65536 with --enforce-eager; that starts and serves, but the current metrics showed spec_decode_num_accepted_tokens_total=0, so it does not improve speed yet.
If you test MTP, use --speculative-config '{"method":"mtp","num_speculative_tokens":2}' and disable thinking in the request payload when you want a plain answer.

⚡ Speed Benchmarks

Tested on 2× NVIDIA RTX 3090 with vLLM (gptq_marlin, tensor-parallel=2, float16).

Table with columns: Metric, Value
Metric	Value
Avg Generation Speed	64.0 tok/s
Median Generation Speed	64.0 tok/s
Peak Generation Speed	65.0 tok/s
Avg Time-to-First-Token	54 ms
Median TTFT	56 ms

Test 1: Short Prompt → 256 Tokens (Streaming)

Table with columns: Run, TTFT, Tokens, Speed, Total Time
Run	TTFT	Tokens	Speed	Total Time
1	60 ms	256	64.0 tok/s	4.04s
2	55 ms	256	64.0 tok/s	4.04s
3	56 ms	256	62.4 tok/s	4.14s

Test 2: Medium Prompt → 512 Tokens (Non-Streaming)

Table with columns: Run, Tokens, Speed, Total Time
Run	Tokens	Speed	Total Time
1	512	62.9 tok/s	8.15s
2	512	63.0 tok/s	8.13s
3	512	62.9 tok/s	8.14s

Test 3: Short Burst → 64 Tokens (Streaming)

Table with columns: Run, TTFT, Tokens, Speed
Run	TTFT	Tokens	Speed
1	50 ms	64	65.0 tok/s
2	56 ms	64	64.9 tok/s
3	56 ms	64	64.7 tok/s
4	54 ms	64	64.9 tok/s
5

📊 Quality Evaluation

Wikitext-2 test perplexity: 6.366 (n_ctx=1024)

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider