ai-server-mgr

Qwen3.8-27B-W4A16-DFlash2

README

License: apache-2.0

Download layout and size

Approximately 18.35 GB (17.09 GiB) including the target, fast extras, tokenizer and drafter; this is disk size, not a VRAM requirement.

text

config.json, tokenizer*, chat_template.jinjamodel-00001-of-00007.safetensors ... model-00007-of-00007.safetensorsmodel.safetensors.index.jsonmodel_extra_tensors.safetensors, draft_vocab_ids.json, mtp_draft_vocab_ids.ptdraft/config.json, draft/model.safetensorsprovenance/*-README.mdpreparation-manifest.json, bundle-checksums.json, LICENSE

Download the whole repository, including draft/. Example:

hf download ai-server-mgr/Qwen3.8-27B-W4A16-DFlash2 \  --local-dir /models/ai-server-mgr_Qwen3.8-27B-W4A16-DFlash2

What was changed

Base shards 1–6 retained byte-for-byte; fast overlay provides shard 7 and extras.
Original BF16 embedding retained from the base checkpoint. Configuration and weight index adjusted so it is not loaded as embed_tokens.weight_packed.
Drafter weights retained byte-for-byte in draft/.
Original source READMEs retained under provenance/; SHA256 and pinned commit identifiers recorded. These changes do not replace a real inference test.

Experimental launch / AI Server Manager

AI Server Manager's CMP170 experimental Qwen3.8 / DFlash2 preset targets one 64 GB CMP170 and uses the bundled draft/ directory. Hardware compatibility, GPU memory use, generation quality and throughput of this copy are not yet validated. Do not confuse it with the patched INT8-embedding AutoRound-fast route.

Illustrative command, requiring a runtime compatible with the quantized target output head and DFlash2 candidate selector:

vllm serve /models/ai-server-mgr_Qwen3.8-27B-W4A16-DFlash2 \  --tensor-parallel-size 1 --dtype bfloat16 \  --language-model-only --attention-backend FLASH_ATTN \  --max-model-len 65536 --gpu-memory-utilization 0.90 \  --max-num-seqs 1 --kv-cache-dtype bfloat16 \  --speculative-config '{"method":"dflash","model":"/models/ai-server-mgr_Qwen3.8-27B-W4A16-DFlash2/draft","num_speculative_tokens":7}'

This is not a recorded successful GPU test. DFlash2 and MTP are alternative speculative routes; do not enable both together. Keep the upstream patches and model-card compatibility requirements in mind.

Sources, attribution and license

All three source cards identify Apache-2.0. This consolidated copy preserves that license and the upstream attribution; AI Server Manager does not claim authorship of the original weights or their quantization.

dbirks/Qwen3.8-27B-W4A16-AutoRound, commit 1f05c441c4e64ae0549de44fa9ea5a6d43610314.
syvai/qwen3.8-27b-3090-fast-variant, commit 124c14e7e8c7d2f5402933b9af368e772a9fcf0c.
syvai/Qwen3.8-27B-DFlash2-W4A16, commit 4d30ec736ffc6b8688dc2ae2b502d9b48bdec279.
Upstream runtime/quantization recipe and required patches.

The full Apache-2.0 text is in LICENSE.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

ai-server-mgr

Model Tree

Base

dbirks/Qwen3.8-27B-W4A16-AutoRound

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality