What was done
- FP32 → BF16 for all large matrices (small 1-D tensors kept as stored).
- Re-sharded so every decoder layer sits in one file (NVFP4 fuses scales across qkv and gate/up).
- Quantized with llm-compressor
model_free_ptq, scheme NVFP4A16 (data-free, no calibration set),
running directly on safetensors. That path keeps the MTP head, which a transformers-based path drops.
- Kept in BF16:
lm_head, embeddings, the vision tower, the MTP head, conv1d, and the small
linear_attn.in_proj_a / in_proj_b projections.
mtp.* is in quantization_config.ignore. If vLLM read the BF16 MTP head as 4-bit, speculative decoding
would silently run at ~0% acceptance.
Verified after export: 400 quantized linears, 15 MTP tensors, MTP on the ignore list, 20.6 GB of weights.
Made on a free Google Colab T4 runtime, streaming one source shard at a time to fit Colab's disk.
The script is included as pipeline.py.
Run with vLLM (2× 16 GB GPUs, no P2P)
docker run -d --name vllm-feyospace --gpus all --ipc=host --shm-size 16g \ -p 127.0.0.1:8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -e NCCL_P2P_DISABLE=1 -e NCCL_IB_DISABLE=1 -e NCCL_CUMEM_ENABLE=0 \ vllm/vllm-openai:latest \ Anonym0usWork1221/feyospace-nvfp4a16 \ --dtype bfloat16 --tensor-parallel-size 2 --disable-custom-all-reduce \ --max-model-len auto --max-num-seqs 1 --max-num-batched-tokens 4096 \ --gpu-memory-utilization 0.88 --kv-cache-dtype fp8 \ --language-model-only --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder
Needs a vLLM build with Qwen3.8 support and a Blackwell GPU for NVFP4 kernels.
Check SpecDecoding metrics in the logs: a mean acceptance length around 2 or more means MTP works.
Honest caveats
- Uncensoring is not measured. The source repo's
evaluation.json has only a mechanistic
refusal-direction score (mean residual 20.1 → 16.3) and marks the behavioural test as "not tested".
- Benchmark files (
secbench_comparison.json, seceval_comparison.json) are the original publisher's,
measured on the unquantized model, not on this build.
- No accuracy evaluation was run on this quant. NVFP4 W4A16 usually costs a small amount of quality.
- W4A16 may decode slightly slower than a W4A4 NVFP4 build on the same hardware.
- Avoid TurboQuant KV (
turboquant_k8v4) with this model family on vLLM dev builds. In our tests it produced
repeated-token garbage. Use --kv-cache-dtype fp8.
- Always run a quick correctness check (a needle-in-context question) before trusting speed numbers.
Credits
All model training credit belongs to the Feyospace authors (arXiv 2609.08418) and to OBLITERATUS for the
abliteration. This repo only changes the storage format. License follows the source: Apache-2.0.