Lazarus-Ai

ReAligned-Qwen3.5-35B-A3B-NVFP4

README

License: apache-2.0

Overview

This is an NVFP4 quantization of Lazarus-Ai/ReAligned-Qwen3.5-35B-A3B, a Qwen3.5-based MoE model realigned by Eric Hartford (LazarusAI / QuixiAI) to reduce China-state ideological censorship, refusal behavior, and state-narrative framing while preserving the underlying model's general capabilities. See the base model card for the full training method, evaluation, intended use, limitations, and ethical considerations — all of which apply to this checkpoint unchanged.

The quantization shrinks the model from 70.2 GB (BF16) to 21.7 GB (0.31×, ~4.95 effective bits per weight), bringing it within reach of workstation-class multi-GPU boxes.

Quantization

Quantized with QuixiAI/Model-Optimizer, a hardware-agnostic fork of NVIDIA's TensorRT Model Optimizer. The fork removes the CUDA-only assumptions from the quantization pipeline so that calibration, fake-quant, compression, and export run on any PyTorch device — CUDA, ROCm, Intel XPU, Apple MPS, or plain CPU — with vendor kernels kept as optional fast paths.

To prove that point, this entire checkpoint was produced without any NVIDIA hardware: calibrated, quantized, exported, and serving-validated on 4× Intel Arc Pro B60 GPUs (96 GB VRAM total) using the PyTorch XPU backend.

Table

Source checkpoint	`Lazarus-Ai/ReAligned-Qwen3.5-35B-A3B` (BF16, 70.2 GB)
Quantized size	21.7 GB (0.31×, ~4.95 avg bits/weight)
Recipe	`huggingface/qwen3_5_moe/ptq/w4a16_nvfp4-fp8_attn-kv_fp8_cast`
MoE experts, shared-expert MLPs, lm_head	NVFP4 weights (W4A16, block size 16)
Self-attention + linear-attention projections	FP8 (weights and activations)
KV cache	FP8
Routers, norms, embeddings, conv, vision tower	BF16 (unquantized)
Calibration	512 samples, `nemotron-post-training-v3` blend, max calibration

The layer-precision layout intentionally matches the recipe NVIDIA ships for the sibling architecture (nvidia/Qwen3.6-35B-A3B-NVFP4) — the exported hf_quant_config.json was verified pattern-for-pattern against it. The checkpoint is standard ModelOpt HF format and loads anywhere ModelOpt NVFP4 checkpoints are supported.

Reproduce this quant

Works on any PyTorch device (cuda, xpu, mps, or CPU). Reference setup: 4× 24 GB GPUs + 256 GB RAM; ~25 minutes end to end. Smaller GPU budgets work — the sequential device map spills overflow layers to system RAM.

bash
# 1. Get the toolkit
git clone https://github.com/QuixiAI/Model-Optimizer.git
cd Model-Optimizer
pip install -e ".[hf]" --no-build-isolation
pip install -r examples/hf_ptq/requirements.txt

# 2. Authenticate (calibration data is gated on the HF Hub)
hf auth login

# 3. Quantize + export
python examples/hf_ptq/hf_ptq.py \
  --pyt_ckpt_path Lazarus-Ai/ReAligned-Qwen3.5-35B-A3B \
  --recipe huggingface/qwen3_5_moe/ptq/w4a16_nvfp4-fp8_attn-kv_fp8_cast \
  --dataset nemotron-post-training-v3 \
  --calib_size 512 \
  --use_seq_device_map \
  --gpu_max_mem_percentage 0.6 \
  --skip_generate \
  --export_path ./ReAligned-Qwen3.5-35B-A3B-nvfp4

Notes:

--use_seq_device_map --gpu_max_mem_percentage 0.6 caps per-GPU weight placement so the FP4 packing of the large fused-expert tensors has headroom; without it, 24 GB-class cards OOM during export. On GPUs with ≥48 GB you can drop both flags.
--device defaults to auto (current accelerator, CPU fallback) — no flag needed on non-CUDA hardware.
No HF access? Substitute --dataset cnn_dailymail (ungated, slightly less representative calibration).
Verify the result: hf_quant_config.json in the output should list W4A16_NVFP4 (group 16) for experts/shared-expert/lm_head and FP8 for the attention projections, with kv_cache_quant_algo: FP8.

Usage

vLLM

bash
vllm serve Lazarus-Ai/ReAligned-Qwen3.5-35B-A3B-NVFP4 \
  --tensor-parallel-size 2 \
  --max-model-len 8192

Validated end-to-end with vLLM on Intel XPU (TP2 across two Arc Pro B60s, ~10.9 GB weights per GPU). On NVIDIA hardware, native NVFP4 GEMMs require Blackwell-class GPUs; on other hardware the MoE runs through W4A16 / dequantization kernels. FP8-capable GPUs (Ada/Hopper+, MI300+, Intel Arc B-series) execute the attention projections natively.

Prompt format

Use the native Qwen chat template via tokenizer.apply_chat_template. System prompts matter — ReAligned is steerable, and downstream deployers can set tone, refusal boundaries, and policy behavior through the system prompt:

text
<|im_start|>system
You are ReAligned, a helpful, direct, and fact-seeking assistant. Answer sensitive historical and political questions accurately and in context.<|im_end|>
<|im_start|>user
What happened in Tiananmen Square in 1989?<|im_end|>
<|im_start|>assistant

Suggested inference settings

Table with columns: Setting, Suggested value
Setting	Suggested value
Temperature	0.5–0.8
Top-p	0.9–0.95
Repetition penalty	1.0–1.1

Limitations

All limitations of the base model apply. In addition:

Quantization is lossy. NVFP4 weights and FP8 attention introduce small numerical deviations from the BF16 checkpoint. Structural validation and serving canaries pass; full accuracy benchmarks against the BF16 baseline have not yet been published for this checkpoint.
Hardware-dependent kernels. Throughput and numerics depend on which backend serves the NVFP4 MoE layers (native FP4, W4A16, or dequantization-based emulation).

Citation

bibtex
@misc{hartford2026realignedqwen35,
  author       = {Eric Hartford},
  title        = {ReAligned-Qwen3.5},
  year         = {2026},
  organization = {QuixiAI and LazarusAI},
  url          = {https://huggingface.co/QuixiAI/ReAligned-Qwen3.5}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Lazarus-Ai

Model Tree

Base

Lazarus-Ai/ReAligned-Qwen3.5-35B-A3B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality