Smoke results (WikiText-2, seq_len 2048, MPS)
Table with columns: Variant, Perplexity, Δ vs BF16| Variant | Perplexity | Δ vs BF16 |
|---|
| BF16 (upstream) | 7.9722 | — |
| Affine 4-bit | 8.5230 | +0.55 (+6.9 %) |
| QuadRotor-Full 4-bit | 8.5660 | +0.59 (+7.4 %) |
Random Haar quaternions without calibration are approximately neutral
vs plain affine 4-bit on this model. See the source repo for details
on why and what to try next.
Algorithm
Per group of 64 weight elements:
- Split norm
ρ = ‖x‖₂ from direction x̄ = x/ρ
- Reshape
x̄ into 16 blocks of 4 coordinates
- Sample 16 pairs of unit quaternions from a per-tensor seed
(SHA-256 of tensor name → 32-bit seed → Haar-on-S³)
- Apply
T(v) = q_L · v · q̄_R per block
- Group-wise affine 4-bit quantize the rotated coordinates
Storage
Table with columns: Per group of 64, Bytes| Per group of 64 | Bytes |
|---|
| Quantized codes (uint8, 1 byte/code) | 64 |
scale (FP32) | 4 |
bias (FP32) | 4 |
ρ (FP32) | 4 |
| Total | 76 B / group → ~1 GB for 1.1 B params |
Bit-packed nibble storage (2 codes per byte) would give a further ~2x
reduction. Future work.
Loading
This model uses a custom format (model.safetensors with sub-keys
<tensor>.quantized / .scales / .biases / .rho, plus a
quadrotor.json sidecar). Load via the quadrotor
package:
import json, torchfrom pathlib import Pathfrom huggingface_hub import snapshot_downloadfrom safetensors.torch import load_filefrom quadrotor.quantize import QuadRotorConfigfrom quadrotor.state_dict import decode_state_dict p = Path(snapshot_download("majentik/TinyLlama-1.1B-Chat-v1.0-QuadRotor-Full-4bit"))sidecar = json.loads((p / "quadrotor.json").read_text())cfg = QuadRotorConfig(variant=sidecar["variant"], bits=sidecar["bits"], group_size=sidecar["group_size"])state = load_file(str(p / "model.safetensors"))restored = decode_state_dict(state, cfg, sidecar)
Source
License
Apache 2.0 (inherited from base model).