labeebaryan

Qwen3.5-0.8B-Base-CliffQuant-W4A16-G128

README

License: apache-2.0

Measured result

Experiment 001 evaluates 64 windows and 16,320 target tokens per environment. Negative deltas favor CliffQuant.

Table with columns: Held-out environment, AbsMax NLL, CliffQuant NLL, CliffQuant - AbsMax
Held-out environment	AbsMax NLL	CliffQuant NLL	CliffQuant - AbsMax
Macro mean	`2.555326`	`2.500527`	`-0.054799`
General	`3.473953`	`3.410937`	`-0.063016`
Code	`1.513257`	`1.454046`	`-0.059211`
Math	`1.631195`	`1.613147`	`-0.018048`
Multilingual	`3.602898`	`3.523977`	`-0.078921`

The frozen release gate allowed at most +0.01 macro NLL regression and +0.02 in any environment. Both gates passed, and all four environments improved.

The held-out reconstruction proxy also passed on 4,096 groups from all 150 target matrices:

Table with columns: Scale policy, Module-macro worst-environment weighted MSE
Scale policy	Module-macro worst-environment weighted MSE
CliffQuant minimax	`0.0002711874`
Pooled-WMSE	`0.0002740030`
AbsMax	`0.0003770210`

Held-out reconstruction proxy

Quantization configuration

Table with columns: Field, Value
Field	Value
Base model	`Qwen/Qwen3.5-0.8B-Base`
Base revision	`dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68`
Format	GPTQ v1
Target weights	150 language-tower linear matrices
Weight bits	4
Activation precision	16-bit model runtime
Group size	128
Symmetric

CliffQuant changes scale selection, not the packed inference format. No custom kernel is required.

Load with GPTQModel

bash
python -m pip install "gptqmodel==7.3.4"

python
import os

os.environ.setdefault("TORCH_COMPILE_DISABLE", "1")

from gptqmodel import BACKEND, GPTQModel

model = GPTQModel.load(
    "labeebaryan/Qwen3.5-0.8B-Base-CliffQuant-W4A16-G128",
    device="cuda:0",
    backend=BACKEND.GPTQ_TORCH,
    trust_remote_code=False,
)

tokens = model.generate(
    "Exact minimax quantization",
    max_new_tokens=16,
)[0]
print(model.tokenizer.decode(tokens, skip_special_tokens=True))

The environment line keeps the portable GPTQ_TORCH path usable on Windows without Triton; remove it if your TorchInductor/Triton setup is working. This exact eager path was exercised against the released checkpoint.

This is a quantized base model, not an instruction-tuned assistant. The upstream model card describes it as intended for fine-tuning, in-context learning experiments, and research or development rather than direct interaction.

Verification

10,000 / 10,000 exact solver comparisons matched exhaustive FP16 enumeration with zero mismatches.
The full CliffQuant scale run completed all 3,883,008 W4/G128 groups.
In 114 groups (0.0029%), a direct-candidate objective tie triggered conservative exhaustive-grid verification.
Independent unpacking checked 150 module-buffer sets and 497,025,024 signed codes.
All 338 non-target tensors remained byte-exact, with zero dense target weights left behind.
Every target module's qweight and scales payload differs from the matched AbsMax checkpoint.
A fresh empty-directory installation passed deterministic text and image-text generation twice.
The fail-closed publication builder independently reloaded both checkpoints, reran held-out NLL, replayed the frozen corpora, recomputed the proxy and bootstrap gates, and regenerated the figures before release.

Important identities:

Table with columns: Artifact, SHA256
Artifact	SHA256
`model.safetensors`	`b88ea7316a858c2691c1e755a5eb73ebfc08e19b45273a481a262f2099893beb`
CliffQuant scale run	`1292f4102dad57b070071c00d92ba43cd096b64b6434b3b9a3b3b41b73db3cf5`
Solver certificate	`a90b704c0d2e044c4a85c9dc45f604eedbfa5306c5f8719f149980b93d502cfd`
Held-out raw NLL arrays	`fdb24aaa192df5a91189eabebeb2831056e06a84f9ff127fae67b70e72e78934`
Publication release manifest	`b2c5b71820320a08028d5ea2069d136020f0c9c5801901695748e2ae9ed9d447`

The checkpoint itself was built from CliffQuant commit d176c4944962d0bcd43a8092059fbf5098653bc6; later publication work does not retroactively change that build provenance.

Calibration and evaluation data

No fine-tuning or additional model training was performed. Pinned dataset revisions were used only to construct deterministic calibration and held-out text windows:

general: WikiText-2 raw train / validation;
code: MBPP train / validation and test;
math: GSM8K train / test; and
multilingual: XNLI train / validation.

Calibration contains 128 total 256-token windows. Held-out evaluation contains 256 disjoint 256-token windows, split equally across the four environments. The repository records the exact revisions, row identities, rendered-text hashes, token-window hashes, and zero-overlap checks.

Limitations and claim boundary

This is one experiment on one base-model size, one quantization format, and one frozen corpus construction. The positive proxy and NLL results do not establish novelty, state of the art, downstream-task accuracy, latency gains, or broad generalization.

Exact breakpoint-based scale optimization already has prior art. CliffQuant's candidate distinction is the exact minimax objective over separately normalized calibration environments, not exact scale optimization by itself. See the prior-art review and claim boundary.

License and attribution

The checkpoint and CliffQuant code are released under Apache-2.0. The base model is also Apache-2.0; retain the upstream Qwen attribution and review its model card before use.

If the method or evidence pipeline helps your work, citation metadata is available in the repository's CITATION.cff.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

labeebaryan

Model Tree

Base

Qwen/Qwen3.5-0.8B-Base

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality