ai-babai

Qwen3.5-9B-NF4

README

License: apache-2.0

Google Colab free tier / NVIDIA T4

This release is designed to fit the 16 GB memory class of an NVIDIA T4 and is a practical option for interactive inference in a free Google Colab notebook when Colab assigns a compatible GPU. In bounded CUDA validation, the loaded public checkpoint plus generation peaked at 8,083 MiB of device usage; the highest observed process-group PSS was below 9.1 GiB. The validation GPU was an RTX 2000 Ada constrained to a 15,360 MiB device budget, not a physical T4, so this is a resource-fit claim rather than a direct T4 benchmark.

An NVIDIA T4 has 16 GB of GPU memory according to the official NVIDIA specification. Google's Colab FAQ notes that free GPU access, assigned GPU models, usage limits, and runtime duration vary and are not guaranteed. If a T4 is available in your free-tier session, select a GPU runtime and use the quick start below.

This is the only and default artifact in this repository: 11 sealed model files totaling 7,903,101,799 bytes (7.36 GiB), including four safetensors shards. The weights were derived directly from the exact upstream commit c202236235762e1c871ad0ccb60c8ee5ba337b9a without training.

Quick start

The released artifact was created and reload-verified with these exact core versions:

bash
python -m pip install \
  "torch==2.8.0" \
  "transformers==5.16.1" \
  "bitsandbytes==0.50.2" \
  "safetensors==0.8.0" \
  "tokenizers==0.23.2" \
  accelerate

Install the CUDA build of PyTorch appropriate for your system. The export host used PyTorch 2.8.0+cu128.

python
import torch
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration

model_id = "ai-babai/Qwen3.5-9B-NF4"

processor = AutoProcessor.from_pretrained(model_id)
model = Qwen3_5ForConditionalGeneration.from_pretrained(
    model_id,
    dtype=torch.float16,
    device_map={"": 0},
    attn_implementation="sdpa",
).eval()

The quantization configuration is already stored in config.json; do not pass a second runtime BitsAndBytesConfig when loading this persisted checkpoint.

What is preserved

Base model: Qwen/Qwen3.5-9B.
Base revision requested and observed: c202236235762e1c871ad0ccb60c8ee5ba337b9a.
Method: bitsandbytes 4-bit NF4, double quantization enabled, FP16 compute, uint8 quantization storage, no CPU offload.
Quantization scope follows bitsandbytes conversion with no skip-module list. Quantization state is present in both language-model and visual-tower weight namespaces; the visual tower is not claimed to remain wholly FP16/BF16.
Tokenizer, chat template, image processor, processor configuration, model configuration, and generation configuration are included with the weights.
The public prequantized-receipt.json records every artifact filename, byte size, SHA-256 hash, model fingerprint, semantic hashes, package versions, and the GPU used for export.

The original artifact was reloaded locally on an NVIDIA A40. Its complete parameter/buffer fingerprint and unpacked bitsandbytes quantization-state fingerprint matched the pre-save model exactly; processor and configuration semantic hashes also matched. The receipt records 763 tensors/buffers and 358 parameters carrying quantization state.

These checks establish provenance, packaging integrity, and exact persistence of the runtime NF4 state. They are not a BF16 quality evaluation. This release does not claim that NF4 preserves BF16 accuracy, benchmark scores, latency, throughput, RAM use, or VRAM use.

Post-publication CUDA validation

The exact public artifact at commit acce7768149fd2dd5eaad7d188153afbd948b2c0 was downloaded, checked against all 11 sealed-file hashes, and loaded on an NVIDIA RTX 2000 Ada Generation GPU. All model parameters and buffers were on CUDA. Multiple bounded runs completed successfully at the execution level: generation returned nonempty text, reached EOS, and did not hit the 256-token limit.

However, a literal-transcription check on a neutral 640×640 synthetic PNG (SHA-256 793c2af735b159f08ae2f8726e4c5ecc2dd5aa230c9a7bb2fd5c250598536ba2) failed. The image visibly contains A7 B3, ROOM 204, and TOTAL 51.25. With thinking disabled and deterministic greedy generation:

Return only one short line containing the visible text. Do not explain. returned 100%.
A strict three-field JSON transcription prompt returned 100% for all three fields.

The same result was reproduced through a separate inference path using the same verified artifact and image. Therefore, the checks establish artifact integrity, CUDA loading, forward generation, and EOS completion, but this checkpoint did not correctly transcribe that bounded synthetic test case. No BF16 control was run, so this result does not establish that NF4 quantization caused the failure or measure quality relative to the upstream model.

Verification

After downloading the repository, verify the artifact without loading the model:

bash
python verify_artifact.py /path/to/downloaded/model

Expected immutable identifiers:

Table with columns: Item, Value
Item	Value
Upstream commit	`c202236235762e1c871ad0ccb60c8ee5ba337b9a`
Artifact tree SHA-256	`14216ff3908aae760af777b6e27cee14561ecb6ba6cca0f2ca3fdcd82d1e6bf8`
Model fingerprint SHA-256	`6c0149dea87b8b48b9c08a24f78c775e879cffc43f851b58fadaaeb281bdca99`
Quantized parameter-name SHA-256	`86c7cb22eb31ed86a253b14a9012cca2a3d526c8002653f1794f089555c16b1c`
Original receipt SHA-256	`91612ffbbbeff8fa52f0df154f3d77930d19ca8eb763d4693d7d8a7526074bfc`

Reproduction scripts and the observed package lock are included in the repository root. They pin the upstream commit, reject non-Linux/non-CUDA execution, quantize directly from the upstream checkpoint, save safetensors shards, reload them, and compare semantic and tensor/quantization-state fingerprints before writing a receipt. export_nf4.py is a path-free public reconstruction of the executed method; the exporter_sha256 field in the receipt binds the exact historical exporter source. Rebuilding can produce different serialized bytes if the environment or libraries differ.

Limitations

Validated only on Linux with NVIDIA CUDA; CPU, Apple Silicon/MPS, ROCm, and other accelerators were not tested.
Memory fit was validated under a T4-sized device cap on an RTX 2000 Ada, not on a physical T4. Google Colab free-tier GPU type and availability vary.
The exact core package versions above are the compatibility target. Other versions may change serialization or loading behavior.
This is a post-training weight-only NF4 conversion, not QAT and not a fine-tuned model.
Pre-quantization can reduce repeated load-time conversion work, but it does not by itself establish a speed or memory advantage for your workload.
A bounded synthetic literal-transcription test failed as documented above; do not treat image-text transcription quality as validated by the technical load and generation checks.
The upstream model's capabilities, risks, context-length considerations, and usage guidance still apply. See the upstream model card.

License and attribution

This is a quantized derivative of Qwen/Qwen3.5-9B by the Qwen team. The upstream repository is licensed under Apache License 2.0, which is included as LICENSE. The conversion and release packaging were prepared by ai-babai.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

ai-babai

Model Tree

Base

Qwen/Qwen3.5-9B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality