malaiwah

GLM-5.2-QSRT-Fruit-Instruct

What is included

13 transformer layers: 3 dense and 10 MoE layers, plus the packaged MTP expert layer.
Hidden size 1,024; MoE intermediate size 512; 256 routed experts per MoE/MTP layer.
2,816 QSRT experts in 11 canonical atom files.
SQG-XOR-Cheb-T12 E4M3 codebook, three-bit trellis payload, fixed P24/P33 pair records, and physical atom rotation.
Canonical qsrt_atoms_v1 storage with complete per-file SHA-256 manifests and a fail-closed QSRT_COMPLETE.json marker.
W4A16 prefill/reference execution and W4A8 decode execution through B12X.
Package-sealed KLD dataset, per-arm request records, per-position BF16 comparisons, objective-contract scorer, and reproduced quality result.

The expert allocation selected by the frozen calibration evidence is recorded in each qsrt-layer-*.json sidecar. Aggregate allocation counts are:

Table with columns: Allocation code, Experts
Allocation code	Experts
`R13=0,R2=0`	2,797
`R13=0,R2=1`	1
`R13=1,R2=0`	11
`R13=1,R2=1`	5
`R13=2,R2=1`	1
`R13=2,R2=2`	1

Size and memory

The apples-to-apples baseline is the complete BF16 tensor set. All three rows below cover the same 5,040,368,896 logical parameters and count only Safetensors files; effective bpw is stored bytes times eight divided by that parameter count, so it includes container and quantization metadata.

Table with columns: Tensor payload, Bytes, GiB, Effective bpw, Relative to BF16
Tensor payload	Bytes	GiB	Effective bpw	Relative to BF16
BF16 source	10,081,800,232	9.3894	16.0017	baseline
Prior SIQ mixed	3,102,116,152	2.8891	4.9236	69.23% smaller
This QSRT model	2,909,352,104	2.7095	4.6177	71.14% smaller

The whole-model rates include 611,183,872 non-routed parameters retained in BF16. Isolating the 4,429,185,024 routed-expert weights gives:

Table with columns: Routed-expert format, Stored bytes, Nominal path bpw, Effective stored bpw
Routed-expert format	Stored bytes	Nominal path bpw	Effective stored bpw
BF16	8,858,370,048	16.0000	16.0000
Prior SIQ mixed (1,856 K3 / 960 K4 experts)	1,879,717,272	3.3409	3.3951
QSRT P24/P33 atoms	1,686,953,224	3.0000	3.0470

QSRT is therefore 10.25% smaller than SIQ on the routed-expert component and 6.21% smaller on the compared tensor files. Package-level totals are not used for the comparison because tokenizer, card, and evaluation evidence files are not model weights. The previous card's 7,593,020,594-byte BF16 row was not the complete BF16 tensor set and has been removed.

W4A8 and W4A16 use the same stored weights, so their loader weight storage is identical; W4A8 changes the decode execution path, not the checkpoint size.

Hugging Face repository-size audit

The Hugging Face model API with blobs=true reported the following immutable snapshot on 2026-08-08. Repository bytes sums every sibling's reported size; Safetensors bytes sums only *.safetensors. These are observed repository payloads, not parameter-count estimates.

Table with columns: Artifact, Revision, Repository bytes, Safetensors bytes
Artifact	Revision	Repository bytes	Safetensors bytes
Fruit Instruct BF16	`678954f6`	10,102,017,674	10,081,800,232
Fruit Instruct prior mixed SIQ	`48452ef3`	3,122,333,594	3,102,116,152

The three full-model rows ground real download/storage scale only. They are not used for Fruit percentage claims because Fruit has 5.04B parameters while the production model has roughly 754B. The apples-to-apples Fruit tensor comparison above remains the codec-size result.

Adjacent-rate evidence

evaluation/fruit-uniform-rate-sweep.json re-encodes the same authenticated expert sample at uniform K2, K3, and K4, with fresh per-endpoint FP16 scales. It measured 16 of 19 predeclared assignments; 3 lacked the minimum routed calibration support and were skipped rather than imputed. These are pre-layer-deduplication expert-local endpoints, not package sizes.

Table with columns: Endpoint, Mean bpw, Mean bytes/expert, Weight NMSE, Captured-H NMSE, Validation routed NMSE
Endpoint	Mean bpw	Mean bytes/expert	Weight NMSE	Captured-H NMSE	Validation routed NMSE
K2	2.0469	402,432	0.082239	0.023648	0.055370
K3	3.0469	599,040	0.020829	0.005855	0.013015
K4	4.0469

Relative to K3, K2 is 3.948x / 4.039x / 4.254x on weight, captured-H, and validation-routed NMSE. K4 is 0.257x / 0.255x / 0.251x on the same metrics.

Runtime and quality qualification

The sealed runtime receipt evaluation/fruit-runtime-qualification.json records the exact BF16 reference, legacy SIQ comparator, and QSRT candidate identities, candidate tensors, producer, GPU and driver, per-arm immutable runtime images, a normalized, path-redacted effective vLLM CLI representation (runtime.argv; <PRIVATE_ROOT> denotes the per-arm runtime-private staging root), environment and software revisions, parsed loader memory fields, decode runs, generation outputs, and full-vocabulary fidelity rows. The package also seals the 32-prompt/5,870-position dataset, all three request records, both per-position BF16 comparison reports, the objective-contract scorer, and its reproduced quality result under evaluation/. Under the fixed hardware, prompt tokens, generation settings, recorded launch order, TP1, and max_num_seqs=1, 3-repetition same-prompt protocol:

Table with columns: Arm, Median client-observed end-to-end generated-token rate (tokens/s), Rate / BF16, Loader weight GiB, Backend / CUDA graph mode
Arm	Median client-observed end-to-end generated-token rate (tokens/s)	Rate / BF16	Loader weight GiB	Backend / CUDA graph mode
BF16	497.20	1.000x	9.730	`inductor` / `FULL_AND_PIECEWISE`
SIQ	484.29	0.974x	3.240	`inductor` / `FULL_AND_PIECEWISE`

All three arms use the same immutable image, software stack, non-eager compilation backend, CUDA-graph mode, and fixed launcher contract; only the model identity, served name, port, and model-specific quantization/load options differ. The rates include request and serving overhead, so they are not decode-only kernel rates or a general throughput benchmark.

Table with columns: Candidate relative to BF16, Mean forward KL, Max forward KL, Top-1 agreement, Top-10 agreement
Candidate relative to BF16	Mean forward KL	Max forward KL	Top-1 agreement	Top-10 agreement
SIQ	0.0530296	3.22484	91.69%	99.93%
QSRT	0.051614	1.9777	92.18%	99.97%

The package-sealed comparison reports record BF16 reference target-token NLL 1.348719, SIQ candidate target-token NLL 1.392518, and QSRT candidate target-token NLL 1.400423. SIQ is closer on this metric even though QSRT is better on every KL/agreement column above.

The package-sealed request records and evaluation/evaluate_quality.py cover 8 matched targeted prompts across BF16, SIQ, and QSRT; rerunning the scorer reproduces evaluation/quality.json and all three 0/8 objective-contract counts. This focused live-runtime suite complements the full-vocabulary fidelity measurement; it is not a standardized leaderboard benchmark.

In the runtime receipt, models.qsrt.revision is 0b9dfa06da2ef89027552cb3c67e34db1963b685; it identifies the qualified candidate and runtime producer, not a Hugging Face commit. The immutable Hub publication identity is the repository revision returned by the Hub API.

Evidence boundary

The completion seal covers every top-level package file and every regular file under evaluation/. qsrt-manifest.json binds every evaluation filename and SHA-256. The KLD dataset fixes 32 document identities, token sequences, and 5,870 prediction positions; both comparison reports agree on every prompt/document/position/target-token mapping. The packaged scorer reproduces the behavior checks and published aggregates from the sealed inputs. Raw-logit chunk hashes remain in the comparison reports, but the raw chunks are not packaged, so the seal supports aggregate/scoring reproduction rather than an independent logits-to-KL recomputation. The adjacent-rate report measures local routed-expert reconstruction on authenticated, document-disjoint calibration rows. These artifacts do not replace workload-specific or standardized benchmark evaluation.

Reproducible runtime

The runtime is pinned to the reviewed commits below:

KQuant encoder: local-inference-lab/kquant#4, encoded with KQuant revision 2d7f784fe262a43b1f782d23845aa3c0188dee58.
B12X kernels: local-inference-lab/b12x#129, packaged producer revision 1e886ab4e0a560e5bc326adfed93ba6680653f63.
vLLM loader: local-inference-lab/vllm#269, packaged producer revision 0b9dfa06da2ef89027552cb3c67e34db1963b685.

The derived image starts from the content-addressed public base docker.io/voipmonitor/vllm@sha256:3230c25ff95f8678a8eeb52a463f0d3b9f96f6ad550418cc51ea12177a55b41c hard-coded by Dockerfile.fruit-qsrt. It installs the exact B12X checkout, copies the base's compiled vLLM extensions into the reviewed source tree, and seals the exact runtime package bytes. MODEL_REVISION resolves the Hub branch once; hf download then uses the resulting immutable commit SHA.

bash
git clone https://github.com/malaiwah/vllm-voipmonitor.git vllm-fruit
git -C vllm-fruit checkout --detach 0b9dfa06da2ef89027552cb3c67e34db1963b685

docker build \
  --file vllm-fruit/Dockerfile.fruit-qsrt \
  --build-arg VLLM_REVISION=0b9dfa06da2ef89027552cb3c67e34db1963b685 \
  --build-arg B12X_REVISION=1e886ab4e0a560e5bc326adfed93ba6680653f63 \
  --tag fruit-qsrt:0b9dfa06da2ef89027552cb3c67e34db1963b685 \
  vllm-fruit

# This digest is an operator-supplied trust root obtained independently of the
# package being authenticated. Never derive it from MODEL_DIR.
test -n "${FRUIT_QSRT_EXPECTED_COMPLETE_SHA256:?set an independently supplied completion digest}"
test "${#FRUIT_QSRT_EXPECTED_COMPLETE_SHA256}" -eq 64

MODEL_REVISION="$(
  curl -fsSL https://huggingface.co/api/models/malaiwah/GLM-5.2-QSRT-Fruit-Instruct \
    | python3 -c 'import json,sys; print(json.load(sys.stdin)["sha"])'
)"
MODEL_DIR="Fruit-QSRT-${MODEL_REVISION}"
test ! -e "${MODEL_DIR}"
hf download malaiwah/GLM-5.2-QSRT-Fruit-Instruct \
  --revision "${MODEL_REVISION}" \
  --local-dir "${MODEL_DIR}"

docker run --rm --gpus '"device=0"' --shm-size=16g \
  --read-only \
  --tmpfs /tmp:rw,exec,nosuid,size=8g \
  --tmpfs /cache:rw,exec,nosuid,size=16g \
  --tmpfs /root/.cache:rw,nosuid,size=1g \
  --publish 8000:8000 \
  --volume "$PWD/${MODEL_DIR}:/model:ro" \
  --env MODEL=/model \
  --env FRUIT_QSRT_EXPECTED_COMPLETE_SHA256="${FRUIT_QSRT_EXPECTED_COMPLETE_SHA256}" \
  fruit-qsrt:0b9dfa06da2ef89027552cb3c67e34db1963b685

The container configuration targets SM120 with CUDA 13.2.1 and PyTorch 2.12.0+cu132 in the content-addressed base, plus nvidia-cutlass-dsl == 4.6.0 in the derived image. The launcher rejects extra vLLM arguments and any value other than TP1, max_num_seqs=1, max_model_len=4096, and max_num_batched_tokens=4096 before importing the GPU runtime. The current B12X sparse-prefill backend requires single-request prefill chunks.

The runtime manifest is an integrity check rooted in the trusted immutable image, not an independent signature. The build host and container operator remain trusted. Run the container read-only, keep writable tmpfs mounts outside /opt/vllm-fruit, /opt/b12x-fruit, and /opt/fruit-runtime, and mount the authenticated model read-only as shown above.

W4A16 is used for prefill and any row count above the W4A8 decode ceiling. W4A8 is selected for decode-sized batches of at most 16 rows. Unsupported shapes, activation modes, metadata, or incomplete manifests fail closed.

Provenance and integrity

Authenticated BF16 source: malaiwah/GLM-5.2-SIQ-Fruit-Instruct-bf16 at immutable revision 678954f65e056a0f508e21eeb9251c655bb9463f.
Authenticated source manifest (safetensors_manifest) SHA-256: 8f23aed5e9b12000ed103a76da772a20730ca53ab7e352d6cb94da2709165245.
Calibration capture ID: 24b290abaddb9eff8d7328a2a22f3c33bb7f6f45b77692393b3549ba0fded0a2.
Calibration manifest SHA-256: be944c8dfc5b550319d26bc2899f0d2ea3f4ca81275ba53034f0cc8ef7b4e9c5.
The encoder authenticated 256 documents / 46,223 tokens from disjoint fit, confirmation, and validation splits.
Full encoding: 2,816 experts, 1,387.92 GPU-seconds, 2.550 GiB peak CUDA allocation.
MANIFEST.sha256, , , , the runtime receipt, KLD dataset and comparisons, matched request records, packaged scorer/result, and bind the published package to the source, producer, and evaluation evidence.

Known limitations

Compact proxy, not the 754B teacher. Capability, knowledge, and long-tail behavior can differ from the full GLM-5.2 model; evaluate it on your workload.
No absolute assistant-quality claim. In the sealed eight-probe suite, BF16, SIQ, and QSRT each passed 0/8 behavior contracts: code, reasoning, instruction following, debugging, structured output, SQL, security, and needle recall. The publication gates establish relative QSRT fidelity, not general assistant capability.
The packaged launcher permits TP1. TP2 atom ownership is unit-tested, but no package-specific TP2 serving benchmark is claimed.
The current sparse-attention prefill backend requires max_num_seqs=1.
The included evidence is a targeted live-runtime qualification rather than a broad standardized downstream benchmark suite.

License

The packaged model files are MIT, matching the authenticated Fruit BF16 source license. B12X and vLLM are Apache-2.0. KQuant is not redistributed in this model repository and remains subject to its upstream repository licensing.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

malaiwah

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

What is included

13 transformer layers: 3 dense and 10 MoE layers, plus the packaged MTP expert layer.
Hidden size 1,024; MoE intermediate size 512; 256 routed experts per MoE/MTP layer.
2,816 QSRT experts in 11 canonical atom files.
SQG-XOR-Cheb-T12 E4M3 codebook, three-bit trellis payload, fixed P24/P33 pair records, and physical atom rotation.
Canonical qsrt_atoms_v1 storage with complete per-file SHA-256 manifests and a fail-closed QSRT_COMPLETE.json marker.
W4A16 prefill/reference execution and W4A8 decode execution through B12X.
Package-sealed KLD dataset, per-arm request records, per-position BF16 comparisons, objective-contract scorer, and reproduced quality result.

The expert allocation selected by the frozen calibration evidence is recorded in each qsrt-layer-*.json sidecar. Aggregate allocation counts are:

Table with columns: Allocation code, Experts
Allocation code	Experts
`R13=0,R2=0`	2,797
`R13=0,R2=1`	1
`R13=1,R2=0`	11
`R13=1,R2=1`	5
`R13=2,R2=1`	1
`R13=2,R2=2`	1

Size and memory

Table with columns: Tensor payload, Bytes, GiB, Effective bpw, Relative to BF16
Tensor payload	Bytes	GiB	Effective bpw	Relative to BF16
BF16 source	10,081,800,232	9.3894	16.0017	baseline
Prior SIQ mixed	3,102,116,152	2.8891	4.9236	69.23% smaller
This QSRT model	2,909,352,104	2.7095	4.6177	71.14% smaller

The whole-model rates include 611,183,872 non-routed parameters retained in BF16. Isolating the 4,429,185,024 routed-expert weights gives:

Table with columns: Routed-expert format, Stored bytes, Nominal path bpw, Effective stored bpw
Routed-expert format	Stored bytes	Nominal path bpw	Effective stored bpw
BF16	8,858,370,048	16.0000	16.0000
Prior SIQ mixed (1,856 K3 / 960 K4 experts)	1,879,717,272	3.3409	3.3951
QSRT P24/P33 atoms	1,686,953,224	3.0000	3.0470

W4A8 and W4A16 use the same stored weights, so their loader weight storage is identical; W4A8 changes the decode execution path, not the checkpoint size.

Hugging Face repository-size audit

Table with columns: Artifact, Revision, Repository bytes, Safetensors bytes
Artifact	Revision	Repository bytes	Safetensors bytes
Fruit Instruct BF16	`678954f6`	10,102,017,674	10,081,800,232
Fruit Instruct prior mixed SIQ	`48452ef3`	3,122,333,594	3,102,116,152

Adjacent-rate evidence

Table with columns: Endpoint, Mean bpw, Mean bytes/expert, Weight NMSE, Captured-H NMSE, Validation routed NMSE
Endpoint	Mean bpw	Mean bytes/expert	Weight NMSE	Captured-H NMSE	Validation routed NMSE
K2	2.0469	402,432	0.082239	0.023648	0.055370
K3	3.0469	599,040	0.020829	0.005855	0.013015
K4	4.0469

Relative to K3, K2 is 3.948x / 4.039x / 4.254x on weight, captured-H, and validation-routed NMSE. K4 is 0.257x / 0.255x / 0.251x on the same metrics.

Runtime and quality qualification

Table with columns: Arm, Median client-observed end-to-end generated-token rate (tokens/s), Rate / BF16, Loader weight GiB, Backend / CUDA graph mode
Arm	Median client-observed end-to-end generated-token rate (tokens/s)	Rate / BF16	Loader weight GiB	Backend / CUDA graph mode
BF16	497.20	1.000x	9.730	`inductor` / `FULL_AND_PIECEWISE`
SIQ	484.29	0.974x	3.240	`inductor` / `FULL_AND_PIECEWISE`

Table with columns: Candidate relative to BF16, Mean forward KL, Max forward KL, Top-1 agreement, Top-10 agreement
Candidate relative to BF16	Mean forward KL	Max forward KL	Top-1 agreement	Top-10 agreement
SIQ	0.0530296	3.22484	91.69%	99.93%
QSRT	0.051614	1.9777	92.18%	99.97%

Evidence boundary

Reproducible runtime

The runtime is pinned to the reviewed commits below:

KQuant encoder: local-inference-lab/kquant#4, encoded with KQuant revision 2d7f784fe262a43b1f782d23845aa3c0188dee58.
B12X kernels: local-inference-lab/b12x#129, packaged producer revision 1e886ab4e0a560e5bc326adfed93ba6680653f63.
vLLM loader: local-inference-lab/vllm#269, packaged producer revision 0b9dfa06da2ef89027552cb3c67e34db1963b685.

bash
git clone https://github.com/malaiwah/vllm-voipmonitor.git vllm-fruit
git -C vllm-fruit checkout --detach 0b9dfa06da2ef89027552cb3c67e34db1963b685

docker build \
  --file vllm-fruit/Dockerfile.fruit-qsrt \
  --build-arg VLLM_REVISION=0b9dfa06da2ef89027552cb3c67e34db1963b685 \
  --build-arg B12X_REVISION=1e886ab4e0a560e5bc326adfed93ba6680653f63 \
  --tag fruit-qsrt:0b9dfa06da2ef89027552cb3c67e34db1963b685 \
  vllm-fruit

# This digest is an operator-supplied trust root obtained independently of the
# package being authenticated. Never derive it from MODEL_DIR.
test -n "${FRUIT_QSRT_EXPECTED_COMPLETE_SHA256:?set an independently supplied completion digest}"
test "${#FRUIT_QSRT_EXPECTED_COMPLETE_SHA256}" -eq 64

MODEL_REVISION="$(
  curl -fsSL https://huggingface.co/api/models/malaiwah/GLM-5.2-QSRT-Fruit-Instruct \
    | python3 -c 'import json,sys; print(json.load(sys.stdin)["sha"])'
)"
MODEL_DIR="Fruit-QSRT-${MODEL_REVISION}"
test ! -e "${MODEL_DIR}"
hf download malaiwah/GLM-5.2-QSRT-Fruit-Instruct \
  --revision "${MODEL_REVISION}" \
  --local-dir "${MODEL_DIR}"

docker run --rm --gpus '"device=0"' --shm-size=16g \
  --read-only \
  --tmpfs /tmp:rw,exec,nosuid,size=8g \
  --tmpfs /cache:rw,exec,nosuid,size=16g \
  --tmpfs /root/.cache:rw,nosuid,size=1g \
  --publish 8000:8000 \
  --volume "$PWD/${MODEL_DIR}:/model:ro" \
  --env MODEL=/model \
  --env FRUIT_QSRT_EXPECTED_COMPLETE_SHA256="${FRUIT_QSRT_EXPECTED_COMPLETE_SHA256}" \
  fruit-qsrt:0b9dfa06da2ef89027552cb3c67e34db1963b685

Provenance and integrity

Authenticated BF16 source: malaiwah/GLM-5.2-SIQ-Fruit-Instruct-bf16 at immutable revision 678954f65e056a0f508e21eeb9251c655bb9463f.
Authenticated source manifest (safetensors_manifest) SHA-256: 8f23aed5e9b12000ed103a76da772a20730ca53ab7e352d6cb94da2709165245.
Calibration capture ID: 24b290abaddb9eff8d7328a2a22f3c33bb7f6f45b77692393b3549ba0fded0a2.
Calibration manifest SHA-256: be944c8dfc5b550319d26bc2899f0d2ea3f4ca81275ba53034f0cc8ef7b4e9c5.
The encoder authenticated 256 documents / 46,223 tokens from disjoint fit, confirmation, and validation splits.
Full encoding: 2,816 experts, 1,387.92 GPU-seconds, 2.550 GiB peak CUDA allocation.
MANIFEST.sha256, , , , the runtime receipt, KLD dataset and comparisons, matched request records, packaged scorer/result, and bind the published package to the source, producer, and evaluation evidence.

Known limitations

Compact proxy, not the 754B teacher. Capability, knowledge, and long-tail behavior can differ from the full GLM-5.2 model; evaluate it on your workload.
No absolute assistant-quality claim. In the sealed eight-probe suite, BF16, SIQ, and QSRT each passed 0/8 behavior contracts: code, reasoning, instruction following, debugging, structured output, SQL, security, and needle recall. The publication gates establish relative QSRT fidelity, not general assistant capability.
The packaged launcher permits TP1. TP2 atom ownership is unit-tested, but no package-specific TP2 serving benchmark is claimed.
The current sparse-attention prefill backend requires max_num_seqs=1.
The included evidence is a targeted live-runtime qualification rather than a broad standardized downstream benchmark suite.

GLM-5.2-QSRT-Fruit-Instruct

README

What is included

Size and memory

Hugging Face repository-size audit

Adjacent-rate evidence

Runtime and quality qualification

Evidence boundary

Reproducible runtime

Provenance and integrity

Known limitations

License

Explore FriendliAI today

README

What is included

Size and memory

Hugging Face repository-size audit

Adjacent-rate evidence

Runtime and quality qualification

Evidence boundary

Reproducible runtime

Provenance and integrity

Known limitations

License