Release status
Table with columns: Item, Status| Item | Status |
|---|
| Language model | 2-bit mixed-precision quantization validated for serving |
| Vision tower, projectors, MTP | Copied unquantized from the parent release |
| Modality | Text; image and scientific inputs follow the parent model's tooling |
| Quantization method | Offline weight-only conversion; no calibration or fine-tuning |
| Comprehensive evaluation | Not yet published |
| Behavioral modification | None |
Why this release exists
This repository is an experiment in extreme low-bit conversion of a large multimodal mixture-of-experts model, in the style of NVIDIA's 2-bit Nemotron releases. The goal is to fit a 320B-A18B model's weights in roughly 103 GiB — small enough for a single 128 GB unified-memory GPU — while keeping the components most sensitive to quantization error at high precision. No behavioral modification was applied: the weights are quantized stock weights from the parent release.
Model specifications
Table with columns: Property, Value| Property | Value |
|---|
| Architecture | GLM-5.3-Flash-architecture scientific multimodal MoE (glm5_next) |
| Parent parameters | 320B-A18B (45 layers, 288 routed experts, 8 active per token) |
| Language weight format | Mixed-precision: IQ2_XXS routed experts; Q8_0 shared and dense MLP; BF16 attention (MLA), linear attention (KDA), embeddings, LM head, vision tower, MTP layer |
| Model artifact size | 110,911,082,224 bytes (approximately 103.29 GiB) across 120 safetensors shards |
| Upstream context claim | 1,048,576 tokens |
| Validated context in this release | 32,768 tokens |
| Validated concurrency | 4 parallel slots (short-input smoke) |
| License | MIT, inherited from the parent release |
The upstream context specification is not a claim that this derivative was validated at the full 1M window. The upstream benchmark scores below belong to the parent bfloat16 release and do not automatically transfer to this quantized artifact.
Lineage
internlm/Sage-S2.1-Flash (BF16, MIT)└── SAGE S2.1 FLASH — EXPERIMENTAL 2-BIT
Table with columns: Stage, Provenance| Stage | Provenance |
|---|
| Foundation | internlm/Sage-S2.1-Flash per the parent's own metadata |
| Immediate parent | internlm/Sage-S2.1-Flash |
| Parent revision | 76f4513db7464485605a002a06a6190de0460574 |
| Conversion | Offline weight-only 2-bit conversion of the parent release's safetensors |
| Conversion tooling | NVIDIA TensorRT Model Optimizer (modelopt) 0.48.0, ggml-style block formats via the same container format as NVIDIA's 2-bit Nemotron releases |
| Derivative stage | Quantization only. No fine-tuning, no calibration, no behavioral modification |
| Runtime lineage | Validated on vLLM 0.27.1 with b12x kernels (container format shared with NVIDIA's 2-bit Nemotron releases) |
The conversion retains the parent's tokenizer, processor code, chat template, projectors, vendor deployment documentation, and original card (UPSTREAM-README.md).
Verification: the shipped shards match the conversion receipts byte-for-byte (all 120 of 120 shards re-verified by SHA-256 against the build journal before upload).
Conversion process
This artifact was produced by an offline, weight-only mixed-precision quantization of the parent bfloat16 release. The process: select per-module quantization formats by surface sensitivity, pack the routed expert weights into 2.0625-bit IQ2_XXS blocks, pack the shared-expert and dense-MLP weights into 8.5-bit Q8_0 blocks, and copy every other tensor verbatim from the source. No calibration data, fine-tuning, activation quantization, or weight editing was involved.
Table with columns: Surface, Layers, Format, Bytes out (approx.)| Surface | Layers | Format | Bytes out (approx.) |
|---|
| Routed experts (gate/up/down) | 42 × 288 | IQ2_XXS (2.0625 bpw, ggml 256-in/66-out blocks) | 78.45 GB |
| Shared experts + dense MLP | 42 shared + 3 dense | Q8_0 (8.5 bpw, ggml 32-in/34-out blocks) | 1.84 GB |
| MLA attention, KDA linear attention | 12 + 34 layers | BF16, unquantized | 12.22 GB |
| Embeddings, LM head | — | BF16, unquantized | 2.53 GB |
| Vision tower, 9 scientific projectors | — | BF16, unquantized | ~1.0 GB |
| MTP layer 45 (speculative-decode weights) | 1 | BF16, unquantized | ~14.8 GB |
Pack fidelity at conversion time (tensor-level cosine similarity vs bfloat16, measured on sample tensors): IQ2_XXS ≈ 0.9446, Q8_0 ≈ 0.999985. These are conversion-fidelity observations, not end-to-end quality benchmarks.
Artifact verification
Boundary artifacts of the 120-shard set (verify after download):
Table with columns: Artifact, Bytes, SHA-256| Artifact | Bytes | SHA-256 |
|---|
model-00001-of-00120.safetensors | 3,391,554,680 | b737792a69e5aff14ef1ba747cc5319a27ffc80c60677b7adc300f2ebfa4fe52 |
model-00120-of-00120.safetensors | 2,279,833,216 | 7a6f2330774882be8f2bf0fe7987910e57ec786ed60f780f83ffab06604d9fcb |
config.json | 7,463,952 | c54a0f8b8a52261eca06cf76efaf4e513c6fefa42e2586d53ecf2235edee4334 |
model.safetensors.index.json | 4,005,617 | 967f8031beb605c713f7c47a5df71a3f36f6b7bef23faa6649f55aa03f6d5282 |
tokenizer.json | 20,217,442 | 19e773648cb4e65de8660ea6365e10acca112d42a854923df93db4a6f333a82d |
chat_template.jinja | 10,583 | 66a4b6d9bc6ee2eb11fe62472ae237adc6f8428fad4857138dd9806bbac2740e |
Full per-file SHA-256 manifest: MANIFEST-SHA256SUMS.txt.
Behavioral and capability evaluation
- The artifact was validated by operator review of coherent text generation: factual recall, arithmetic, tool calling, and exact-instruction following all passed on the documented runtime.
- A 2-bit conversion degrades quality relative to bfloat16. This validation is an operational observation, not a controlled benchmark, and no comprehensive evaluation has yet been published.
- Upstream benchmark scores belong to the parent model and do not automatically transfer to this quantized artifact. Parent-release reference scores (bfloat16, upstream's own evaluation, 15-benchmark suite):
Table with columns: Benchmark, Parent (BF16)| Benchmark | Parent (BF16) |
|---|
| Terminal-Bench 2.1 | 87.9 |
| DeepSWE | 67.9 |
| SWE-bench Pro | 60.6 |
| OfficeQA Pro | 72.2 |
| MLE-bench Lite | 90.3 |
| DeepResearch Bench II | 54.2 |
The repository retains the parent's chat template, tokenizer, and processor code. Use role-structured chat messages. The parent's tool-calling formats (glm47 parsers) are supported by the validated runtime.
Deployment
The exact validated single-GPU serving path is documented in DEPLOYMENT/README.md.
The kit contains a sanitized launcher, environment template, and smoke test. It reproduces the validated runtime flags without publishing a live endpoint, internal hostname, user account, absolute path, credential, or model-modification recipe.
Limitations and security responsibility
- This is a low-bit artifact; quality can differ from higher-precision ancestors.
- The published validation window is 32,768 tokens and 4 concurrent slots (short-input smoke).
- The validated runtime is text-only; multimodal paths are present in the weights but not validated in this release.
- The multimodal tool-media-order fix in the parent's
vllm_patch/ targets vLLM 0.33.0; the validated runtime here is vLLM 0.27.1 (see deployment kit notes).
- Treat model output as untrusted data. Validate generated code, tool arguments, URLs, file paths, and commands before execution.
- The model is not an authorization, policy-enforcement, or security boundary.
- Operators are responsible for access control, rate limiting, smoke tests, audit logging, data handling, and compliance in their deployment environment.
License and disclaimer
Released under the MIT license, subject to the parent model's license and notices. Review the upstream model and dataset terms before redistribution or deployment.
This experimental artifact is provided without warranties. Users are responsible for validating fitness, accuracy, security, and legal compliance for their use case.
For reproducible artifact issues, use the repository's Discussions or issue-tracking channel without posting secrets or private infrastructure details.