daudaudinang

Quyet-1.0-Large-Mixed-NVFP4

README

License: apache-2.0

📌 Executive Summary

Quyet-1.0-Large-Mixed-NVFP4 is an optimized, mixed-precision NVFP4 (W4A16_NVFP4) quantized checkpoint of chinhnc/Quyet-1.0-Large (31.3B parameters, Gemma-4-31B-it architecture).

Calibrated with NVIDIA ModelOpt specifically for the NVIDIA Blackwell architecture (GB10 / GB200 / B200, sm_121 / sm_120), this model compresses the 62.5 GB BF16 weights down to 24.3 GB while slashing active serving VRAM from 71.8 GB to 20.5 GB (-71.4% memory footprint).

In extensive empirical benchmarks across both the JevBench standard suite (231 cases) and the EdgeBench Bilingual Enterprise suite (1,000 cases), the Mixed NVFP4 model achieves 97.84% exact decision parity with original BF16 weights, and delivers 3.2x higher serving throughput on an NVIDIA GB10 system.

Associated Benchmark Dataset: daudaudinang/edgebench-bilingual-jev.

🎯 Model Specifications

Table with columns: Parameter, Specification
Parameter	Specification
Base Model	chinhnc/Quyet-1.0-Large (by Chinh Nguyen, Apache-2.0)
Base Architecture	Gemma-4-31B-it with rank-16 merged LoRA fine-tune
Parameters	31.3 Billion (30.7B text parameters)
Quantization Format	W4A16_NVFP4 (ModelOpt Mixed Precision)
Weight Size on Disk	24.3 GB (3 safetensors shards, down from 62.5 GB BF16)
Active VRAM (Weights)	20.5 GB (fits comfortably in single 32GB+ or unified memory GPUs)
Serving Runtime	vLLM >= 0.28.0 with FlashInfer CUTLASS NVFP4 kernels
Target Hardware	NVIDIA Blackwell architecture (GB10, GB200, B200, B100, RTX 50-series)
Supported Primitives	`choice` (categorical selection), `noul` (binary True/False), `score` (ordered scale)
Prompt Format	Quyet Prompt Version 2 (compact structured JSON state, letter-readout tokens)

🔬 Mixed-Precision Quantization Architecture

Quantizing a 31B decision model to 4-bit requires careful precision balancing to preserve calibrated token probabilities (where tiny logit shifts alter decision boundaries):

NVFP4 (E2M1 with Microscopic Scaling): Heavy Linear/MLP projection layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj) are quantized to 4-bit NVFP4, executed via high-throughput CUTLASS FP4 kernels.
BF16 High-Precision Preservation:
- Embeddings & Norm Layers: Preserved in 16-bit BF16 to prevent input representation collapse.
- Vision Tower: Kept in BF16 for multimodal input fidelity.
- Output Readout / LM Head: Tuned with pre-calibrated temperature scaling per question type.

📊 Comprehensive Benchmark Evaluations

All evaluations below were conducted in standalone, isolated conditions on an NVIDIA GB10 (128GB Unified Memory, Blackwell sm_121) server.

1. Standard JevBench Benchmark (231 Cases)

The standard JevBench suite evaluates core decision robustness and parity against the unquantized baseline:

Table with columns: Metric, Quyet-1.0-Large (BF16), Quyet-1.0-Large (Mixed NVFP4), Delta / Gain
Metric	Quyet-1.0-Large (BF16)	Quyet-1.0-Large (Mixed NVFP4)	Delta / Gain
Overall Accuracy	90.91% (210 / 231)	89.61% (207 / 231)	-1.30% gap
Decision Parity	100.0%	97.84% (226 / 231)	Only 5 borderline cases differ
VRAM Consumption	71.8 GB	20.5 GB	-71.4% memory saved
P50 Latency (C=1)	352 ms

Note on Parity: In 226 out of 231 test cases (97.84%), Mixed NVFP4 yields the identical decision as BF16. The 5 differing cases are subtle borderline sentiment cases where probability confidence was within ±0.03 of the threshold.

2. EdgeBench Bilingual Enterprise Benchmark (1,000 Cases)

Dataset: daudaudinang/edgebench-bilingual-jev (1,000 real-world Southeast Asian enterprise scenarios: CSKH, banking, logistics, insurance, healthcare).

Table with columns: Dimension / Category, Quyet BF16 (1,000 cases), Quyet Mixed NVFP4 (1,000 cases), Observations
Dimension / Category	Quyet BF16 (1,000 cases)	Quyet Mixed NVFP4 (1,000 cases)	Observations
Overall Accuracy	86.20% (862 / 1,000)	86.70% (867 / 1,000)	+0.50% higher accuracy in NVFP4
Serving Throughput	0.21 req/s	0.68 req/s	3.2x higher throughput
Total Test Duration	4,797 seconds (~80 min)	1,480 seconds (~24 min)	-69.1% total execution time

Breakdown by Enterprise Task Category (200 cases each):

🏷️ LABEL (Sentiment, Ticket Priority P0/P1, Churn Intent): 98.50% (197/200) — Exceptional precision.
⚡ NOISY (Long context 10k–15.8k chars, Spam & Old Policies): 95.00% (190/200) — High needle-in-a-haystack resilience.
⏱️ TIME (Timezone offsets, SLA 24h, Shipping Cutoff): 92.00% (184/200) — Robust temporal reasoning.
🌐 MIXED (Code-switching vi-en-zh, Logistics jargon): 91.50% (183/200) — Flawless multi-lingual capability.
🧮 CALC (Multi-step tiered math, VAT, cashback): 56.50% (113/200) — Standard limitation for non-CoT decision models.

Breakdown by Language:

Vietnamese (vi) (500 cases): 86.80% (434 / 500)
English (en) (500 cases): 86.60% (433 / 500)
Chinese Manifest Elements (zh) (206 cases): 90.78% (187 / 206)

3. Context Scaling Profile (Latency vs Length on GB10)

Evaluated standalone on context_scaling_100.jsonl with concurrency $C = 1$ :

Table with columns: Context Length (Tokens), Server Compute Latency (P50), Client RTT (P50), Memory Profile
Context Length (Tokens)	Server Compute Latency ( $P 50 $ )	Client RTT ( $P 50 $ )	Memory Profile
128 tokens	224 ms	231 ms	Baseline floor

🚀 How to Deploy & Serve

1. Run with vLLM (Docker)

bash

docker run -d --gpus all \  --name quyet-nvfp4-server \  -p 18091:18091 \  -v /path/to/Quyet-1.0-Large-Mixed-NVFP4:/model \  vllm/vllm-openai:latest \  vllm serve /model \    --served-model-name Quyet-1.0-Large \    --port 18091 \    --host 0.0.0.0 \    --dtype bfloat16 \    --trust-remote-code \    --max-model-len 16384 \    --enable-prefix-caching

2. Decision Inference via Python

Because Quyet answers by evaluating calibrated next-token probabilities of answer letters (A, B, ...), you can use the official quyet client or the lightweight decision wrapper:

python

import requests url = "http://localhost:18090/infer"  # Quyet Decision Bridge payload = {    "state": "Khách hàng Tuấn: Thẻ ATM của tôi bị nuốt ở cây ATM phố Huế, vui lòng khóa thẻ gấp giúp tôi.",    "questions": {        "intent": {            "type": "choice",            "instructions": "Ý định của khách hàng là gì?",            "criteria": {                "khoa_the": "Yêu cầu khóa thẻ khẩn cấp",                "doi_pin": "Yêu cầu cấp đổi mã PIN",                "khac": "Vấn đề khác"            }        },        "urgent": {            "type": "noul",            "instructions": "Yêu cầu này có tính chất khẩn cấp cần xử lý ngay lập tức không?"        }    }} response = requests.post(url, json=payload).json()print("Kết quả suy luận:", response["answers"])# Output:# {#   "intent": {"choice": "khoa_the", "confidence": 1.0, "probabilities": {"khoa_the": 1.0, "doi_pin": 0.0, "khac": 0.0}},#   "urgent": {"noul": 1.0, "confidence": 1.0}# }

📂 Repository Contents

text

Quyet-1.0-Large-Mixed-NVFP4/├── config.json                     # Model architecture configuration├── hf_quant_config.json            # ModelOpt NVFP4 quantization metadata├── quyet_config.json               # Calibrated temperature parameters├── generation_config.json          # Generation hyperparameters├── model.safetensors.index.json    # Shard index mapping├── model-00001-of-00003.safetensors # Shard 1 (9.92 GB)├── model-00002-of-00003.safetensors # Shard 2 (9.29 GB)├── model-00003-of-00003.safetensors # Shard 3 (5.12 GB)├── tokenizer.json                  # Gemma-4 fast tokenizer (32.1 MB)├── tokenizer_config.json           # Tokenizer settings├── chat_template.jinja             # Jinja chat template├── README.md                       # Comprehensive model card├── LICENSE                         # Apache-2.0└── NOTICE                          # Attribution notice

📚 Citations & Acknowledgements

Quyet 1.0 Base Model: Chinh Nguyen (chinhnc/Quyet-1.0-Large).
Architecture: Gemma-4 by Google DeepMind (Apache-2.0).
Quantization & Benchmarking: VCCorp AI Team on NVIDIA GB10 Blackwell.
Benchmark Datasets: EdgeBench-JEV.

bibtex

@misc{quyet_mixed_nvfp4_2026,  title  = {Quyet-1.0-Large-Mixed-NVFP4: High-Throughput Calibrated Decision Model on NVIDIA Blackwell},  author = {VCCorp AI Team},  year   = {2026},  url    = {https://huggingface.co/daudaudinang/Quyet-1.0-Large-Mixed-NVFP4}} @dataset{edgebench_jev_2026,  title  = {EdgeBench-JEV: Enterprise Operational Decision and Bilingual Reasoning Benchmark},  author = {VCCorp AI Team},  year   = {2026},  url    = {https://huggingface.co/datasets/daudaudinang/edgebench-bilingual-jev}}

License

Licensed under the Apache License, Version 2.0. Please retain the NOTICE file when redistributing this model or any derivative work.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider