GadflyII

GLM-4.7-Flash-NVFP4

README

License: apache-2.0

Quantization Strategy

This model was made via custom quantization and calibration (128 samples, 2048 max seq len, neuralmagic/calibration, all 64 experts) scripts based on NVIDIA's approach for DeepSeek-V3. It uses mixed precision to preserve accuracy:

Table with columns: Component, Precision, Rationale
Component	Precision	Rationale
MLP Experts	FP4 (E2M1)	64 routed experts, 4 active per token
Dense MLP	FP4 (E2M1)	First layer dense MLP
Attention (MLA)	BF16	Low-rank compressed Q/KV projections are sensitive
Norms, Gates, Embeddings	BF16	Standard practice

Performance

Table with columns: Metric, BF16, Uniform FP4, This Model
Metric	BF16	Uniform FP4	This Model
MMLU-Pro	24.83%	16.84%	23.55%
Size	62.4 GB	18.9 GB	20.4 GB
Compression	1x	3.3x	3.1x
Accuracy Loss	-	-8.0%

Usage

Requirements

vLLM: 0.14.0+ (for compressed-tensors NVFP4 support)
transformers: 5.0.0+ (for glm4_moe_lite architecture)
GPU: NVIDIA GPU with FP4 tensor core support (Blackwell, Hopper, Ada Lovelace)

Installation

bash

pip install vllm>=0.14.0pip install git+https://github.com/huggingface/transformers.git

Inference with vLLM

python

from vllm import LLM, SamplingParams model = LLM(    "GadflyII/GLM-4.7-Flash-NVFP4",    tensor_parallel_size=1,    max_model_len=4096,    trust_remote_code=True,    gpu_memory_utilization=0.85,) params = SamplingParams(temperature=0.7, max_tokens=512)outputs = model.generate(["Explain quantum computing in simple terms."], params)print(outputs[0].outputs[0].text)

Serving with vLLM

bash

vllm serve GadflyII/GLM-4.7-Flash-NVFP4 \    --tensor-parallel-size 1 \    --max-model-len 4096 \    --trust-remote-code

Model Details

Base Model: zai-org/GLM-4.7-Flash
Architecture: Glm4MoeLiteForCausalLM
Parameters: 30B total, 3B active per token (30B-A3B)
MoE Configuration: 64 routed experts, 4 active, 1 shared expert
Layers: 47
Context Length: 202,752 tokens (max)
Languages: English, Chinese

Quantization Details

Format: compressed-tensors (NVFP4)
Block Size: 16
Scale Format: FP8 (E4M3)
Calibration: 128 samples from neuralmagic/calibration dataset
Full Expert Calibration: All 64 experts calibrated per sample

Evaluation

MMLU-Pro Overall Results

Table with columns: Model, Accuracy, Correct, Total
Model	Accuracy	Correct	Total
BF16 (baseline)	24.83%	2988	12032
NVFP4 (this model)	23.55%	2834	12032
Difference	-1.28%	-154	-

MMLU-Pro by Category

Table with columns: Category, BF16, NVFP4, Difference
Category	BF16	NVFP4	Difference
Social Sciences	32.70%	31.43%	-1.27%
Other	31.57%	30.08%	-1.49%
Humanities	23.78%	22.56%	-1.22%
STEM	19.94%	18.70%	-1.24%

MMLU-Pro by Subject

Table with columns: Subject, BF16, NVFP4, Difference
Subject	BF16	NVFP4	Difference
Biology	50.35%	47.42%	-2.93%
Psychology	44.99%	42.48%	-2.51%
Economics	36.37%	34.48%	-1.89%
Health	35.21%	34.84%	-0.37%
History

Citation

If you use this model, please cite the original GLM-4.7-Flash:

bibtex

@misc{glm4flash2025,  title={GLM-4.7-Flash},  author={Zhipu AI},  year={2025},  howpublished={\url{https://huggingface.co/zai-org/GLM-4.7-Flash}}}

License

This model inherits the Apache 2.0 license from the base model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

GadflyII

Model Tree

Base

zai-org/GLM-4.7-Flash

Quantized

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Quantization Strategy

Table with columns: Component, Precision, Rationale
Component	Precision	Rationale
MLP Experts	FP4 (E2M1)	64 routed experts, 4 active per token
Dense MLP	FP4 (E2M1)	First layer dense MLP
Attention (MLA)	BF16	Low-rank compressed Q/KV projections are sensitive
Norms, Gates, Embeddings	BF16	Standard practice

Performance

Table with columns: Metric, BF16, Uniform FP4, This Model
Metric	BF16	Uniform FP4	This Model
MMLU-Pro	24.83%	16.84%	23.55%
Size	62.4 GB	18.9 GB	20.4 GB
Compression	1x	3.3x	3.1x
Accuracy Loss	-	-8.0%

Usage

Requirements

vLLM: 0.14.0+ (for compressed-tensors NVFP4 support)
transformers: 5.0.0+ (for glm4_moe_lite architecture)
GPU: NVIDIA GPU with FP4 tensor core support (Blackwell, Hopper, Ada Lovelace)

Installation

bash

pip install vllm>=0.14.0pip install git+https://github.com/huggingface/transformers.git

Inference with vLLM

python

from vllm import LLM, SamplingParams model = LLM(    "GadflyII/GLM-4.7-Flash-NVFP4",    tensor_parallel_size=1,    max_model_len=4096,    trust_remote_code=True,    gpu_memory_utilization=0.85,) params = SamplingParams(temperature=0.7, max_tokens=512)outputs = model.generate(["Explain quantum computing in simple terms."], params)print(outputs[0].outputs[0].text)

Serving with vLLM

bash

vllm serve GadflyII/GLM-4.7-Flash-NVFP4 \    --tensor-parallel-size 1 \    --max-model-len 4096 \    --trust-remote-code

Model Details

Base Model: zai-org/GLM-4.7-Flash
Architecture: Glm4MoeLiteForCausalLM
Parameters: 30B total, 3B active per token (30B-A3B)
MoE Configuration: 64 routed experts, 4 active, 1 shared expert
Layers: 47
Context Length: 202,752 tokens (max)
Languages: English, Chinese

Quantization Details

Format: compressed-tensors (NVFP4)
Block Size: 16
Scale Format: FP8 (E4M3)
Calibration: 128 samples from neuralmagic/calibration dataset
Full Expert Calibration: All 64 experts calibrated per sample

Evaluation

MMLU-Pro Overall Results

Table with columns: Model, Accuracy, Correct, Total
Model	Accuracy	Correct	Total
BF16 (baseline)	24.83%	2988	12032
NVFP4 (this model)	23.55%	2834	12032
Difference	-1.28%	-154	-

MMLU-Pro by Category

Table with columns: Category, BF16, NVFP4, Difference
Category	BF16	NVFP4	Difference
Social Sciences	32.70%	31.43%	-1.27%
Other	31.57%	30.08%	-1.49%
Humanities	23.78%	22.56%	-1.22%
STEM	19.94%	18.70%	-1.24%

MMLU-Pro by Subject

Table with columns: Subject, BF16, NVFP4, Difference
Subject	BF16	NVFP4	Difference
Biology	50.35%	47.42%	-2.93%
Psychology	44.99%	42.48%	-2.51%
Economics	36.37%	34.48%	-1.89%
Health	35.21%	34.84%	-0.37%
History

Citation

If you use this model, please cite the original GLM-4.7-Flash:

bibtex

@misc{glm4flash2025,  title={GLM-4.7-Flash},  author={Zhipu AI},  year={2025},  howpublished={\url{https://huggingface.co/zai-org/GLM-4.7-Flash}}}

License

This model inherits the Apache 2.0 license from the base model.