JC1DA

Qwopus3.8-27B-Flash-INT4-W4A16

README

License: apache-2.0

📊 Benchmark Results

ARC-Easy

Table with columns: Model, Accuracy, Acc Norm
Model	Accuracy	Acc Norm
Qwopus3.8-27B-Flash (FP16)	83.71%	77.06%
Qwopus3.8-27B-Flash (W4A16)	84.13%	78.54%

The W4A16 quantized version matches or slightly exceeds the FP16 original on ARC-Easy.

[!NOTE] Additional benchmarks (MMLU, HumanEval, BBH, WikiText2 PPL) are in progress. Results will be added to this README when complete.

💾 Quantization Details

Table with columns: Parameter, Value
Parameter	Value
Algorithm	AutoRound + GPTQ
Weight bits	4
Activation bits	16
Group size	128
Version	AutoRound 0.12.3

Model size: ~19GB (vs ~54GB FP16) — 65% reduction

The quantization preserves the following layers in FP16 for quality:

Linear attention projection matrices (in_proj_a, in_proj_b) across all 64 layers
MTP (Multi-Token Prediction) module
Visual encoder

🚀 Usage

Works with any framework supporting GPTQ/AutoRound checkpoints:

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Jackrong/Qwopus3.8-27B-Flash-W4A16/Jackrong_Qwopus3.8-27B-Flash-w4g128"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto"
)

Or via vLLM for high-throughput serving:

bash
vllm serve Jackrong/Qwopus3.8-27B-Flash-W4A16/Jackrong_Qwopus3.8-27B-Flash-w4g128 \
    --dtype auto --tensor-parallel-size 1

🎯 Recommended Use Cases

Local MTP speculative decoding where decoder throughput and draft acceptance matter.
Long-running agent workflows with repeated tool calls and iterative edits.
Resource-conscious workloads where wall-clock time and generated-token cost are operational constraints.
Memory-constrained environments where the 65% size reduction enables local deployment on smaller GPUs.

⚠️ Limitations

Accuracy–Efficiency Trade-off: The underlying Flash fine-tuning prioritizes practical inference efficiency. On the reported 2,500-question MMLU-Pro mixed set, Qwopus scores 91.28% versus 92.73% for the Qwen3.8 base comparison.
Reasoning Stability: Although the long-tail behavior is improved in the reported evaluation, edge cases may still exhibit reasoning drift, loops, or inefficient trajectories.
Agent Dependence: Tool-use quality and task completion remain dependent on the prompt, tool environment, orchestration layer, and feedback loop.
Experimental Release: This is an independent experimental release for research, local evaluation, and technical exploration.

🙏 Acknowledgements

Special thanks to:

Qwen for the Qwen3.8-27B base-model foundation.
Jackrong for the Qwopus3.8-27B-Flash fine-tuning.
Unsloth for practical, efficient large-model fine-tuning tooling.
The open-source community for training tools, evaluation methods, and technical discussion.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

JC1DA

Model Tree

Base

Jackrong/Qwopus3.8-27B-Flash

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality