prithivMLmods

Qwen3.5-9B-DS-v4-Flash-v3.0-FP8

README

License: apache-2.0

Quantization Details

Quantization was performed using llmcompressor with the following recipe:

yaml
default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: ['re:.*lm_head', 're:.*embed_tokens$', 're:.*visual.*', 're:.*model.visual.*',
        're:.*linear_attn.*']
      scheme: FP8_DYNAMIC
      bypass_divisibility_checks: false
      requires_calibration_data: false

Linear layers are quantized to FP8 with dynamic per-tensor activation scaling, so no calibration dataset is required (requires_calibration_data: false). The lm_head, embedding table, any vision-tower (visual) components, and linear_attn layers are excluded from quantization and remain at full precision to preserve output-head fidelity and numerical stability.

Table

Base model	prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0
Quantization scheme	FP8_DYNAMIC (Linear layers only)
Format	compressed-tensors
Calibration data required	No (dynamic activation scaling)
Excluded from quantization	lm_head, embed_tokens, visual (if present), linear_attn

Use with vLLM

Qwen3.5-9B-DS-v4-Flash-v3.0-FP8 is served through vLLM with native support for compressed-tensors FP8 checkpoints.

Requirements

torch >= 2.11.0
vllm >= 0.19.1
A GPU with FP8 support recommended (Hopper or Blackwell class) for best throughput; also runs on Ampere with FP8 dequantized on the fly.

Serve

bash
vllm serve prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0-FP8 \
  --max-model-len 32768

Client request

python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

messages = [
    {
        "role": "user",
        "content": "Explain how a transformer model processes text."
    }
]

response = client.chat.completions.create(
    model="prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0-FP8",
    messages=messages,
    temperature=0.0,
    max_tokens=512,
)

print(response.choices[0].message.content)

Quick Start with Transformers

bash
pip install transformers accelerate

python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0-FP8",
    torch_dtype="auto",
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained(
    "prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0-FP8"
)

messages = [
    {
        "role": "user",
        "content": "Explain how a transformer model processes text."
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=512
)

print(
    tokenizer.decode(
        outputs[0][inputs.shape[-1]:],
        skip_special_tokens=True
    )
)

Training Details (Base Model)

Table with columns: Setting, Value
Setting	Value
Base Model	prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0
Original Backbone	Qwen/Qwen3.5-9B
Training Method	Multi-stage Supervised Fine-Tuning (SFT)
Maximum Sequence Length	32,768 tokens (Long Context)
Training Precision	BF16 (Full Precision)
Training & Alignment Framework	TRL
Training Datasets	, , and additional high-quality reasoning datasets

Intended Use and Limitations

Intended use, known limitations, training data composition, and responsible use guidance are unchanged from the base model. See the Qwen3.5-9B-DS-v4-Flash-v3.0 model card for full details.

Reasoning Research: Studying long-context reasoning and multi-stage training techniques.
Mathematical Reasoning: Solving complex mathematical problems with multi-step reasoning.
Scientific Reasoning: Performing structured scientific analysis and problem solving.
Coding Assistance: Improving code understanding and generation through long-context reasoning.
Instruction Following: Evaluating and improving instruction-following capabilities.
Efficient Local Deployment: Reduced memory footprint enables 9B reasoning inference on smaller GPUs.

Limitations

Experimental Model: Behavior may differ from the base model in certain scenarios.
Reasoning Artifacts: Complex reasoning chains may occasionally produce incorrect intermediate steps or conclusions.
Training Biases: Performance reflects the characteristics and coverage of the reasoning datasets used during training.
Quantization Drift: FP8 dynamic quantization may introduce minor numerical differences relative to the bf16 base model; downstream accuracy should be validated for production use cases.

License

Released under the Apache 2.0 License, consistent with the base model.

Acknowledgements

Qwen/Qwen3.5-9B: Base model used for this project.
TRL - Transformers Reinforcement Learning: TRL is a full stack library providing tools to train transformer language models with methods including Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), Direct Preference Optimization (DPO), Reward Modeling, and more.
Transformers: Transformers provides state-of-the-art machine learning models for text, computer vision, audio, video, and multimodal tasks, supporting both inference and training.
llmcompressor: Used to produce the FP8 dynamic quantization for this release.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

prithivMLmods

Model Tree

Base

prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality