prithivMLmods

Q3.5-9B-DS-v4-Flash-v2.0-fp8

README

License: apache-2.0

Model Overview

Table with columns: Setting, Details
Setting	Details
Quantized Model	`prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0-fp8`
Underlying Model	prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0
Base Architecture	Qwen/Qwen3.5-9B
Quantization Method	LLM Compressor
Quantization Scheme	`FP8_DYNAMIC` (Linear layers only)
Format	`compressed-tensors`
Calibration Required	No (Runtime dynamic activation scaling)
Excluded from Quantization	`lm_head`, `embed_tokens`, `visual`, `model.visual`, `linear_attn` (preserved in full precision)
License	Apache-2.0

Quantization Details

Quantization was performed using LLM Compressor with dynamic per-tensor activation scaling applied to standard Linear projections. Sensitive layers—including input embeddings, the LM head, linear attention mechanisms, and vision blocks—were excluded from quantization to maintain mathematical reasoning integrity and prevent output degradation.

Quantization Recipe

yaml
default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore:
        - 're:.*lm_head'
        - 're:.*embed_tokens$'
        - 're:.*visual.*'
        - 're:.*model.visual.*'
        - 're:.*linear_attn.*'
      scheme: FP8_DYNAMIC
      bypass_divisibility_checks: false
      requires_calibration_data: false

Key Highlights

High-Throughput FP8: Reduces memory footprint to ~9–10 GB, enabling fast local execution on consumer GPUs (RTX 3090/4090, L40S, A100, H100).
DeepSeek V4 Flash Traces: Fine-tuned on ~3K long-context DeepSeek V4 Flash reasoning traces covering complex mathematics, technical coding, and analytical benchmarks.
Preserved Sensitivity: Unquantized attention projection exceptions (linear_attn) and heads prevent numerical drift during extended multi-step generation.
Native vLLM & Transformers Compatibility: Direct drop-in support via the compressed-tensors standard.

Quick Start & Inference

1. High-Throughput Serving with vLLM (Recommended)

Install vLLM:

bash
pip install vllm

Launch an OpenAI-compatible API server:

bash
vllm serve prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0-fp8 \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90 \
    --trust-remote-code

Run inference via Python:

python
from vllm import LLM, SamplingParams

llm = LLM(
    model="prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0-fp8",
    trust_remote_code=True,
    max_model_len=32768
)

sampling_params = SamplingParams(
    temperature=0.6,
    top_p=0.95,
    max_tokens=2048
)

messages = [
    {"role": "user", "content": "Solve the following problem step-by-step: Let f(x) = x^3 - 3x + 1. Find the number of distinct real roots in the interval [-2, 2]."}
]

outputs = llm.chat(messages, sampling_params)
print(outputs[0].outputs[0].text)

2. Transformers & Compressed-Tensors

bash
pip install transformers compressed-tensors accelerate torch

python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0-fp8"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto",
    trust_remote_code=True
)

messages = [
    {
        "role": "user",
        "content": "Explain how multi-head latent attention reduces KV cache overhead in large language models."
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

with torch.no_grad():
    outputs = model.generate(
        inputs,
        max_new_tokens=1024,
        temperature=0.6,
        top_p=0.95,
        do_sample=True
    )

print(
    tokenizer.decode(
        outputs[0][inputs.shape[-1]:],
        skip_special_tokens=True
    )
)

Model Files & Quantizations

Table with columns: Resource, Link
Resource	Link
Full Precision (BF16)	`prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0`
FP8 Compressed Tensors	`prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0-fp8`
GGUF (llama.cpp)	`prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0-GGUF`

Intended Use

Reasoning Research: High-efficiency research into long-context reasoning chains and distillation behavior.
Mathematical & Scientific Problem Solving: Structured multi-step derivations with lower compute latency.
Constrained VRAM Deployments: Single-GPU local or edge deployment setups requiring 32K context windows without requiring 24GB+ FP16 allocations.

Limitations & Risks

Experimental Output: The underlying model utilizes abliteration and multi-stage distillation; refusal behaviors may be significantly minimized.
Quantization Artifacts: Although FP8 dynamic scaling retains high fidelity, small numerical discrepancies can occasionally manifest in long reasoning chains.
Hardware Support: FP8 native hardware speedups require NVIDIA Ada Lovelace, Hopper, or newer architectures. Older architectures fall back to emulated or dequantized kernels.

Acknowledgements

Qwen Team: For the foundational Qwen/Qwen3.5-9B architecture.
vLLM Project: For llm-compressor and the compressed-tensors specification.
DeepSeek AI: For foundational inspiration and distillation trace topologies.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

prithivMLmods

Model Tree

Base

prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality