steven0226

qwen3vl-8b-chartqa-awq

README

License: apache-2.0

Lineage

Base: unsloth/Qwen3-VL-8B-Instruct-unsloth-bnb-4bit
QLoRA adapter: steven0226/qwen3vl-8b-chartqa-lora
Merged reference: steven0226/qwen3vl-8b-chartqa-merged-16bit
This AWQ checkpoint

Formal quantization used an NVIDIA A100-SXM4-40GB, 256 ChartQA calibration examples, maximum calibration length 2,048, seed 3407, llm-compressor 0.12.0, and SMOKE_TEST=false.

Weight files are 7.55 GB versus 17.53 GB for merged 16-bit: 56.9% smaller, or 2.32× compression.

Quality gate

Complete ChartQA test set, relaxed accuracy with 5% numeric tolerance, isolated vLLM subprocesses:

Table with columns: Split, n, Merged 16-bit, AWQ W4A16 g32, Change
Split	n	Merged 16-bit	AWQ W4A16 g32	Change
Human	1,250	77.28%	76.56%	-0.72 pp
Augmented	1,250	95.20%	94.48%	-0.72 pp
Overall	2,500	86.24%	85.52%	-0.72 pp — PASS

The predefined maximum degradation was 2 pp. Paired bootstrap 95% CI for the overall change was [-1.40, -0.04] pp.

Evaluation weight revision: e81d9332446307adc1b219ed326c8e55cead9015.

A100 vLLM benchmark

Run v2-aa4442870cfd; one A100-SXM4-40GB; vLLM 0.25.1+cu129; same 64 ChartQA requests per level; 64 warmup + 64 measured requests; fixed 64-token output; all 8 levels completed with zero failures.

Table with columns: Concurrency, Merged tok/s, AWQ tok/s, AWQ throughput, Merged E2E p95, AWQ E2E p95
Concurrency	Merged tok/s	AWQ tok/s	AWQ throughput	Merged E2E p95	AWQ E2E p95
1	67.29	123.24	+83.2%	1,007.16 ms	562.00 ms
4	231.02	356.95	+54.5%	1,169.61 ms	776.13 ms
8

Latency and throughput benchmark

AWQ lowered TPOT p95 and E2E p95 at every tested concurrency. The tradeoff is TTFT p95 at concurrency 4/8/16, which increased by 9.0%/20.9%/18.8%.

Full artifacts: bench/benchmark_results.json, bench/benchmark_table.md, and eval_quant/results.json.

vLLM usage

Use Linux with the verified CUDA 12.9 stack. Install vLLM before importing torch:

bash
pip install -U uv
uv pip install --system --no-cache \
  "vllm==0.25.1+cu129" "torch==2.11.0+cu129" \
  "torchvision==0.26.0+cu129" "torchaudio==2.11.0+cu129" \
  --extra-index-url https://wheels.vllm.ai/0.25.1/cu129 \
  --extra-index-url https://download.pytorch.org/whl/cu129 \
  --index-strategy unsafe-best-match

python
import base64
from pathlib import Path

from vllm import LLM, SamplingParams

MODEL = "steven0226/qwen3vl-8b-chartqa-awq"
REVISION = "e81d9332446307adc1b219ed326c8e55cead9015"

image_path = Path("chart.png")
image_uri = "data:image/png;base64," + base64.b64encode(image_path.read_bytes()).decode()

messages = [{
    "role": "user",
    "content": [
        {"type": "image_url", "image_url": {"url": image_uri}},
        {"type": "text", "text": "Which category has the highest value?\nAnswer the question using a single word or phrase."},
    ],
}]

llm = LLM(
    model=MODEL,
    revision=REVISION,
    max_model_len=4096,
    limit_mm_per_prompt={"image": 1, "video": 0},
)
params = SamplingParams(temperature=0, max_tokens=64)
answer = llm.chat([messages], sampling_params=params)[0].outputs[0].text
print(answer)

Intended use

Chart question answering and controlled chart-reading experiments
GPU serving with vLLM on compatible NVIDIA hardware
Research and portfolio demonstrations of QLoRA → AWQ deployment

Limitations

Primarily evaluated on English ChartQA short answers; not validated for general documents or safety-critical use.
Fine-tuning gains were concentrated on the augmented subset.
Benchmark outputs were forced to 64 tokens; each level had only 64 measured requests, so p95 is exploratory.
vLLM's ~36 GB reserved memory in this run was controlled by gpu_memory_utilization=0.88 and is not evidence of reduced AWQ VRAM reservation.
Production deployment should add repeated trials, realistic output lengths, and soak testing.

中文摘要

這是 ChartQA 微調後 Qwen3-VL-8B 的 AWQ W4A16 g32 部署版本。語言模型 Linear 權重採 4-bit symmetric group quantization；vision tower 與 lm_head 維持原精度。完整 2,500 題評估為 85.52%，相較 merged 16-bit 下降 0.72 個百分點，通過預設 2 pp 品質門檻。權重檔由 17.53 GB 降至 7.55 GB；A100 vLLM benchmark 在所有測試 concurrency 都提升吞吐並降低 TPOT/E2E p95，但高併發 TTFT p95 較高。

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

steven0226

Model Tree

Base

steven0226/qwen3vl-8b-chartqa-merged-16bit

Quantized

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer