GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
kulsoom-abdullah
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
kulsoom-abdullah
Model Tree
Qwen/Qwen2-VL-7B-Instruct
Input Modalities
Output Modalities
Supported Functionality
Evaluated on 1,000 samples from a held-out patient (REAL-Colon video 002-004), frame-disjoint from training, stratified 50 rows across each of 20 question types.
Read the floor before the accuracy. Seven of the 20 question types have a single gold answer across all 50 of their test rows, so 35% of the test set cannot distinguish a model from a lookup table.
A control with no perception at all. Something that is not a model — it hears nothing, sees nothing, and knows only which of the 20 question types was asked, emitting one canned answer per type — scores:
| no-perception control | full 1,000 | discriminative 650 |
|---|---|---|
| most common training answer per type (a deployable system could do this) | 57.3% | 42.0% |
| most common test answer per type (requires test labels — upper bound only) | 64.8% | 45.8% |
This model scores 57.1% on the full test set and 49.5% on the 650 rows from the 13 question types whose answers vary. So on aggregate it does not beat a lookup table built from training answers; on the discriminative subset it clears that floor by 7.5 points. Aggregate accuracy on this benchmark measures answer priors more than it measures vision — prefer the 650-row figure when comparing systems.
Full floor analysis: https://github.com/kulsoom-abdullah/SurgViVQA-Audio/blob/main/docs/RESULTS.md
| full 1,000 | discriminative 650 | |
|---|---|---|
| base model, question as text | 36.6% | — |
| base model, question as audio | 10.7% | — |
| this adapter, question as audio | 57.1% | 49.5% |
| Qwen3-VL-32B, question as text, zero-shot | 55.6% | 54.3% |
| question type | this model | floor |
|---|---|---|
scope_outside (is the scope outside the patient) | 96% | 50% |
fluid_occlusion_level | 80% | 50% |
Both are balanced 25/25 splits, so these are genuine visual discrimination.
lesion_screen_position; this model sits at the floor, emitting one constant
string. Same resolution, same frames — a capability gap, not a data limit.mucosa_visibility (48% vs 64–80%) and occlusion_check
(50% vs 86–88%).Given audio and no question text, the un-fine-tuned base reproduced the question it had heard on 488 of 1,000 rows, and 472 of those (96.7%) matched the question actually spoken, out of 20 candidates — against a 5% chance rate and a 4.9% permutation null.
Audio comprehension is therefore present before task fine-tuning. What this adapter adds is answer format and partial visual grounding.
The question must be supplied as audio. The text prompt carries no question content.
python
import torch, librosafrom PIL import Imagefrom transformers import (Qwen2VLForConditionalGeneration, AutoProcessor, AutoTokenizer,WhisperFeatureExtractor, BitsAndBytesConfig,)from peft import PeftModelBASE = "kulsoom-abdullah/Qwen2-Audio-7B-Transcription"ADAPTER = "kulsoom-abdullah/surgvivqa-qwen7b-audio"PROMPT = "Answer the question concisely based on the visual and audio evidence."bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True,)base = Qwen2VLForConditionalGeneration.from_pretrained(BASE, quantization_config=bnb, attn_implementation="sdpa", trust_remote_code=True,)model = PeftModel.from_pretrained(base, ADAPTER).eval()tokenizer = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True, use_fast=False)processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct", use_fast=False)processor.tokenizer = tokenizerfe = WhisperFeatureExtractor.from_pretrained("openai/whisper-large-v3-turbo")frames = [Image.open(p).convert("RGB") for p in frame_paths] # 8 frames, 384pxaudio, _ = librosa.load("spoken_question.mp3", sr=16000, mono=True)input_features = fe(audio, sampling_rate=16000, return_tensors="pt").input_features# 1500 audio tokens are injected ahead of the visual content; see# src/evaluate_checkpoint.py in the GitHub repo for the full token layout.
This requires the forked transformers in the GitHub repo. Stock transformers silently
drops the audio pathway — Qwen2VLForConditionalGeneration.forward has no input_features
parameter, so the argument is discarded without error and the model answers from the frames
alone. Verify before trusting any output:
python
import inspectfrom transformers import Qwen2VLForConditionalGeneration as Massert "input_features" in inspect.signature(M.forward).parameters
| Method | QLoRA — 4-bit NF4 base (double quant, bf16 compute) + bf16 LoRA |
| LoRA | r=64, alpha=16, dropout 0.05, bias="none" |
| Target modules | q/k/v/o projections and MLP gate/up/down, all 28 decoder layers (392 tensors) |
modules_to_save | audio_projector [3584, 1280], trained in full precision |
| Optimizer | 8-bit AdamW, lr 2e-4, warmup ratio 0.05 |
| Schedule | 3 epochs requested; early-stopped at step 1,100 (epoch 1.91), patience 3 |
| Best checkpoint |
Data. 2,302 training / 398 eval rows from REAL-Colon videos 002-001/002/003; 1,000 test rows from 002-004, frame-disjoint. Questions rendered to speech with edge-tts across 41 English voices split 23/9/9 with no voice appearing in two splits, stratified by accent region and gender, seeded.
Not latency. Measured on matched hardware, weights, frames, and decoding budget, direct audio input is 0.88× — 13% slower than an ASR-then-text pipeline (1,896 ms vs 1,677 ms, n=100). The entire difference is prefill: the speech encoder emits a fixed 1,500 tokens for a 30-second window while these questions run under 4 seconds, so most of that context is encoded silence. Transcription costs only 73 ms.
The case for direct audio is that it removes transcription errors from the path to the answer, and keeps prosody and background sound available to the model rather than discarding them at a text bottleneck. Neither is exercised by this task, which uses clean synthetic speech. Longer utterances would shift the latency balance, since transcription cost scales with speech length and the audio prefix does not.
en-IN voices are
included as the nearest available South Asian English and are explicitly not labeled
Pakistani-accented.Built on Qwen2-VL and Whisper large-v3-turbo. Data from the REAL-Colon dataset (CC BY). Question templates follow the SurgViVQA benchmark; I am not affiliated with its authors.
step 800, eval_loss 0.0649 (from 0.1300 at step 100) |
| Hardware | 1× RTX 4090, ~4h20m |