GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Potestates
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Potestates
Model Tree
Qwen/Qwen3.6-27B
Input Modalities
Output Modalities
Supported Functionality
It is a two-stage adapter. The first stage is ordinary supervised fine-tuning on the challenge data. The second stage corrects a specific, measurable failure.
The base fine-tune had learned an unconditional habit of predicting "clip" — about 40% of the class-list answers in the training data contain one, so the model said clip almost regardless of the image. On the in-distribution procedure this is a good bet (answers containing clip were 78.5% correct). On the out-of-distribution procedure it was fatal: 99 out of 99 answers containing clip were wrong.
A dedicated probe showed the habit was not a perception error. The model said clip on 92.2% of frames with no clamps present and 94.2% of frames with clamps present — essentially independent of what was in the picture. It was a language prior, not a misidentification, and it could not be fixed by telling the model the procedure type (every procedure in the training data is clip-heavy, so that field carries no signal).
The correction stage therefore adds out-of-distribution frames where clips are genuinely
absent, forcing the prediction to be grounded in pixels. Frames come from
SAR-RARP50 (EndoVis 2022,
CC BY-NC-SA 4.0), with foreign-object labels derived from its hand-drawn segmentation
masks — suturing needle → Needle; instruments ignored; ambiguous frames dropped.
Measured on 180 held-out prostatectomy frames, the rate of answers wrongly containing a clip fell from 89.4% to 0.6%, and the official out-of-distribution object-recognition bucket rose +6.58 points.
Known trade-off. This adapter buys that gain partly by collapsing its answer diversity on unfamiliar footage (it answers "needle" on 172 of those 180 frames), which cost −2.62 points on the in-distribution object-recognition bucket. Net gain over the uncorrected adapter is +1.14 points. A revision that keeps the correction without the collapse is in progress.
LoRA r=32, alpha=64, dropout=0.05, 16 target module names resolving to 614
modules: the full vision tower (attention, MLP, patch_embed, and the
vision→language merger), and the full language tower — including the 48 Gated
DeltaNet layers of this hybrid-attention model (in_proj_qkv, in_proj_z, in_proj_a,
in_proj_b, out_proj), which a naive target list silently misses.
python
import torchfrom transformers import AutoModelForImageTextToText, AutoProcessorfrom peft import PeftModelfrom PIL import ImageBASE = "Qwen/Qwen3.6-27B"model = AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16, device_map="cuda:0").eval()model = PeftModel.from_pretrained(model, "Potestates/qwen3.6-27b-frame-lora").eval()processor = AutoProcessor.from_pretrained(BASE)image = Image.open("frame.png").convert("RGB") # native resolution — do not resizequestion = "List all foreign objects that are visible in this video frame. " \"Please provide the class names or answer with none."messages = [{"role": "user", "content": [{"type": "image", "image": image},{"type": "text", "text": question},]}]text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)with torch.inference_mode():out = model.generate(**inputs, max_new_tokens=32, do_sample=False)print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:],skip_special_tokens=True)[0].strip().rstrip(" ."))
enable_thinking=False. Leaving the reasoning trace on burns the token budget.Counting (answer_format=number) is the weakest category and has resisted every
intervention we tried: self-consistency (−1.7), high-resolution upsampling (−0.4),
count-weighted loss (−2.1), a weakly-supervised density head (0.000 exact match),
enumeration prompting (flat), and an external detector (−13). Linear probes across
layers read counts worse than the model's own generation, which suggests the count
is not linearly present in the hidden states at all.
Base model: Qwen3.6-27B (Apache-2.0). Correction data: SAR-RARP50, EndoVis 2022 (CC BY-NC-SA 4.0) — non-commercial.