GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
ambient-intelligence-labs
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
ambient-intelligence-labs
Model Tree
Qwen/Qwen3.5-4B
Input Modalities
Output Modalities
Supported Functionality
Given egocentric video arriving as ~8-second chunks plus the user's opening query, the model decides
after each chunk whether to speak or stay silent. Rather than generating
$interrupt$<utterance> / $silent$, it emits a single token — yes or no — and the decision
is read off those two logits:
markdown
p_interrupt = softmax([logit_no, logit_yes])[1] interrupt if p_interrupt >= tau
Operating point: tau = 0.55. Utterances are templated at inference; the challenge metric scores only the timing decision, not the wording.
python
import torchfrom transformers import AutoModelForImageTextToText, AutoProcessorfrom peft import PeftModelBASE = "Qwen/Qwen3.5-4B"proc = AutoProcessor.from_pretrained(BASE)model = PeftModel.from_pretrained(AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16, device_map="cuda"),"ambient-intelligence-labs/egoproactive-4b-lora").eval()YES = proc.tokenizer.encode("yes", add_special_tokens=False)[0]NO = proc.tokenizer.encode("no", add_special_tokens=False)[0]# inputs: cumulative frames (strided to 32) + query + last 4 dialogue turnslogits = model(**inputs).logits[0, -1]p_interrupt = torch.softmax(torch.stack([logits[NO], logits[YES]]).float(), 0)[1].item()speak = p_interrupt >= 0.55
The dialogue history is required — without it the model has no way to know it has already spoken and fires on every chunk. Frames are cumulative from the start of the video, strided to a cap of 32, resized to a 512px maximum side.
Fine-tuned on the released validation videos (seen twice) plus a synthetic corpus of 234 clips annotated by a tool-calling video agent that inspects the footage before placing each cue — 13,730 rows, 46.5% interrupt. lr 1e-4 cosine, warmup 0.03, batch size 1 × 8 accumulation, bf16 with gradient checkpointing, 1 epoch.
ambient-intelligence-labs/egoproactive-synth-annotationsbibtex
@techreport{umapathi2026speak,title = {Ambient @ EgoProactive 2026 : Proactive Egocentric Assistance withVisually Grounded Supervision},author = {Umapathi, Logesh Kumar},year = {2026},institution = {Team Ambient},note = {Wearable AI Challenge @ ECCV 2026, EgoProactive track}}