GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
ambient-intelligence-labs
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
ambient-intelligence-labs
Model Tree
Qwen/Qwen3.5-4B
Input Modalities
Output Modalities
Supported Functionality
Given egocentric video arriving as ~8-second chunks plus the user's opening query, the model decides
after each chunk whether to speak or stay silent. Rather than generating
$interrupt$<utterance> / $silent$, it emits a single token — yes or no — and the decision
is read off those two logits:
text
p_interrupt = softmax([logit_no, logit_yes])[1] interrupt if p_interrupt >= tauOperating point: tau = 0.55. Utterances are templated at inference; the challenge metric scores only the timing decision, not the wording.
python
import torchfrom transformers import AutoModelForImageTextToText, AutoProcessorfrom peft import PeftModel BASE = "Qwen/Qwen3.5-4B"proc = AutoProcessor.from_pretrained(BASE)model = PeftModel.from_pretrained( AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16, device_map="cuda"), "ambient-intelligence-labs/egoproactive-4b-lora").eval() YES = proc.tokenizer.encode("yes", add_special_tokens=False)[0]NO = proc.tokenizer.encode("no", add_special_tokens=False)[0] # inputs: cumulative frames (strided to 32) + query + last 4 dialogue turnslogits = model(**inputs).logits[0, -1]p_interrupt = torch.softmax(torch.stack([logits[NO], logits[YES]]).float(), 0)[1].item()speak = p_interrupt >= 0.55The dialogue history is required — without it the model has no way to know it has already spoken and fires on every chunk. Frames are cumulative from the start of the video, strided to a cap of 32, resized to a 512px maximum side.
Fine-tuned on the released validation videos (seen twice) plus a synthetic corpus of 234 clips annotated by a tool-calling video agent that inspects the footage before placing each cue — 13,730 rows, 46.5% interrupt. lr 1e-4 cosine, warmup 0.03, batch size 1 × 8 accumulation, bf16 with gradient checkpointing, 1 epoch.
ambient-intelligence-labs/egoproactive-synth-annotationsbibtex
@techreport{umapathi2026speak, title = {Ambient @ EgoProactive 2026 : Proactive Egocentric Assistance with Visually Grounded Supervision}, author = {Umapathi, Logesh Kumar}, year = {2026}, institution = {Team Ambient}, note = {Wearable AI Challenge @ ECCV 2026, EgoProactive track}}import torchfrom transformers import AutoModelForImageTextToText, AutoProcessorfrom peft import PeftModel BASE = "Qwen/Qwen3.5-4B"proc = AutoProcessor.from_pretrained(BASE)model = PeftModel.from_pretrained( AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16, device_map="cuda"), "ambient-intelligence-labs/egoproactive-4b-lora").eval() YES = proc.tokenizer.encode("yes", add_special_tokens=False)[0]NO = proc.tokenizer.encode("no", add_special_tokens=False)[0] # inputs: cumulative frames (strided to 32) + query + last 4 dialogue turnslogits = model(**inputs).logits[0, -1]p_interrupt = torch.softmax(torch.stack([logits[NO], logits[YES]]).float(), 0)[1].item()speak = p_interrupt >= 0.55@techreport{umapathi2026speak, title = {Ambient @ EgoProactive 2026 : Proactive Egocentric Assistance with Visually Grounded Supervision}, author = {Umapathi, Logesh Kumar}, year = {2026}, institution = {Team Ambient}, note = {Wearable AI Challenge @ ECCV 2026, EgoProactive track}}