Usage
- Input: Global task instruction, subtask goal, and an ordered history of scene and wrist images. Use up to eight timestamps with 224 x 224 images from both views.
- Output:
DONE, NOT_DONE, or UNKNOWN.
Loading
Use PyTorch, Transformers with Qwen3-VL support, PEFT, and Accelerate. This example assumes a bfloat16-capable GPU.
import torchfrom transformers import AutoProcessor, Qwen3VLForConditionalGenerationfrom peft import PeftModel BASE_ID = "Qwen/Qwen3-VL-4B-Instruct"BASE_REVISION = "ebb281ec70b05090aa6165b016eac8ec08e71b17"ADAPTER_ID = "FEGE16/robot-task-observer-completion-history" processor = AutoProcessor.from_pretrained(BASE_ID, revision=BASE_REVISION)base = Qwen3VLForConditionalGeneration.from_pretrained( BASE_ID, revision=BASE_REVISION, dtype=torch.bfloat16, device_map="auto",)model = PeftModel.from_pretrained(base, ADAPTER_ID).eval()
Provide the task prompt and image history using the base model's multimodal chat format, then call model.generate.
Project overview and method figure
License: Apache 2.0. See LICENSE and NOTICE.