thotab15

molmo_action_ambiguity

Model Details

Model Description

This model is the ambiguity-detection and clarifying-question generation component of a dialogue system for mobile robots which is used to solve underspecified tasks through dialoge between human and robot. Given a scene image and an instruction, the fine-tuned Molmo-7B-D decides whether the instruction is ambiguous and, if so, generates a clarifying question; otherwise it outputs "none". Unlike prior work that handles only object-reference ambiguity, it resolves action-level ambiguity within a single unified framework.

Developed by: Thota Bhuvana Chandra
Model type: Vision-Language Model (LoRA adapter) for action-ambiguity detection and clarifying-question generation
Language(s) (NLP): English
Finetuned from model: allenai/Molmo-7B-D-0924

Uses

Direct Use

Takes an image and a natural-language instruction as input, determines whether the instruction is ambiguous, and generates an appropriate clarifying question (or "none" when the instruction is clear). Intended for human-robot dialogue in kitchen, household, and office settings.

Downstream Use

Integrated into a robot pipeline: the robot camera feed and user command enter the system; ambiguous commands trigger a clarifying question that resolves the user's intent before the resolved instruction is passed downstream for object localisation and execution, while clear commands route directly onward.

Out-of-Scope Use

Has not been validated on a physical robot platform; evaluation was offline on static image-instruction pairs, so noisy perception and real-time latency are untested.
Supports only a single round of clarification, not multi-turn dialogue.
Trained and evaluated only on kitchen, household, and office scenes with a limited object vocabulary (fruits, bottles, mugs, cups, glasses, bowls, and similar movable items plus surfaces such as tables and placemats).

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Increasing the training-set size could improve recall without affecting precision. Validation on a physical robot platform is recommended before real-world deployment.

How to Get Started with the Model

This repository contains the LoRA adapter (plus the Molmo processor and tokenizer files). At inference you load the base model allenai/Molmo-7B-D-0924, apply this adapter on top, then provide a scene image and a task instruction. The model generates a clarifying question ending in "?" (up to an 80-token limit) when the task is ambiguous, or "none" when it is clear. Temperature-scaled sampling with T = 0.2 is used to balance diversity and determinism.

Requirements

bash
pip install torch transformers peft accelerate einops pillow requests

Molmo uses custom modeling/processing code, so trust_remote_code=True is required. A CUDA GPU with sufficient VRAM for a 7B model is recommended.

Download

You can either let from_pretrained pull the files automatically (shown below), or download the adapter explicitly:

python
from huggingface_hub import snapshot_download

adapter_path = snapshot_download(repo_id="thotab15/molmo_action_ambiguity")

Inference

python
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig
from peft import PeftModel

BASE_MODEL = "allenai/Molmo-7B-D-0924"
ADAPTER    = "thotab15/molmo_action_ambiguity"

# Processor (handles both image and text)
processor = AutoProcessor.from_pretrained(
    BASE_MODEL, trust_remote_code=True, torch_dtype="auto", device_map="auto"
)

# Load base model and apply the fine-tuned LoRA adapter
base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL, trust_remote_code=True, torch_dtype="auto", device_map="auto"
)
model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

# Load a scene image and the instruction
image = Image.open("scene.jpg").convert("RGB")          # or load from a URL
instruction = "Pick up the orange"

inputs = processor.process(images=[image], text=instruction)
inputs = {k: v.to(model.device).unsqueeze(0) for k, v in inputs.items()}

# Generate the clarifying question (or "none")
with torch.no_grad():
    output = model.generate_from_batch(
        inputs,
        GenerationConfig(max_new_tokens=80, do_sample=True, temperature=0.2),
        tokenizer=processor.tokenizer,
    )

generated_tokens = output[0, inputs["input_ids"].size(1):]
response = processor.tokenizer.decode(generated_tokens, skip_special_tokens=True)
print(response)
# Ambiguous instruction -> e.g. "Which orange should I pick up?"
# Clear instruction      -> "none"

Interpreting the output

If the response is a question ending in "?", the instruction was ambiguous; pass the question to the user, collect their answer, and route the resolved instruction to your downstream object-localisation and execution stages.
If the response is "none", the instruction is treated as clear and can be routed directly onward.

Training Details

Training Data

A curated dataset built on the AmbRes framework, extended from object-reference ambiguity to action-level ambiguity. It comprises 40 images and 800 annotations.

Training Procedure

Training Hyperparameters

Adaptation: LoRA on the query and value matrices of the attention layers
Rank: r = 128
Scaling factor: α = 256
Trainable parameters: ~21 million (0.3% of the total model)
Optimizer: AdamW
Learning rate: 1 × 10⁻⁴
Weight decay: 0.01
Gradient clipping: 1.0
Epochs: 3
Batch size: 4
Inference sampling temperature: 0.2
Max generation length: 80 tokens

Evaluation

Testing Data, Factors & Metrics

Testing Data

The test split of the dataset: 400 real-world scenarios (260 ambiguous, 140 unambiguous) from kitchen, office, and household settings, drawn from images disjoint from the training set.

Factors

Results are disaggregated by ambiguity type: WHICH, WHERE, and HOW MUCH.

Metrics

Precision, recall, and F1 for ambiguity classification; task success rate. Task success requires all of: correctly identifying the ambiguity type, generating a clarifying question of the correct type, and returning a valid resolution output after the user's response. For unambiguous instructions, success is determined solely by correct ambiguity-type classification with direct routing to grounding.

Results

Overall performance:

Table with columns: Model, Precision, Recall, F1, Task Success
Model	Precision	Recall	F1	Task Success
Finetuned Molmo-7B-D (this model)	0.96	0.65	0.77	0.62

Model Card Author

Thota Bhuvana Chandra

Framework versions

PEFT 0.13.2

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

thotab15

Model Tree

Base

allenai/Molmo-7B-D-0924

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

Model Details

Model Description

Developed by: Thota Bhuvana Chandra
Model type: Vision-Language Model (LoRA adapter) for action-ambiguity detection and clarifying-question generation
Language(s) (NLP): English
Finetuned from model: allenai/Molmo-7B-D-0924

Uses

Direct Use

Downstream Use

Out-of-Scope Use

Has not been validated on a physical robot platform; evaluation was offline on static image-instruction pairs, so noisy perception and real-time latency are untested.
Supports only a single round of clarification, not multi-turn dialogue.
Trained and evaluated only on kitchen, household, and office scenes with a limited object vocabulary (fruits, bottles, mugs, cups, glasses, bowls, and similar movable items plus surfaces such as tables and placemats).

Recommendations

How to Get Started with the Model

Requirements

bash
pip install torch transformers peft accelerate einops pillow requests

Molmo uses custom modeling/processing code, so trust_remote_code=True is required. A CUDA GPU with sufficient VRAM for a 7B model is recommended.

Download

You can either let from_pretrained pull the files automatically (shown below), or download the adapter explicitly:

python
from huggingface_hub import snapshot_download

adapter_path = snapshot_download(repo_id="thotab15/molmo_action_ambiguity")

Inference

python
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig
from peft import PeftModel

BASE_MODEL = "allenai/Molmo-7B-D-0924"
ADAPTER    = "thotab15/molmo_action_ambiguity"

# Processor (handles both image and text)
processor = AutoProcessor.from_pretrained(
    BASE_MODEL, trust_remote_code=True, torch_dtype="auto", device_map="auto"
)

# Load base model and apply the fine-tuned LoRA adapter
base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL, trust_remote_code=True, torch_dtype="auto", device_map="auto"
)
model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

# Load a scene image and the instruction
image = Image.open("scene.jpg").convert("RGB")          # or load from a URL
instruction = "Pick up the orange"

inputs = processor.process(images=[image], text=instruction)
inputs = {k: v.to(model.device).unsqueeze(0) for k, v in inputs.items()}

# Generate the clarifying question (or "none")
with torch.no_grad():
    output = model.generate_from_batch(
        inputs,
        GenerationConfig(max_new_tokens=80, do_sample=True, temperature=0.2),
        tokenizer=processor.tokenizer,
    )

generated_tokens = output[0, inputs["input_ids"].size(1):]
response = processor.tokenizer.decode(generated_tokens, skip_special_tokens=True)
print(response)
# Ambiguous instruction -> e.g. "Which orange should I pick up?"
# Clear instruction      -> "none"

Interpreting the output

If the response is a question ending in "?", the instruction was ambiguous; pass the question to the user, collect their answer, and route the resolved instruction to your downstream object-localisation and execution stages.
If the response is "none", the instruction is treated as clear and can be routed directly onward.

Training Details

Training Data

A curated dataset built on the AmbRes framework, extended from object-reference ambiguity to action-level ambiguity. It comprises 40 images and 800 annotations.

Training Procedure

Training Hyperparameters

Adaptation: LoRA on the query and value matrices of the attention layers
Rank: r = 128
Scaling factor: α = 256
Trainable parameters: ~21 million (0.3% of the total model)
Optimizer: AdamW
Learning rate: 1 × 10⁻⁴
Weight decay: 0.01
Gradient clipping: 1.0
Epochs: 3
Batch size: 4
Inference sampling temperature: 0.2
Max generation length: 80 tokens

Evaluation

Testing Data, Factors & Metrics

Testing Data

The test split of the dataset: 400 real-world scenarios (260 ambiguous, 140 unambiguous) from kitchen, office, and household settings, drawn from images disjoint from the training set.

Factors

Results are disaggregated by ambiguity type: WHICH, WHERE, and HOW MUCH.

Metrics

Results

Overall performance:

Table with columns: Model, Precision, Recall, F1, Task Success
Model	Precision	Recall	F1	Task Success
Finetuned Molmo-7B-D (this model)	0.96	0.65	0.77	0.62

Model Card Author

Thota Bhuvana Chandra

Framework versions

PEFT 0.13.2

molmo_action_ambiguity

README

Model Details

Model Description

Uses

Direct Use

Downstream Use

Out-of-Scope Use

Recommendations

How to Get Started with the Model

Requirements

Download

Inference

Interpreting the output

Training Details

Training Data

Training Procedure

Training Hyperparameters

Evaluation

Testing Data, Factors & Metrics

Testing Data

Factors

Metrics

Results

Model Card Author

Framework versions

Explore FriendliAI today

README

Model Details

Model Description

Uses

Direct Use

Downstream Use

Out-of-Scope Use

Recommendations

How to Get Started with the Model

Requirements

Download

Inference

Interpreting the output

Training Details

Training Data

Training Procedure

Training Hyperparameters

Evaluation

Testing Data, Factors & Metrics

Testing Data

Factors

Metrics

Results

Model Card Author

Framework versions