SamuelBang

LeRF-9B

README

License: apache-2.0

Model Summary

Vision-language models (VLMs) struggle with perspective taking: when a question asks about spatial relations from another entity's or an imagined observer's viewpoint, they often fall back to the camera view. LeRF trains a VLM to build and use an explicit entity-centered reference frame. Given an image and a question, the model decides whether a frame is needed. If it is, it grounds the reference entity and predicts the frame's projected origin and its front / left / up axes. A lightweight renderer draws the frame on the image, and the model reasons over this visual cue. No external perception models or 3D reconstruction are used.

How It Works

Two-turn inference.

Turn 1 (thinking off). The model outputs either a single draw_reference_frame call — reference_object (text), plus origin, axis_front, axis_left, axis_up as [x, y] points in [0, 1000] normalized image coordinates — or the literal string NO_TOOL_CALL when the question can be answered from the camera view.
Tool. The frame is rendered on the image (red = front, green = left, blue = up; a ⊙/⊗ depth marker replaces a strongly foreshortened axis).
Turn 2 (thinking on). The model sees the rendered image (not the numeric coordinates), reasons over it, and answers in \boxed{}.

Training.

Stage I: SFT. Learns reference-entity grounding and projected frame prediction from pose datasets (ImageNet3D, Omni6DPose-SOPE, BEDLAM). About 20% of the data are no-tool-call examples that teach selective tool use.
Stage II: RL (GRPO). Trained on MultihopSpatial and SpatialReasoner-RL with a binary final-answer reward. The frame-prediction turn is excluded from the policy gradient, so only the reasoning/answer turn receives the advantage.

Usage

The model is meant to be used together with the draw_reference_frame tool and the two-turn client from the GitHub repository. The system prompt and tool schema live in tool/system_prompt.txt and tool/prompts.py.

1. Serve with vLLM (OpenAI-compatible API):

bash

vllm serve SamuelBang/LeRF-9B --served-model-name lerf --max-model-len 32768 \    --limit-mm-per-prompt '{"image": 2}' --trust-remote-code

2. Run the client:

bash

git clone https://github.com/bangx7/LeRF.git && cd LeRF/toolpip install pillow requests # single questionpython inference.py --model lerf --image example.jpg \    --question "From the man's perspective, which object is on his left?" \    --options cup bed laptop phone --save_renders renders/ # batch: one JSON per line with image / question / options (and optionally answer, id)python inference.py --model lerf --input questions.jsonl --output results.jsonl

Recommended settings (matching training): temperature 0.6, images downscaled to at most 1M pixels, a 10,240-token thinking budget within a 12,288-token answer turn.

Tested with Python 3.12, CUDA 13.0, PyTorch 2.11, vLLM 0.24.0 and transformers 5.10.4.

Evaluation

Accuracy (%) on OmniSpatial perspective taking (OmniSpatial-PT: Ego / Allo / Hypo), 3DSRBench (Orientation / Multi-Object) and ViewSpatial-Bench (person-perspective Object View Orientation / Relative Direction). Bold marks the best open-source result.

Table with columns: Method, Ego, Allo, Hypo, 3DSR Ori, 3DSR M-Obj, VS P-Obj, VS P-Rel
Method	Ego	Allo	Hypo	3DSR Ori	3DSR M-Obj	VS P-Obj	VS P-Rel
Proprietary
GPT-5.6-Luna (medium)	83.33	49.73	45.78	60.04

See the paper for more baselines, reference-frame estimation accuracy and ablations.

Acknowledgements

Built on Qwen3.5, LLaMA-Factory and verl.

Citation

bibtex

@misc{xiao2026lerflearningreferencecoordinate,      title={LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning},       author={Bang Xiao and Wenqi Jia and Ozgur Kara and Tiancheng Shen and Yibo Yang and Bolin Lai and Junho Kim and James Matthew Rehg},      year={2026},      eprint={2609.36219},      archivePrefix={arXiv},      primaryClass={cs.CV},      url={https://arxiv.org/abs/2609.36219}, }

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

SamuelBang

Model Tree

Base

Qwen/Qwen3.5-9B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Summary

How It Works

Two-turn inference.

Turn 1 (thinking off). The model outputs either a single draw_reference_frame call — reference_object (text), plus origin, axis_front, axis_left, axis_up as [x, y] points in [0, 1000] normalized image coordinates — or the literal string NO_TOOL_CALL when the question can be answered from the camera view.
Tool. The frame is rendered on the image (red = front, green = left, blue = up; a ⊙/⊗ depth marker replaces a strongly foreshortened axis).
Turn 2 (thinking on). The model sees the rendered image (not the numeric coordinates), reasons over it, and answers in \boxed{}.

Training.

Stage I: SFT. Learns reference-entity grounding and projected frame prediction from pose datasets (ImageNet3D, Omni6DPose-SOPE, BEDLAM). About 20% of the data are no-tool-call examples that teach selective tool use.
Stage II: RL (GRPO). Trained on MultihopSpatial and SpatialReasoner-RL with a binary final-answer reward. The frame-prediction turn is excluded from the policy gradient, so only the reasoning/answer turn receives the advantage.

Usage

1. Serve with vLLM (OpenAI-compatible API):

bash

vllm serve SamuelBang/LeRF-9B --served-model-name lerf --max-model-len 32768 \    --limit-mm-per-prompt '{"image": 2}' --trust-remote-code

2. Run the client:

bash

git clone https://github.com/bangx7/LeRF.git && cd LeRF/toolpip install pillow requests # single questionpython inference.py --model lerf --image example.jpg \    --question "From the man's perspective, which object is on his left?" \    --options cup bed laptop phone --save_renders renders/ # batch: one JSON per line with image / question / options (and optionally answer, id)python inference.py --model lerf --input questions.jsonl --output results.jsonl

Recommended settings (matching training): temperature 0.6, images downscaled to at most 1M pixels, a 10,240-token thinking budget within a 12,288-token answer turn.

Tested with Python 3.12, CUDA 13.0, PyTorch 2.11, vLLM 0.24.0 and transformers 5.10.4.

Evaluation

Table with columns: Method, Ego, Allo, Hypo, 3DSR Ori, 3DSR M-Obj, VS P-Obj, VS P-Rel
Method	Ego	Allo	Hypo	3DSR Ori	3DSR M-Obj	VS P-Obj	VS P-Rel
Proprietary
GPT-5.6-Luna (medium)	83.33	49.73	45.78	60.04

See the paper for more baselines, reference-frame estimation accuracy and ablations.

Acknowledgements

Built on Qwen3.5, LLaMA-Factory and verl.

Citation

bibtex

@misc{xiao2026lerflearningreferencecoordinate,      title={LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning},       author={Bang Xiao and Wenqi Jia and Ozgur Kara and Tiancheng Shen and Yibo Yang and Bolin Lai and Junho Kim and James Matthew Rehg},      year={2026},      eprint={2609.36219},      archivePrefix={arXiv},      primaryClass={cs.CV},      url={https://arxiv.org/abs/2609.36219}, }