Uni-MOPD
Strong Starts, Simple Recipes: Unified Multi-Teacher On-Policy Distillation
from Multimodal Specialists
Multi-teacher on-policy distillation (MOPD) integrates separately trained domain
specialists into a unified model: the student generates responses to training
queries, and the teacher of each query's domain supervises them. Existing
methods refine the distillation procedure while the student starts from a base
model or a multi-domain supervised fine-tuning checkpoint, which leaves
capability integration largely to distillation itself.
Unified MOPD (Uni-MOPD) treats the starting point as a design variable and
pairs a strong start with a simple recipe: a training-free uniform merge
of specialists from the same base model initializes the student, and short MOPD
with uniform domain sampling and full-vocabulary reverse KL then refines it.
The bank covers five domains: STEM reasoning, general visual question answering,
document/OCR/chart understanding, GUI grounding, and embodied understanding. All
five specialists are post-trained from the same base checkpoint,
Qwen/Qwen3-VL-30B-A3B-Instruct, an instruction-tuned
mixture-of-experts vision-language model, and follow different post-training
paths, each ending
with reinforcement learning on its own domain. Because they share the base
model, the specialists share one architecture and vocabulary, which permits both
weight-space merging and token-level distillation.
Table with columns: Repository, What it is| Repository | What it is |
|---|
Uni-MOPD | The unified student |
Uni-MOPD-Merge | The uniform merge that initializes the student |
Uni-MOPD-Teacher-STEM | STEM reasoning specialist |
Uni-MOPD-Teacher-VQA | General visual question answering specialist |
Uni-MOPD-Teacher-OCR | Document/OCR/chart understanding specialist (RL from a multi-domain SFT start) |
Uni-MOPD-Teacher-OCR-Bank2 | Document/OCR/chart understanding specialist (OCR-domain SFT and RL) |
Uni-MOPD-Teacher-GUI | GUI grounding specialist |
Uni-MOPD-Teacher-Embodied | Embodied understanding specialist |
Uni-MOPD-Teacher-OCR-Bank2 comes from the paper's second specialist bank.
The two OCR specialists follow different post-training paths: the first
starts its domain reinforcement learning from a multi-domain SFT checkpoint,
which gives it broader coverage, while the second is trained with supervised
fine-tuning and reinforcement learning on the document/OCR domain only.
All of them sit under the AmberHeart namespace; Uni-MOPD-Teacher-GUI is this
repository.
This model
One of the five specialists in the bank. It is post-trained
from the shared base checkpoint along its own domain's path, ending with
reinforcement learning on that domain, and it leads its own domain among the
five.
GUI grounding
For GUI grounding, use this prompt:
You are a GUI grounding agent. Given a screenshot and an element description, locate the element and output a click action. Briefly reason about the target element location, then output the click action as JSON. The expected JSON output format is as follows: {"action": {"name": "click", "button": "left", "click_type": "single", "x": 250, "y": 845}} Here, "name" is the action to perform, "button" is the mouse button, "click_type" is how the click is performed, and "x" and "y" represent the target location on screen. The coordinate system is normalized: output x and y as integers from 0 to 1000, where the entire screenshot maps to this range.
The model answers with brief reasoning followed by the click action as JSON,
with x and y as integers in [0, 1000] normalized to the screenshot.
For instruction-driven grounding (clicking the target of a natural-language
command rather than a described element), only the first paragraph changes:
You are a GUI interaction agent. Given a screenshot and a natural language instruction, identify the target element and output a click action.
The rest of the prompt and the output format are identical.
Weights are bfloat16, in 14 safetensors shards.
Usage
from transformers import AutoProcessor, Qwen3VLMoeForConditionalGeneration model_id = "AmberHeart/Uni-MOPD-Teacher-GUI"model = Qwen3VLMoeForConditionalGeneration.from_pretrained( model_id, dtype="auto", device_map="auto")processor = AutoProcessor.from_pretrained(model_id) messages = [{ "role": "user", "content": [ {"type": "image", "url": "<your-image>"}, {"type": "text", "text": "Describe this image."}, ],}]inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt",).to(model.device)out = model.generate(**inputs, max_new_tokens=128)print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Requires transformers>=4.57.1.
License
Apache License 2.0, inherited from the base model. See LICENSE for the license
text and NOTICE for the statement of modifications.