What's in This Release (v0.1, T2 stage)
This repository currently hosts Wukong-T2-Adapter: a QLoRA adapter (rank 16, 51.5M trainable params, 206MB) on top of unsloth/Qwen2.5-VL-7B-Instruct, fine-tuned for visual-spatial question answering from egocentric images.
adapter_config.json + adapter_model.safetensors — the trained adapter (final checkpoint, step 49730)
- Tokenizer / preprocessor / chat template files — everything needed to load and chat
eval_results.jsonl — per-item evaluation records (question, reference, prediction, score) for full reproducibility
Training setup: 198,919 samples (195,919 VSI-style spatial QA derived from Hypersim + 3,000 programmatic egocentric↔allocentric coordinate-transform QA), 2 epochs, total batch size 8, ~18.5 hours on a single RTX 5090 with Unsloth. Final training loss ≈ 0.19.
Roadmap stages beyond T2 (semantic waypoint prediction, embodiment-specific action adapters, world-action predictive pathway) are under active development and will be released as separate artifacts.
Evaluation
Held-out spatial QA split from our own data pipeline (never seen in training, n=500), scored in VSI-Bench style: exact match for multiple-choice items; MRA (max(0, 1 − |pred−ref|/ref)) for numeric items.
Table with columns: Question type, n, Score| Question type | n | Score |
|---|
| relative_direction_camera | 57 | 0.877 |
| relative_distance_object | 130 | 0.846 |
| relative_size_object | 23 | 0.826 |
| relative_direction_object | 108 | 0.815 |
| absolute_count | 38 | 0.805 |
| absolute_size_object | 27 |
Breakdown by answer format: multiple-choice (n=345) accuracy 0.832; numeric (n=155) MRA 0.671.
Table with columns: Benchmark, What it measures, Status| Benchmark | What it measures | Status |
|---|
| VSI-style spatial QA (own held-out split) | Visual-spatial intelligence from egocentric images | 0.782 (see above) |
| VSI-Bench (official) | Visual-spatial intelligence from egocentric video | Planned |
| SPAR-Bench | 3D spatial perception and reasoning | Planned |
| R2R-CE / RxR-CE (VLN-CE, Habitat) | Instruction-following navigation (SR / SPL) | Planned |
| Real-robot trials (Unitree) | Sim-to-real transfer | Planned |
Note: the reported 0.782 is measured on our own held-out split, not the official VSI-Bench; official-benchmark evaluation is on the roadmap.
Usage
from unsloth import FastVisionModel
model, tokenizer = FastVisionModel.from_pretrained(
"unsloth/Qwen2.5-VL-7B-Instruct",
load_in_4bit=True,
max_seq_length=4096,
)
model.load_adapter("FINWHALE/Wukong")
FastVisionModel.for_inference(model)
A FastAPI serving wrapper and an evaluation script are available in the companion training pipeline (to be linked from this card).
Intended Use
- Spatial question answering and measurement estimation from egocentric images
- Research on spatial intelligence, world models, and embodied reasoning
- A foundation for downstream fine-tuning on specific robot embodiments (navigation adapters on the roadmap)
Limitations
- Metric-precision spatial estimation (exact distances, absolute cardinal directions) remains the weakest capability (0.55–0.61) — an open challenge shared by all current VLM/VLA systems.
- This release is a spatial-reasoning VLM stage: it does not output motor commands. Closed-loop navigation requires the upcoming waypoint/action adapters.
- Long-horizon imagination may drift; closed-loop correction with real observations is required in deployment.
- Not intended for safety-critical deployment without additional validation.
Citation
@misc{wukong2026,
title = {Wukong: An Open Spatial-Reasoning Vision-Language-Action Model},
author = {FINWHALE},
year = {2026},
url = {https://huggingface.co/FINWHALE/Wukong}
}
License
Apache 2.0. Please also comply with the licenses of the underlying base models and datasets.