zhudanburujiandan

Qwen2-VL-7B-DriveLM-LoRA-v3

README

License: apache-2.0

TL;DR

Table with columns: Metric (100 held-out val QAs, seed=42), Zero-shot, v1 (visual only, 154k train), v3 (visual + speed, 154k train), Δ
Metric (100 held-out val QAs, seed=42)	Zero-shot	v1 (visual only, 154k train)	v3 (visual + speed, 154k train)	Δ
Exact match	9	62	66	+4
Fuzzy substring	11	65	70	+5
Verbose (pred > 3× GT)	14	0	0	0

Per DriveLM level (25 QAs each):

Table with columns: Level, v1 exact, v3 exact, Δ
Level	v1 exact	v3 exact	Δ
Perception	16	16	0
Prediction	22	22	0
Planning	13	13	0
Behavior	11 (44%)	15 (60%)

All gains land on the behavior level — v3 matches v1 on perception / prediction / planning and adds +4 exact points where the label class is directly speed-derived ("driving fast / normal / slowly / not moving").

A companion v3-20k variant (trained on 20k QAs instead of 154k) trades total exact for even stronger behavior — see the v3-20k branch — 17/25 behavior (68%) at 64 total exact.

What changed vs v1

One thing only: at inference time, prepend "Ego speed: X.Y m/s.\n" to the user prompt (X.Y = ego vehicle's speed magnitude in m/s). Everything else is identical to v1 — same LoRA rank, same base, same DriveLM QA schema.

Where to get the speed value:

From nuScenes: differentiate ego_pose.translation between consecutive samples (script below).
From a running vehicle: read wheel-speed / CAN-bus / GPS-derived speed.
From estimation: any monocular / visual-odometry estimate is fine — models is tolerant of noise (v3 was trained on pose-diff speeds which have ~0.1-0.3 m/s noise).

Why it works: behavior labels in DriveLM are a discretization of ego speed. Numeric grounding lets the LLM directly threshold-classify instead of guessing from static visual cues. See W17 negative log for the ablation that ruled out visual-temporal approaches.

Intended use

Same as v1 (front-view dashcam → DriveLM-style Q&A), plus:

Robotics / research prototypes where you already have odometry (wheel encoders, IMU, GPS) and want a driving-focused VLM to condition on it.
Behavior classification as a substitute for hand-tuned speed thresholds.

Non-intended use

Same limits as v1 (single front camera, English only, DriveLM QA style, not a safety-critical policy).
Do not feed adversarial speed values expecting the model to hallucinate — the numeric prefix strongly biases behavior labels.

How to use

python
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
from qwen_vl_utils import process_vision_info
import torch

BASE = "Qwen/Qwen2-VL-7B-Instruct"
ADAPTER = "zhudanburujiandan/Qwen2-VL-7B-DriveLM-LoRA-v3"

model = Qwen2VLForConditionalGeneration.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="cuda", attn_implementation="sdpa",
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

processor = AutoProcessor.from_pretrained(BASE, min_pixels=100*28*28, max_pixels=512*28*28)

ego_speed_m_s = 5.5  # from odometry / pose diff
question = "Predict the behavior of the ego vehicle."

user_text = f"Ego speed: {ego_speed_m_s:.1f} m/s.\n{question}"
messages = [
    {"role": "system",
     "content": "You are a driving perception assistant. You see the front camera view of "
                "the ego vehicle. Answer the question concisely based only on what is visible. "
                "You will also be told the current ego speed in m/s at the start of each prompt."},
    {"role": "user", "content": [
        {"type": "image", "image": "path/to/front_cam.jpg"},
        {"type": "text",  "text": user_text},
    ]},
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, _ = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, padding=True, return_tensors="pt").to("cuda")
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=192, do_sample=False)
print(processor.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])

Expected outputs (behavior questions):

Ego speed 0.0 m/s → "The ego vehicle is going straight. The ego vehicle is not moving."
Ego speed 3.1 m/s → "The ego vehicle is going straight. The ego vehicle is driving slowly."
Ego speed 5.2 m/s → "The ego vehicle is going straight. The ego vehicle is driving with normal speed."
Ego speed 7.3 m/s → "The ego vehicle is going straight. The ego vehicle is driving fast."
Ego speed 10+ m/s → "The ego vehicle is going straight. The ego vehicle is driving very fast."

Other levels (perception / prediction / planning) still work; the speed prefix is harmless (and slightly helpful) there.

Deriving speed from nuScenes samples

If you're evaluating on nuScenes / DriveLM directly, differentiate ego pose:

python
from nuscenes.nuscenes import NuScenes
import numpy as np

nusc = NuScenes(version="v1.0-trainval", dataroot="path/to/nuscenes", verbose=False)

def sample_speed_m_s(sample_token: str) -> float:
    s = nusc.get('sample', sample_token)
    scene_samples = [x for x in nusc.sample if x['scene_token'] == s['scene_token']]
    scene_samples.sort(key=lambda x: x['timestamp'])
    idx = next(i for i, x in enumerate(scene_samples) if x['token'] == sample_token)
    lo = max(0, idx - 1); hi = min(len(scene_samples) - 1, idx + 1)
    def pose(x):
        sd = nusc.get('sample_data', x['data']['CAM_FRONT'])
        return np.array(nusc.get('ego_pose', sd['ego_pose_token'])['translation']), sd['timestamp']
    pa, ta = pose(scene_samples[lo]); pb, tb = pose(scene_samples[hi])
    dt = (tb - ta) / 1e6
    return float(np.linalg.norm(pb - pa) / dt) if dt > 0 else 0.0

Central difference (previous + next samples) gives ~0.5-1s smoothing that matches training.

Speed → label empirical distribution (why this works)

On DriveLM's 3,652 behavior QAs (train split), the "driving fast / normal / slowly / not moving" categories map cleanly onto pose-diff speed:

Table with columns: Label, Median speed, p10 – p90
Label	Median speed	p10 – p90
not moving	0.00 m/s	0.00 – 1.14
slow	3.11 m/s	0.68 – 4.85
normal	5.22 m/s	3.69 – 6.59
fast	7.29 m/s	5.77 – 8.81

The p10/p90 bands barely overlap, so thresholds around [1.5, 4.0, 6.0] m/s already recover the labels almost perfectly. What v3 learns is essentially that mapping (with a small residual for tie-breaking via visual scene).

Training details

Same as v1, except:

Prompts prepend "Ego speed: X.Y m/s.\n" (computed from nuScenes ego_pose diffs).
Same 154,363 train QAs, 1 epoch on 1× L40S 48 GB, 19h50min.
eval_loss plateau 0.175 (v1 was 0.19; small but real gain in loss too).
LoRA r=32, α=64, target = q/k/v/o + gate/up/down_proj. Trainable 80.7 M / 8.37 B (0.96%).

Compute delta vs v1: same ~20h. Speed prefix costs ~15 extra tokens per sample; no measurable slowdown.

v1 (154k, visual only) — 62 exact, behavior 44%. Repo.
W15 (3-cam panorama) — negative (52 exact). Per-camera resolution loss hurts perception.
W16 (4-frame temporal from DriveLM's own keyframes, non-uniform 0.5–13s gaps) — 53 exact, behavior unchanged. First hint that time signal alone isn't enough.
W17 (4-frame temporal from real nuScenes samples, uniform 0.5s gaps) — 54 exact, behavior dropped to 8/25 (32%). Ruled out "denser video → better behavior".
v3 (this) — solved via numeric grounding, not visual data.

Takeaway: for VLMs to classify a physical quantity that has a native numeric label (speed, distance, count), feed the number. Denser video, more cameras, and longer training all failed on this axis; a single float in the prompt lifted 44% → 60%.

Known limitations

Same as v1 (see v1 README) plus:

Requires an odometry input at inference. If you can't provide one, use v1 (this adapter still runs without the prefix — behavior degrades to roughly v1 numbers or worse).
Speed unit locked to m/s. Pass in the wrong unit (km/h, mph) and the model silently mis-classifies. There is no unit safeguard in the prompt.
Value range trained on ≈ 0–13 m/s (nuScenes urban driving). Higher speeds (highway 30+ m/s) are extrapolation; the model has never seen "driving very very fast" in training text.
Same DriveLM QA schema narrowness applies (short answers, English only).

Citation

Same base + dataset citations as v1 (see v1 README):

bibtex
@article{qwen2vl,
  title={{Qwen2-VL}: Enhancing Vision-Language Model's Perception of the World at Any Resolution},
  author={Wang, Peng and Bai, Shuai and others},
  journal={arXiv preprint arXiv:2409.12191},
  year={2024},
}

@inproceedings{drivelm,
  title={{DriveLM}: Driving with Graph Visual Question Answering},
  author={Sima, Chonghao and Renz, Katrin and others},
  booktitle={ECCV},
  year={2024},
}

License

Apache-2.0 (inherits from Qwen2-VL-7B-Instruct and DriveLM-nuScenes v1.1).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

zhudanburujiandan

Model Tree

Base

Qwen/Qwen2-VL-7B-Instruct

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

TL;DR

Table with columns: Metric (100 held-out val QAs, seed=42), Zero-shot, v1 (visual only, 154k train), v3 (visual + speed, 154k train), Δ
Metric (100 held-out val QAs, seed=42)	Zero-shot	v1 (visual only, 154k train)	v3 (visual + speed, 154k train)	Δ
Exact match	9	62	66	+4
Fuzzy substring	11	65	70	+5
Verbose (pred > 3× GT)	14	0	0	0

Per DriveLM level (25 QAs each):

Table with columns: Level, v1 exact, v3 exact, Δ
Level	v1 exact	v3 exact	Δ
Perception	16	16	0
Prediction	22	22	0
Planning	13	13	0
Behavior	11 (44%)	15 (60%)

A companion v3-20k variant (trained on 20k QAs instead of 154k) trades total exact for even stronger behavior — see the v3-20k branch — 17/25 behavior (68%) at 64 total exact.

What changed vs v1

Where to get the speed value:

From nuScenes: differentiate ego_pose.translation between consecutive samples (script below).
From a running vehicle: read wheel-speed / CAN-bus / GPS-derived speed.
From estimation: any monocular / visual-odometry estimate is fine — models is tolerant of noise (v3 was trained on pose-diff speeds which have ~0.1-0.3 m/s noise).

Intended use

Same as v1 (front-view dashcam → DriveLM-style Q&A), plus:

Robotics / research prototypes where you already have odometry (wheel encoders, IMU, GPS) and want a driving-focused VLM to condition on it.
Behavior classification as a substitute for hand-tuned speed thresholds.

Non-intended use

Same limits as v1 (single front camera, English only, DriveLM QA style, not a safety-critical policy).
Do not feed adversarial speed values expecting the model to hallucinate — the numeric prefix strongly biases behavior labels.

How to use

python
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
from qwen_vl_utils import process_vision_info
import torch

BASE = "Qwen/Qwen2-VL-7B-Instruct"
ADAPTER = "zhudanburujiandan/Qwen2-VL-7B-DriveLM-LoRA-v3"

model = Qwen2VLForConditionalGeneration.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="cuda", attn_implementation="sdpa",
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

processor = AutoProcessor.from_pretrained(BASE, min_pixels=100*28*28, max_pixels=512*28*28)

ego_speed_m_s = 5.5  # from odometry / pose diff
question = "Predict the behavior of the ego vehicle."

user_text = f"Ego speed: {ego_speed_m_s:.1f} m/s.\n{question}"
messages = [
    {"role": "system",
     "content": "You are a driving perception assistant. You see the front camera view of "
                "the ego vehicle. Answer the question concisely based only on what is visible. "
                "You will also be told the current ego speed in m/s at the start of each prompt."},
    {"role": "user", "content": [
        {"type": "image", "image": "path/to/front_cam.jpg"},
        {"type": "text",  "text": user_text},
    ]},
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, _ = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, padding=True, return_tensors="pt").to("cuda")
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=192, do_sample=False)
print(processor.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])

Expected outputs (behavior questions):

Ego speed 0.0 m/s → "The ego vehicle is going straight. The ego vehicle is not moving."
Ego speed 3.1 m/s → "The ego vehicle is going straight. The ego vehicle is driving slowly."
Ego speed 5.2 m/s → "The ego vehicle is going straight. The ego vehicle is driving with normal speed."
Ego speed 7.3 m/s → "The ego vehicle is going straight. The ego vehicle is driving fast."
Ego speed 10+ m/s → "The ego vehicle is going straight. The ego vehicle is driving very fast."

Other levels (perception / prediction / planning) still work; the speed prefix is harmless (and slightly helpful) there.

Deriving speed from nuScenes samples

If you're evaluating on nuScenes / DriveLM directly, differentiate ego pose:

python
from nuscenes.nuscenes import NuScenes
import numpy as np

nusc = NuScenes(version="v1.0-trainval", dataroot="path/to/nuscenes", verbose=False)

def sample_speed_m_s(sample_token: str) -> float:
    s = nusc.get('sample', sample_token)
    scene_samples = [x for x in nusc.sample if x['scene_token'] == s['scene_token']]
    scene_samples.sort(key=lambda x: x['timestamp'])
    idx = next(i for i, x in enumerate(scene_samples) if x['token'] == sample_token)
    lo = max(0, idx - 1); hi = min(len(scene_samples) - 1, idx + 1)
    def pose(x):
        sd = nusc.get('sample_data', x['data']['CAM_FRONT'])
        return np.array(nusc.get('ego_pose', sd['ego_pose_token'])['translation']), sd['timestamp']
    pa, ta = pose(scene_samples[lo]); pb, tb = pose(scene_samples[hi])
    dt = (tb - ta) / 1e6
    return float(np.linalg.norm(pb - pa) / dt) if dt > 0 else 0.0

Central difference (previous + next samples) gives ~0.5-1s smoothing that matches training.

Speed → label empirical distribution (why this works)

On DriveLM's 3,652 behavior QAs (train split), the "driving fast / normal / slowly / not moving" categories map cleanly onto pose-diff speed:

Table with columns: Label, Median speed, p10 – p90
Label	Median speed	p10 – p90
not moving	0.00 m/s	0.00 – 1.14
slow	3.11 m/s	0.68 – 4.85
normal	5.22 m/s	3.69 – 6.59
fast	7.29 m/s	5.77 – 8.81

Training details

Same as v1, except:

Prompts prepend "Ego speed: X.Y m/s.\n" (computed from nuScenes ego_pose diffs).
Same 154,363 train QAs, 1 epoch on 1× L40S 48 GB, 19h50min.
eval_loss plateau 0.175 (v1 was 0.19; small but real gain in loss too).
LoRA r=32, α=64, target = q/k/v/o + gate/up/down_proj. Trainable 80.7 M / 8.37 B (0.96%).

Compute delta vs v1: same ~20h. Speed prefix costs ~15 extra tokens per sample; no measurable slowdown.

v1 (154k, visual only) — 62 exact, behavior 44%. Repo.
W15 (3-cam panorama) — negative (52 exact). Per-camera resolution loss hurts perception.
W16 (4-frame temporal from DriveLM's own keyframes, non-uniform 0.5–13s gaps) — 53 exact, behavior unchanged. First hint that time signal alone isn't enough.
W17 (4-frame temporal from real nuScenes samples, uniform 0.5s gaps) — 54 exact, behavior dropped to 8/25 (32%). Ruled out "denser video → better behavior".
v3 (this) — solved via numeric grounding, not visual data.

Known limitations

Same as v1 (see v1 README) plus:

Requires an odometry input at inference. If you can't provide one, use v1 (this adapter still runs without the prefix — behavior degrades to roughly v1 numbers or worse).
Speed unit locked to m/s. Pass in the wrong unit (km/h, mph) and the model silently mis-classifies. There is no unit safeguard in the prompt.
Value range trained on ≈ 0–13 m/s (nuScenes urban driving). Higher speeds (highway 30+ m/s) are extrapolation; the model has never seen "driving very very fast" in training text.
Same DriveLM QA schema narrowness applies (short answers, English only).

Citation

Same base + dataset citations as v1 (see v1 README):

bibtex
@article{qwen2vl,
  title={{Qwen2-VL}: Enhancing Vision-Language Model's Perception of the World at Any Resolution},
  author={Wang, Peng and Bai, Shuai and others},
  journal={arXiv preprint arXiv:2409.12191},
  year={2024},
}

@inproceedings{drivelm,
  title={{DriveLM}: Driving with Graph Visual Question Answering},
  author={Sima, Chonghao and Renz, Katrin and others},
  booktitle={ECCV},
  year={2024},
}

License

Apache-2.0 (inherits from Qwen2-VL-7B-Instruct and DriveLM-nuScenes v1.1).

Qwen2-VL-7B-DriveLM-LoRA-v3

README

TL;DR

What changed vs v1

Intended use

Non-intended use

How to use

Deriving speed from nuScenes samples

Speed → label empirical distribution (why this works)

Training details

Related work

Known limitations

Citation

License

Explore FriendliAI today

README

TL;DR

What changed vs v1

Intended use

Non-intended use

How to use

Deriving speed from nuScenes samples

Speed → label empirical distribution (why this works)

Training details

Related work

Known limitations

Citation

License