GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
josefchen
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
josefchen
Model Tree
autotrust/JEV-27B-VL
Input Modalities
Output Modalities
Supported Functionality
The client sends the task as free text and samples ten primary-camera frames uniformly from the start to the end of the attempt. It can append the last frame from each additional camera. It reads the scores of false and true at one answer position, then applies the released calibration to return P(success). This takes one forward pass without a generated explanation.
Both attempts were done on an SO-101 arm, on a task that was out of training. RoboJev is asked every half a second, each time with ten frames distributed throughout the video so far. The overlay displays its answer as it updates.
Compared against 4 frontier language models and 2 decision APIs, on the same episodes:
Median serial response time is 0.26 seconds on one H100. The recorded frontier API medians are 5–7 seconds per call. Local model time and API wall-clock time include different overheads.
Our catalogue constructed from the Hub contains 68,555 episodes in 7,641 sampled public LeRobot datasets, approximately 501 hours of video. Grading all of these via RoboJev-27B would take an estimated 4.9 GPU-hours, or 13.16asopposedto637 using GPT-6 Astra. We did not regrade the catalogue for this release yet.
The release contains 20,628 predictions across 29 model/protocol entries on 927 human-labelled episodes. The 65 ambiguous community clips remain available but are excluded from metrics, leaving 862 scored episodes. Entries include earlier releases and different protocols, not 29 independent architectures.
The released RoboJev and the ten-frame JEV baseline use ten primary-camera frames plus available final auxiliary-camera views. Their image hashes and task text match across all 727 SO-101 and unseen-domain episodes. The frontier API comparisons use four primary-camera frames and verbal probability answers. Other entries identify their own protocol, including whole-video input. The frontier API comparisons do not isolate training effects at equal input size; the matched-input JEV comparison does hold the visual evidence fixed. Training-label teachers also overlap the API baseline panel; see Training below.
The community evaluation set contains 200 episodes from public LeRobot evaluation datasets (SO-100, SO-101, Koch and other arms). Each episode was labeled by one person. 135 out of these have a clear outcome and were scored. All 106 owners represented by these 200 datasets were excluded from community fitting. Labels come from one annotator, without an inter-annotator agreement estimate.
| Model | AUROC | Accuracy |
|---|---|---|
| RoboJev-27B | 0.984 | 0.933 |
| RoboJev-27B, first release (four frames) | 0.961 | 0.904 |
| JEV-27B-VL | 0.947 | 0.852 |
| GPT-6 Astra (four frames) | 0.991 | 0.926 |
| GPT-6.1 Sol (four frames) | 0.990 | 0.926 |
| Claude Opus 5.5 (four frames) | 0.988 | 0.941 |
The first-release and community JEV rows above are retained historical aggregates. Their individual predictions are not available in the current release snapshot, so they cannot be independently recomputed from that snapshot. The paper marks this distinction.
The SO-101 test contains 268 human-labelled runs: 111 successes and 157 failures across eight tasks. Two entire tasks account for 80 runs; the remaining 188 runs are held out by recording session. Other ArmnetBench sessions contribute training data.
| Model | AUROC | Balanced accuracy | ECE | AUROC, held-out tasks |
|---|---|---|---|---|
| RoboJev-27B | 0.993 | 0.952 | 0.032 | 0.986 |
| RoboJev-27B, first release (four frames) | 0.987 | 0.924 | 0.052 | 0.973 |
| JEV-27B-VL | 0.979 | 0.881 | 0.114 | 0.978 |
| TypeSafe Jev 1.14 | 0.975 | 0.829 | 0.177 | 0.958 |
| Perplexity Decider v1.2 | 0.954 | 0.811 | 0.116 | 0.947 |
| Inception Mercury Decide | 0.938 | 0.816 | 0.109 | 0.949 |
| Liquid d1-ultra | 0.918 | 0.814 | 0.062 | 0.878 |
| Robometer-4B (whole video) | 0.777 | 0.677 | 0.251 | 0.812 |
RBM-1M OOD contains 459 runs (263 successes and 196 failures) from eight held-out lab/robot domains, including Franka, Koch, Trossen, xArm and SO-101 arms. These are eight domain labels, not eight independently verified institutions. The GPT-6.1 Sol entry has 458 recorded answers; its missing answer is excluded.
| Model | AUROC | Balanced accuracy | ECE |
|---|---|---|---|
| JEV-27B-VL | 0.900 | 0.794 | 0.152 |
| RoboJev-27B | 0.890 | 0.782 | 0.148 |
| Perplexity Decider v1.2 | 0.885 | 0.761 | 0.168 |
| RoboJev-27B, first release (four frames) | 0.872 | 0.761 | 0.173 |
| Liquid d1-ultra | 0.859 | 0.735 | 0.143 |
| Inception Mercury Decide | 0.859 | 0.756 | 0.205 |
| TypeSafe Jev 1.14 | 0.841 | 0.721 | 0.197 |
| Robometer-4B (whole video) | 0.818 | 0.598 | 0.414 |
AUROC is the probability that a randomly selected success receives a higher score than a randomly selected failure. The default threshold for both accuracy and balanced accuracy is 0.5. Balanced accuracy is the mean of the recall on successes and the recall on failures. ECE is expected calibration error, with 10 bins.
In comparison with the previous version on the same episodes, in a paired bootstrap with 95% intervals on the difference in AUROC:
| Test | This release | First release | Difference [95% interval] |
|---|---|---|---|
| Community evaluations, whole attempt | 0.984 | 0.961 | +0.023 [+0.003, +0.049] |
| SO-101, whole attempt | 0.993 | 0.987 | +0.006 [+0.001, +0.014] |
| Unseen labs, whole attempt | 0.890 | 0.872 | +0.018 [+0.002, +0.035] |
| SO-101, last frames only | 0.966 | 0.938 | +0.028 [+0.009, +0.051] |
| Unseen labs, last frames only | 0.776 | 0.779 | unresolved: -0.003 [-0.021, +0.015] |
With identical inputs, adaptation improves SO-101 balanced accuracy over JEV-27B-VL by 7.08 percentage points (label-stratified paired-bootstrap 95% interval: +3.04 to +11.45). On unseen domains, the AUROC difference is −0.0092 (−0.0218 to +0.0026): the evidence does not establish an improvement over the base model. For 28 unseen-lab episodes, the four best models in the initial RoboJev release confidently disagree with the label. For 8 of them that we checked, the video cuts off before the outcome. The disputed examples are in benchmark/bench_ood_v1_disputed.jsonl.
Versus GPT-6 Astra, the unseen-lab AUROC difference is +0.028 (95%CI +0.005 to +0.051). The SO-101 balanced-accuracy difference is +0.019 (−0.016 to +0.052), which does not establish a difference. On the 80 runs from fully held-out tasks, RoboJev reaches 92.5% balanced accuracy; Astra and Sol each reach 96.25%. The complete release includes all 29 recorded model/protocol entries and their available predictions. Full comparison
The cost of inference at 0.192per1,000answersononeH100at2.69 an hour, one answer at a time, does not include download times, decoding or waiting time. The shared demo Space includes GPU allocation time which is why it's slower than a single H100.
Serve the model using vLLM and then call it from the small client in this repo.
bash
vllm serve josefchen/RoboJev-27B --served-model-name robojev --max-model-len 32768 \ --limit-mm-per-prompt '{"image": 16}' --logprobs-mode processed_logprobs --trust-request-chat-templatepython
from robojev import RoboJev judge = RoboJev("http://localhost:8000")judge.done("Put the ring on the peg", "attempt.mp4") # cuts ten frames from the videojudge.done("Put the ring on the peg", frames) # or pass ten frames, start to endjudge.done("Put the ring on the peg", "attempt.mp4", views={"wrist": "wrist_last.jpg"}) # with other cameras' last framesThe done method returns P(completed). robojev.py requires requests, pillow, and ffmpeg (for video files). The model requires 1x 80 GB GPU (bf16).
If you want to draw the probability on a video or live camera input:
bash
python robojev_overlay.py "Put the ring on the peg" attempt.mp4 out.mp4python robojev_overlay.py "Put the ring on the peg" --camera 0 # requires opencv-pythonThe lerobot_plugin directory exposes a reward model for recorded episodes and a processor for thresholded terminal rewards in LeRobot. Downstream policy-learning improvements have not been measured. See its README for integration details.
bash
hf download josefchen/RoboJev-27B --include "lerobot_plugin/*" --local-dir robojevpip install ./robojev/lerobot_pluginThe prompt ends at [decision]:. The client reads the log-probabilities of false and true at that position, adds the inherited biases from decision.json, divides by the validation-fitted temperature and applies a two-way softmax. decision.json also contains the number of frames the model reads.
With the server up and robojev_grade.py, robojev.py and decision.json all in a single directory:
bash
python robojev_grade.py owner/eval_dataset --server http://localhost:8000 --out results.csvFor each episode, the command prints the episode index, task, P(done), and verdict (Done or Not yet). The summary includes the number graded, the success rate at the threshold (default = 0.5), and how many episodes are worth a look (with P(done) between 0.2 and 0.8). Results can also be saved as CSV with the --out option.
It will pick a 3rd person camera and parse ten frames/episode via HTTP byte ranges. Use --camera to specify a camera, --episodes to specify episodes, and --header for additional server headers. No token is required for public datasets, but set HF_TOKEN when using private ones. LeRobot v2.0 and v2.1 require requests, pillow and ffmpeg, and v3.0 also requires pyarrow to read its Parquet episode metadata.
robojev.py: clientrobojev_overlay.py: adds a probability overlay on a video/camera feedrobojev_grade.py: grades episodes from a LeRobot dataset on the Hublerobot_plugin/: LeRobot reward model & reward stepbenchmark/: test sets + disputed episodes + scores for all modelsexamples/: the two videos abovedecision.json: config file, prompt format, token ids, biases, temperature, frame countWeights occupy 54.7 GB in bf16 and are intended for one 80 GB GPU. Initialization is autotrust/JEV-27B-VL: its decision adapter is merged before the robot-specific LoRA. The pinned checkpoint configuration declares Qwen3_5ForConditionalGeneration and model_type=qwen3_5. Earlier release prose used a different foundation-model name; the paper does not claim that unresolved lineage as verified.
The current release fits 52,598 episode records, with the following post-split counts. This is not an independently deduplicated count of physical attempts. The earlier 33,000-example description refers to the first release.
| Source family | Fit episode records |
|---|---|
| Selected RBM-1M processed sources | 28,170 |
| ArmnetBench | 1,388 |
| Teacher-labelled LeRobot community records | 10,216 |
| Community demonstrations | 3,718 |
| ViFailback | 4,962 |
| DROID successful conversion | 4,144 |
| Total | 52,598 |
The executed annotation pipeline requires GPT-6.1 Sol and GPT-6 Astra, each judging four primary-camera frames. Both must agree on the binary outcome and each must assign a probability at most 0.1 or at least 0.9. If a Claude Opus 5.5 answer is available, it must agree on the binary side and can veto the candidate; it need not independently meet that confidence margin. The soft target averages all available teacher probabilities. The training loader then applies the final 0.1 confidence margin to that mean.
All 11,387 saved unique target values match the executed consensus artifact. The loader reports 11,415 accepted community records before exclusions, leaving an unresolved 28-record discrepancy. Its 282 held-out Koch records map to 272 unique validation IDs. The paper reports these limits and does not equate loader records with unique physical attempts. After validation and robot exclusions, 10,216 community records enter fitting. All 106 owners in the human-labelled community evaluation are excluded from fitting.
The required annotation teachers overlap the frontier API benchmark panel. The benchmark labels are held out, but these comparisons are not independent of teacher-assisted training. Earlier prose describing Sol/Opus consensus on ten frames was incorrect; this card follows the executed run, as does the paper.
The language component uses rank-32 LoRA, alpha 64 and dropout 0.05, with 233,455,616 trainable parameters. The vision encoder and visual merger stay frozen. Training uses one epoch at learning rate 5e-5, AdamW with zero weight decay, 5% warmup and cosine decay, gradient clipping at 1.0, and seed 7. A per-GPU batch of 1 with accumulation 2 gives an optimizer batch of 14 on the documented seven-GPU run. The source records report approximately 5.5 hours on seven H100s; a shorter resumed-process log is not the total duration.
Original-record cross-entropy uses inverse-class-frequency weights within each source, clipped to [0.25, 4]; a source with only one outcome keeps weight 1. Synthetic negatives receive weight 0.5. These weights apply to cross-entropy, not the KL term. Repetition and augmentation produce 102,172 training examples in the epoch. Eligible episodes use ten-frame evidence with probability 0.6; other branches retain four-frame and final-view inputs. Resizing, JPEG recompression and colour jitter augment the images. Truncated positives and mismatched instructions provide heuristic negatives, which can be noisy.
On selected research sources, DROID and ViFailback, an untempered two-token KL penalty KL(base || adapted) with weight 1 anchors the model to JEV's decision. ArmnetBench and community examples are unanchored. This differs from the four-frame evidence used to create community teacher labels.
The run-specific selection procedure chooses the best held-out community AUROC among nonzero LoRA scales whose AUROC is no more than 0.005 below the base on each protected validation group. Those groups include AutoEval, FINO-Net, RoboArena's 0825 release and the held-out community Koch group. The run-specific validation procedure selects scale 1.0. Benchmarks were inspected during development, so this is not a claim of a fully blind test history. Temperature is fitted on held-out validation predictions; inherited decision-head biases are retained. The released decision.json uses temperature 0.9, tokens false (3721) and true (1802), and biases +0.00011960109 and −0.00011981795 respectively.
Use RoboJev for offline episode review, dataset grading and research into task-conditioned reward signals. The probabilities are model estimates, not measured physical success rates. The model is not a safety controller.
The released weights and inference tools declare Apache-2.0. Training data retain their source-specific terms. The release records Apache-2.0 for ArmnetBench; MIT for RoboArena, LIBERO, Meta-World and ViFailback; and CC BY 4.0 for RoboReward, MolmoAct and DROID. Open X-Embodiment has source-specific licences. The copies of SOAR, AutoEval and FINO-Net accessed through RBM-1M did not state an explicit dataset licence in the audit. Community source availability has changed, and the complete historical training licence chain has not been independently reconstructed. The paper's source ledger records those gaps. It does not redistribute the training corpus.
bibtex
@misc{chen2026robojev, title = {RoboJev: An Open Vision-Language Judge for Robot Success Detection}, author = {Josef Liyanjun Chen}, year = {2026}, month = oct, eprint = {2610.33718}, archivePrefix = {arXiv}, primaryClass = {cs.RO}, url = {https://arxiv.org/abs/2610.33718}}The robot clips used in the launch video were taken from these public LeRobot datasets hosted by HuggingFace. Thanks for all your help sharing your evaluation data! If you see that one of these datasets is yours, and would like to have its clip removed, please open a discussion on this repo.
vllm serve josefchen/RoboJev-27B --served-model-name robojev --max-model-len 32768 \ --limit-mm-per-prompt '{"image": 16}' --logprobs-mode processed_logprobs --trust-request-chat-templatefrom robojev import RoboJev judge = RoboJev("http://localhost:8000")judge.done("Put the ring on the peg", "attempt.mp4") # cuts ten frames from the videojudge.done("Put the ring on the peg", frames) # or pass ten frames, start to endjudge.done("Put the ring on the peg", "attempt.mp4", views={"wrist": "wrist_last.jpg"}) # with other cameras' last framespython robojev_overlay.py "Put the ring on the peg" attempt.mp4 out.mp4python robojev_overlay.py "Put the ring on the peg" --camera 0 # requires opencv-pythonhf download josefchen/RoboJev-27B --include "lerobot_plugin/*" --local-dir robojevpip install ./robojev/lerobot_pluginpython robojev_grade.py owner/eval_dataset --server http://localhost:8000 --out results.csv@misc{chen2026robojev, title = {RoboJev: An Open Vision-Language Judge for Robot Success Detection}, author = {Josef Liyanjun Chen}, year = {2026}, month = oct, eprint = {2610.33718}, archivePrefix = {arXiv}, primaryClass = {cs.RO}, url = {https://arxiv.org/abs/2610.33718}}