yuyu199741

raya-decision-v1

README

License: apache-2.0

What is Raya?

Raya is not a chat model. It does not generate text. Instead, it performs single-forward-pass decision making: given a context (text/image/video), a question, and a set of candidate options, it outputs a probability distribution over the candidates in one forward pass — no autoregressive generation, no text output.

Key Design Principles (Jev Paradigm)

No autoregression: Zero generation steps. A single forward pass produces the decision. Latency in milliseconds.
Masked projection: Last-position logits are projected onto candidate label tokens only (local softmax over candidates, not the full vocabulary). Output is structurally verifiable — the model physically cannot produce anything outside the candidate set.
RLCD calibration: Post-training with strictly proper scoring rules (Brier/log score) + GRPO for confidence calibration. Confidence scores are meaningful and can be used for gating (e.g., escalate to System-2 when uncertain).
Dynamic candidate space: Candidates are passed as structured JSON with each request. The model is agnostic to the number/content of options — 3 or 10 options is just a matter of how many logit positions to extract.

What Makes Raya Different

Table with columns: Model, Modality, Calibration, Decision Format
Model	Modality	Calibration	Decision Format
Laya	Text only	No	Text generation
OpenJev	Text only	No	Text generation
jev-visual	Image only	No	Text generation
Raya	Text + Image + Video	Yes (RLCD)	Masked projection, no generation

Model Architecture

Base: Qwen3.5-2B (natively multimodal, 2026-02 release, Apache 2.0)
Architecture: Hybrid Gated DeltaNet + Gated Attention (3:1 ratio, 24 layers) + sparse MoE FFN
Context window: 262K tokens
Parameters: 2.2B (all parameters fine-tuned in Stage 2)

Training Pipeline

Table with columns: Stage, Method, Parameters, Data, Goal
Stage	Method	Parameters	Data	Goal
Step 0	Teacher distillation (MiniMax-M3)	—	Soft labels for ambiguous cases	Distribution estimation
Step 1	SFT-LoRA	17M LoRA params	race + koniq + gqa + nextqa (~250K)	Learn decision format
Step 2	Full fine-tuning	All 2.2B params	Same as Step 1	Deepen judgment ability

Performance

Held-out Evaluation (90 samples, never seen during training)

Table with columns: Modality, Accuracy, vs Zero-shot Baseline
Modality	Accuracy	vs Zero-shot Baseline
Text (RACE)	86.67%	+50.0pt
Image (KonIQ + GQA)	95.00%	+42.5pt
Video (NExT-QA)	90.00%	+25.0pt
Overall	91.11%	+41.1pt

Calibration (ECE)

Table with columns: Stage, ECE
Stage	ECE
Before RLCD (Full FT)	0.1221
After RLCD	0.0855

Target: ECE ≤ 0.08. High-confidence buckets (0.9-1.0) are nearly perfectly calibrated (gap = 0.2%).

Latency (measured on NVIDIA H20)

Table with columns: Modality, Latency, Bottleneck
Modality	Latency	Bottleneck
Text	43ms	✅ Production-ready
Image	61ms	✅ Production-ready
Video	1373ms	⚠️ 64% is CPU video decoding, not model forward

Usage

Installation

bash

pip install torch transformers pillow# For video inference:pip install av

Text Decision

python

import torchfrom transformers import AutoModelForImageTextToText, AutoTokenizer model = AutoModelForImageTextToText.from_pretrained(    "yuyu199741/raya-decision-v1", dtype=torch.bfloat16).cuda().eval()tokenizer = AutoTokenizer.from_pretrained("yuyu199741/raya-decision-v1") text = """User: 帮我把明天下午3点的会议改到4点。这个请求的意图是什么? A. 修改会议时间B. 创建新会议C. 取消会议D. 查询会议详情 Answer with exactly one label from the options above.""" inputs = tokenizer(text, return_tensors="pt").to("cuda")with torch.no_grad():    logits = model(**inputs).logits # Masked projection: extract logits at candidate label token positionslast_logits = logits[0, -1, :]label_token_ids = {    "A": tokenizer.encode(" A", add_special_tokens=False)[0],    "B": tokenizer.encode(" B", add_special_tokens=False)[0],    "C": tokenizer.encode(" C", add_special_tokens=False)[0],    "D": tokenizer.encode(" D", add_special_tokens=False)[0],} selected = torch.tensor([last_logits[tid].item() for tid in label_token_ids.values()])probs = torch.softmax(selected, dim=-1) for label, prob in zip(label_token_ids.keys(), probs):    print(f"{label}: {prob:.2%}")

Image Decision

python

from PIL import Imagefrom transformers import AutoProcessor processor = AutoProcessor.from_pretrained("Qwen/Qwen3.5-2B")  # Load from base repo image = Image.open("photo.jpg").convert("RGB")text = """<|vision_start|><|image_pad|><|vision_end|> Rate the overall technical quality of this image. A. very poorB. poorC. fairD. goodE. excellent Answer with exactly one label from the options above.""" enc = processor(text=[text], images=[image], return_tensors="pt").to("cuda")with torch.no_grad():    logits = model(**enc).logits# ... same masked projection logic

Video Decision

python

text = """<|vision_start|><|video_pad|><|vision_end|> What did the person do first in this video? A. opened the doorB. sat downC. picked up the cupD. left the room Answer with exactly one label from the options above.""" enc = processor(    text=[text], videos=["clip.mp4"], return_tensors="pt",    fps=None, num_frames=8, cap_pixels_per_frame=True,  # Fixed frame count, not fps-based).to("cuda")with torch.no_grad():    logits = model(**enc).logits# ... same masked projection logic

Critical Rules

Do NOT use model.generate(): This model does not generate text. It produces logits at the last position, and you extract the candidate label probabilities via masked projection.
Candidate labels are A/B/C/D...: The prompt must render options as "A. option text" format. The model only looks at the logits of these label tokens.
Video must use fixed frame count: Use num_frames=8, cap_pixels_per_frame=True. Do NOT use fps parameter (fps-based sampling explodes token count for long videos).
Processor loads from base repo: AutoProcessor.from_pretrained("Qwen/Qwen3.5-2B"), not from the model directory (save_pretrained doesn't save processor config).

Training Data

Table with columns: Dataset, Modality, Split, Size, License
Dataset	Modality	Split	Size	License
RACE	Text	train	40,000	research-only
KonIQ-10k	Image	train	7,058	research-only (KonIQ-10k EULA / YFCC100M)
GQA	Image	train_balanced	200,000	research-only (GQA / Visual Genome)

Note: All training data is research-only licensed. This model is for research use only.

Limitations

Not a general-purpose chat model: It only does decision-making from candidate options. It cannot answer open-ended questions or generate text.
Video frame sampling: Videos are sampled at 8 fixed frames (not fps-based). This is sufficient for action/sequence understanding but may miss frame-level details in long videos.
Research license: All training datasets are research-only. Commercial use requires re-training with commercially licensed data.
Calibration near target but not exact: ECE = 0.0855, slightly above the 0.08 target. High-confidence predictions are well-calibrated; low-confidence predictions have fewer samples and less statistical significance.
2B model ceiling: Video understanding is inherently limited by the 2B parameter budget. For more demanding video tasks, consider a larger base model.

Citation

If you use Raya in your research, please cite:

bibtex

@misc{raya2026,  title={Raya: Multimodal Fast Decision Model with Jev Paradigm},  author={yuyu199741},  year={2026},  howpublished={\url{https://huggingface.co/yuyu199741/raya-decision-v1}},}

Acknowledgments

Base model: Qwen3.5-2B by Alibaba Cloud
Jev paradigm: TypeSafe AI's Jev decision paradigm
Training framework: PyTorch + Hugging Face Transformers + PEFT + Accelerate

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

yuyu199741

Model Tree

Base

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

What is Raya?

Key Design Principles (Jev Paradigm)

No autoregression: Zero generation steps. A single forward pass produces the decision. Latency in milliseconds.
Masked projection: Last-position logits are projected onto candidate label tokens only (local softmax over candidates, not the full vocabulary). Output is structurally verifiable — the model physically cannot produce anything outside the candidate set.
RLCD calibration: Post-training with strictly proper scoring rules (Brier/log score) + GRPO for confidence calibration. Confidence scores are meaningful and can be used for gating (e.g., escalate to System-2 when uncertain).
Dynamic candidate space: Candidates are passed as structured JSON with each request. The model is agnostic to the number/content of options — 3 or 10 options is just a matter of how many logit positions to extract.

What Makes Raya Different

Table with columns: Model, Modality, Calibration, Decision Format
Model	Modality	Calibration	Decision Format
Laya	Text only	No	Text generation
OpenJev	Text only	No	Text generation
jev-visual	Image only	No	Text generation
Raya	Text + Image + Video	Yes (RLCD)	Masked projection, no generation

Model Architecture

Base: Qwen3.5-2B (natively multimodal, 2026-02 release, Apache 2.0)
Architecture: Hybrid Gated DeltaNet + Gated Attention (3:1 ratio, 24 layers) + sparse MoE FFN
Context window: 262K tokens
Parameters: 2.2B (all parameters fine-tuned in Stage 2)

Training Pipeline

Table with columns: Stage, Method, Parameters, Data, Goal
Stage	Method	Parameters	Data	Goal
Step 0	Teacher distillation (MiniMax-M3)	—	Soft labels for ambiguous cases	Distribution estimation
Step 1	SFT-LoRA	17M LoRA params	race + koniq + gqa + nextqa (~250K)	Learn decision format
Step 2	Full fine-tuning	All 2.2B params	Same as Step 1	Deepen judgment ability

Performance

Held-out Evaluation (90 samples, never seen during training)

Table with columns: Modality, Accuracy, vs Zero-shot Baseline
Modality	Accuracy	vs Zero-shot Baseline
Text (RACE)	86.67%	+50.0pt
Image (KonIQ + GQA)	95.00%	+42.5pt
Video (NExT-QA)	90.00%	+25.0pt
Overall	91.11%	+41.1pt

Calibration (ECE)

Table with columns: Stage, ECE
Stage	ECE
Before RLCD (Full FT)	0.1221
After RLCD	0.0855

Target: ECE ≤ 0.08. High-confidence buckets (0.9-1.0) are nearly perfectly calibrated (gap = 0.2%).

Latency (measured on NVIDIA H20)

Table with columns: Modality, Latency, Bottleneck
Modality	Latency	Bottleneck
Text	43ms	✅ Production-ready
Image	61ms	✅ Production-ready
Video	1373ms	⚠️ 64% is CPU video decoding, not model forward

Usage

Installation

bash

pip install torch transformers pillow# For video inference:pip install av

Text Decision

python

import torchfrom transformers import AutoModelForImageTextToText, AutoTokenizer model = AutoModelForImageTextToText.from_pretrained(    "yuyu199741/raya-decision-v1", dtype=torch.bfloat16).cuda().eval()tokenizer = AutoTokenizer.from_pretrained("yuyu199741/raya-decision-v1") text = """User: 帮我把明天下午3点的会议改到4点。这个请求的意图是什么? A. 修改会议时间B. 创建新会议C. 取消会议D. 查询会议详情 Answer with exactly one label from the options above.""" inputs = tokenizer(text, return_tensors="pt").to("cuda")with torch.no_grad():    logits = model(**inputs).logits # Masked projection: extract logits at candidate label token positionslast_logits = logits[0, -1, :]label_token_ids = {    "A": tokenizer.encode(" A", add_special_tokens=False)[0],    "B": tokenizer.encode(" B", add_special_tokens=False)[0],    "C": tokenizer.encode(" C", add_special_tokens=False)[0],    "D": tokenizer.encode(" D", add_special_tokens=False)[0],} selected = torch.tensor([last_logits[tid].item() for tid in label_token_ids.values()])probs = torch.softmax(selected, dim=-1) for label, prob in zip(label_token_ids.keys(), probs):    print(f"{label}: {prob:.2%}")

Image Decision

python

from PIL import Imagefrom transformers import AutoProcessor processor = AutoProcessor.from_pretrained("Qwen/Qwen3.5-2B")  # Load from base repo image = Image.open("photo.jpg").convert("RGB")text = """<|vision_start|><|image_pad|><|vision_end|> Rate the overall technical quality of this image. A. very poorB. poorC. fairD. goodE. excellent Answer with exactly one label from the options above.""" enc = processor(text=[text], images=[image], return_tensors="pt").to("cuda")with torch.no_grad():    logits = model(**enc).logits# ... same masked projection logic

Video Decision

python

text = """<|vision_start|><|video_pad|><|vision_end|> What did the person do first in this video? A. opened the doorB. sat downC. picked up the cupD. left the room Answer with exactly one label from the options above.""" enc = processor(    text=[text], videos=["clip.mp4"], return_tensors="pt",    fps=None, num_frames=8, cap_pixels_per_frame=True,  # Fixed frame count, not fps-based).to("cuda")with torch.no_grad():    logits = model(**enc).logits# ... same masked projection logic

Critical Rules

Do NOT use model.generate(): This model does not generate text. It produces logits at the last position, and you extract the candidate label probabilities via masked projection.
Candidate labels are A/B/C/D...: The prompt must render options as "A. option text" format. The model only looks at the logits of these label tokens.
Video must use fixed frame count: Use num_frames=8, cap_pixels_per_frame=True. Do NOT use fps parameter (fps-based sampling explodes token count for long videos).
Processor loads from base repo: AutoProcessor.from_pretrained("Qwen/Qwen3.5-2B"), not from the model directory (save_pretrained doesn't save processor config).

Training Data

Table with columns: Dataset, Modality, Split, Size, License
Dataset	Modality	Split	Size	License
RACE	Text	train	40,000	research-only
KonIQ-10k	Image	train	7,058	research-only (KonIQ-10k EULA / YFCC100M)
GQA	Image	train_balanced	200,000	research-only (GQA / Visual Genome)

Note: All training data is research-only licensed. This model is for research use only.

Limitations

Not a general-purpose chat model: It only does decision-making from candidate options. It cannot answer open-ended questions or generate text.
Video frame sampling: Videos are sampled at 8 fixed frames (not fps-based). This is sufficient for action/sequence understanding but may miss frame-level details in long videos.
Research license: All training datasets are research-only. Commercial use requires re-training with commercially licensed data.
Calibration near target but not exact: ECE = 0.0855, slightly above the 0.08 target. High-confidence predictions are well-calibrated; low-confidence predictions have fewer samples and less statistical significance.
2B model ceiling: Video understanding is inherently limited by the 2B parameter budget. For more demanding video tasks, consider a larger base model.

Citation

If you use Raya in your research, please cite:

bibtex

@misc{raya2026,  title={Raya: Multimodal Fast Decision Model with Jev Paradigm},  author={yuyu199741},  year={2026},  howpublished={\url{https://huggingface.co/yuyu199741/raya-decision-v1}},}

Acknowledgments

Base model: Qwen3.5-2B by Alibaba Cloud
Jev paradigm: TypeSafe AI's Jev decision paradigm
Training framework: PyTorch + Hugging Face Transformers + PEFT + Accelerate