How it is built
The training data is built around a simulated iPhone that exposes Orvena's real tool catalog and MCP servers whose
tool lists were captured from Notion, Linear, GitHub, Todoist, Airtable, Atlassian, Gmail, Google Drive, Google
Calendar, Granola, Home Assistant and Zapier. Each scenario is a request together with the phone's data (calendar, contacts, reminders,
places, weather, health) and checks on the calls the assistant should make and what its answer must contain. Only
answers that passed the checks were kept. Of the 3,397 scenarios,
Qwen3.6-35B-A3B answered 1,126 with thinking off and 724 with thinking
on, and the base model itself answered 1,547. That gives 11,161 rounds, one per model call, of which 574 were held out for
validation.
Two smaller sets keep the model from overreaching: requests no tool can do and plain questions asked with
connections installed, so it learns to answer or say what it can't do; and 600 instruction-following examples
(IFEval-style prompts from argilla/ifeval-like-data,
answered by the base model and kept only when IFEval's checker passed), so it still follows formatting instructions
like the base model does.

Orvena 4B keeps the architecture of its base model, Qwen3.5-4B: 24 Gated
DeltaNet layers, 8 full-attention layers and the vision encoder. A LoRA adapter (rank 16, alpha 320) on all 248 linear
layers of the language model was trained for one epoch and merged into these weights.
All of it was trained with thinking off, and the chat template in this repository answers without thinking unless
you pass enable_thinking=True. The MTP head was retrained on the fine-tuned model's own outputs and accepts 89.4%
of drafted tokens on held-out rounds (the base model's head: 89.0%).
Results
Phone tasks
The Orvena bench has 547 held-out scenarios in 30 languages, none of which were used for training. They run through
Orvena's own agent loop with the 4-bit model, on simulated phone data and MCP servers with the real services' tool
lists. A scenario passes only when the right tools are called with the right arguments and the answer contains what
it should, in the user's language. Orvena 4B and the base model run with thinking off. Apple's on-device model runs through the app's Apple
Intelligence backend in the same loop; it supports 13 of the 30 languages, and the app only sends it
requests in those. The scenarios, the tool catalog and every model's results are published as
Orvena/orvena-bench.

Table with columns: Orvena 4B, Base model, Apple on-device | Orvena 4B | Base model | Apple on-device |
|---|
| All 547 scenarios | 440 (80.4%) | 345 (63.1%) | 104 (19.0%) |
| In the 13 languages Apple's model supports (260) | 220 (84.6%) | 175 (67.3%) | 77 (29.6%) |
| Everyday requests with connections installed (100) | 83 | 60 | 23 |
The 13 languages are Danish, German, English, Spanish, French, Italian, Japanese, Korean, Dutch,
Portuguese, Swedish, Turkish and Vietnamese.

Table with columns: Category, Scenarios, Orvena 4B, Base model, Apple on-device| Category | Scenarios | Orvena 4B | Base model | Apple on-device |
|---|
| Multi-step tasks | 70 | 43 | 17 | 1 |
| MCP connections | 62 | 51 | 45 | 8 |
| Places | 38 | 35 | 26 | 12 |
| Calendar | 31 | 20 | 13 | 4 |
| Web | 31 | 29 | 26 | 15 |
| Alarms and timers | 25 | 20 | 17 | 5 |
| No tool needed | 23 | 16 | 19 | 6 |
| Weather | 22 | 18 | 18 | 7 |
| Reminders | 21 | 18 | 14 | 0 |
| Recovering from tool errors | 19 | 18 | 12 | 3 |
| Travel times | 19 | 19 | 17 | 1 |
| Follow-up questions | 18 | 13 | 9 | 0 |
| Scheduled tasks | 16 | 12 | 9 | 0 |
| Things it can't do | 13 | 9 | 10 | 6 |
| Music | 10 | 10 | 8 | 3 |
| Health | 7 | 7 | 6 | 3 |
| Photos | 7 | 6 | 6 | 3 |
| Contacts | 6 | 5 | 6 | 1 |
| Files | 5 | 4 | 3 | 0 |
| Device | 4 | 4 | 4 | 3 |
Multi-step tasks chain several tools in one request. "I've got that job interview tomorrow, set me an alarm for when
I need to leave, I'm going on the tube" needs the calendar, the current location, the transit time and an alarm;
"my notes from the flat viewing have the plumber's number in them somewhere, remind me tomorrow at 10 to call him"
needs a file search before the reminder. That is where the fine-tune changed the most: 43 of 70 against 17.

The MCP scenarios by connection, Orvena 4B on two seeds (labels give seed 7 / seed 8). Over all 62, Orvena 4B passes
51 and 49, the base model 45 and Apple's on-device model 8.

Public benchmarks
Public tool-use benchmarks bring their own tools, formats and rules for when to call, ask or decline, and none of them
were in Orvena 4B's training data. Where a call is clearly needed, Orvena 4B matches its base model, and when it does call,
more of its nested call sequences are fully right. It more often calls a tool where it should ask for a missing detail
or say it can't, and that is the main goal for the next version. Section 8 of the
technical report has the full results for BFCL v4, NESTFUL, ToolSandbox, When2Call,
MCPMark, IFEval, MGSM and MMLU.
On an iPhone
iPhone 17 Pro Max with the 4-bit MLX build in Orvena's agent loop: 14 tasks (tool chains, requests missing a
detail, and requests it can't do), two rounds each, with the phone cooled between runs.
Table with columns: Orvena 4B, Base model, thinking off, Base model, thinking auto | Orvena 4B | Base model, thinking off | Base model, thinking auto |
|---|
| Median time per task | 15.1 s | 16.3 s | 37.0 s |
| Tokens generated per turn | 169 | 213 | 757 |
Thinking auto was Orvena's default before Orvena 4B. Decoding runs at about 21 tokens per second on this phone, and
23 tokens per second with the MTP head as a drafter (83% of drafted tokens accepted).


Using it
vLLM
vllm serve Orvena/Orvena-4B --max-model-len 32768 --language-model-only \ --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder
With the MTP head for speculative decoding, add
--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'. Drop --language-model-only to
send images.
Tools go in the usual OpenAI tools field. To turn thinking on, send "chat_template_kwargs": {"enable_thinking": true};
the fine-tune was trained without it. generation_config.json sets the sampling Orvena uses, temperature 0.7 and top-p
0.8; runtimes that don't read it should be given these values.
Other runtimes
The weights keep the Qwen3.5 architecture and layout, so runtimes that support it, including Transformers and SGLang,
run Orvena 4B as they are. For Apple silicon, use Orvena/Orvena-4B-MLX-4bit.
Notes
- Orvena 4B was trained with Orvena's system prompt and tool catalog, but any OpenAI-style tool list works.
License
Apache-2.0, like Qwen3.5-4B. Built with Qwen. NOTICE lists what was changed.