beyoru

Ops-I

README

License: apache-2.0

Results — AutomationBench 600

Table with columns: Model, Pass rate, Passed / 600, Partial credit, Wall clock
Model	Pass rate	Passed / 600	Partial credit	Wall clock
beyoru/Ops-I	10.17%	61	45.54%	1.96 h
gemma-4-31B-it (thinking on)	10.17%	61	41.64%	2.15 h
Qwen3.6-27B (base)	9.67%	58	46.00%	1.95 h

For scale, the reference models in the upstream README (run at their highest reasoning effort) land at Claude Opus 4.8 30.33%, GPT-5.6 Sol 29.17%, GLM 5.2 20.33%, Gemini 3.5 Flash 14.83%.

Per domain

Tasks passed out of 100, partial credit in parentheses.

Table with columns: Model, Finance, HR, Marketing, Operations, Sales, Support
Model	Finance	HR	Marketing	Operations	Sales	Support
beyoru/Ops-I	20 (57.0)	3 (10.1)	5 (51.4)	18 (60.5)	10 (38.2)	5 (55.9)
Qwen3.6-27B base	21 (57.7)	3 (12.4)	3 (45.1)	17 (60.9)	8 (41.4)

Finance and operations carry the score. HR is the floor for every model tested (1-3 / 100).

Known weakness: it acts too much

The dominant failure mode is not inability to act — it is failure to refrain.

Table with columns: Assertion type, Ops-I
Assertion type	Ops-I
Positive ("do send / do create")	3028 / 5636 = 53.7%
Negative ("do not send / do not create")	65 / 585 = 11.1%

Some negative assertion types never pass, aggregated across all four models tested: gmail_message_not_sent 0/203, slack_message_not_in_channel 0/166, google_sheets_row_not_exists 0/161.

This is uniform across two model families and four checkpoints, so treat it as a property of the current open-weight generation rather than of Ops-I specifically. If you deploy this in a workflow that can send email, post to chat, or write to a sheet, gate those actions behind confirmation.

Quickstart

Serve with SGLang

bash
python3 -m sglang.launch_server --model-path beyoru/Ops-I --served-model-name Ops-I \
  --host 0.0.0.0 --port 30014 --tp 4 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --mem-fraction-static 0.85

Serve with vLLM

bash
vllm serve beyoru/Ops-I --served-model-name Ops-I \
  -tp 4 --max-model-len 16384 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3

Call it

python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30014/v1", api_key="EMPTY")

tools = [{
    "type": "function",
    "function": {
        "name": "crm_update_contact",
        "description": "Update a contact's details in the CRM.",
        "parameters": {
            "type": "object",
            "properties": {"contact_id": {"type": "string"}, "phone": {"type": "string"}},
            "required": ["contact_id", "phone"],
        },
    },
}]

r = client.chat.completions.create(
    model="Ops-I",
    messages=[{"role": "user", "content": "Update Le Van A's phone to 0912345678."}],
    tools=tools,
    tool_choice="auto",
)
print(r.choices[0].message.tool_calls)

Training

Evaluation notes

All numbers above come from a single run per model (k=1) on the full public 600-task set, temperature left at server default. No task was excluded.
AutomationBench has no separate train split — the 600 public tasks are both the dataset and the scoring set. Ops-I was not trained on them; if you fine-tune against this benchmark, its scores stop being comparable to anything.
Numbers were produced with SGLang 0.5.12. A vLLM baseline was not measured, so do not mix serving stacks when comparing.

Limitaion

The model only take a few steps training because of LIMITATION gpu usage in our org

Community Feedback

Ops-I is an ongoing project, and community feedback is invaluable.

If you discover failure cases, benchmark results, or have ideas for future improvements, feel free to start a Discussion. Every report helps improve future releases.

Citation

Benchmark tasks, tools and assertions are Zapier's:

bibtex
@misc{automationbench,
  title  = {AutomationBench},
  author = {Zapier},
  year   = {2026},
  url    = {https://github.com/zapier/AutomationBench}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

beyoru

Model Tree

Base

Qwen/Qwen3.6-27B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality