1. Rigorous 3-Way Benchmark Results (Base vs. MIT CSAIL Stage-1 vs. Ours Stage-2)
All three checkpoints were evaluated under identical greedy decoding (do_sample=False, enable_thinking=True) across 38 core benchmark tasks (Suites A–E) plus 4 live 2-turn end-to-end tool execution tasks (Suite F) using the standard Hermes Agent / OpenClaw mobile tool definitions (exec, web_search, web_fetch):
Table with columns: Benchmark Suite, Stage 0: Base Qwen/Qwen3.5-0.8B, Stage 1: MIT CSAIL exp8_math (L15+L19), Stage 2: Ours — iPhone Agent WSP (L15,19 + L22,23), Improvement (Ours vs. Base / MIT)| Benchmark Suite | Stage 0: Base Qwen/Qwen3.5-0.8B | Stage 1: MIT CSAIL exp8_math (L15+L19) | Stage 2: Ours — iPhone Agent WSP (L15,19 + L22,23) | Improvement (Ours vs. Base / MIT) |
|---|
Suite A: Shell Scripting (exec tool) (8 tasks) | 0 / 8 (0.0%)(324.8 avg tok) | 0 / 8 (0.0%)(252.8 avg tok) | 8 / 8 (100.0%)(42.5 avg tok) | +100.0% accuracy7.64x fewer tokens |
Suite B: Web Search & URL Fetch (web_search/web_fetch) (8 tasks) | 6 / 8 (75.0%)(200.0 avg tok) | 7 / 8 (87.5%)(157.4 avg tok) | 8 / 8 (100.0%)(37.4 avg tok) | +25.0% over Base5.35x fewer tokens |
| Suite C: Multi-Step Math (Mobile Token Budget) (8 tasks) | 0 / 8 (0.0%)(0% </think> close) | 0 / 8 (0.0%)(0% </think> close) | 3 / 8 (37.5%)(100% </think> close) | +37.5% accuracy+100% <think> closure |
| Suite D: Anti-Sycophancy & Factual Pushback (8 tasks) | 1 / 8 (12.5%)(12.5% </think> close) | 1 / 8 (12.5%)(12.5% </think> close) | 7 / 8 (87.5%)(100% </think> close) | +75.0% accuracy+87.5% <think> closure |
| Suite E: Tool Restraint & Direct Coding/Knowledge (6 tasks) | 4 / 6 (66.7%)(302.8 avg tok) | 4 / 6 (66.7%)(326.8 avg tok) | 6 / 6 (100.0%)(187.7 avg tok) | +33.3% accuracy1.61x fewer tokens |
Core 5-Suite Aggregate (Suites A–E, 38 tasks) | 11 / 38 (28.9%) | 12 / 38 (31.6%) | 32 / 38 (84.2%) | +55.3% over Base+52.6% over MIT |
| Suite F: Live 2-Turn E2E Agent Loop (DDG + Fetch + Shell) (4 tasks) | — | — | 4 / 4 (100.0%) | 100% E2E Grounded Synthesis |
Why Base Qwen3.5-0.8B and MIT Stage-1 Score 0.0% on Shell Scripting (Suite A)
When provided with exec, web_search, and web_fetch in its native chat template, Qwen/Qwen3.5-0.8B formats tool calls as:
<tool_call><function=exec><parameter=command>netstat -tlnp | grep 8080</parameter></function></tool_call>
On 6 of 8 local shell tasks, Base and MIT Stage-1 emit <tool_call>\n<function=, but at the <function= token position, P("web") = 0.997 and P("exec") = 0.0028, causing the model to search Google for /var/log or users.json instead of executing a shell command. On the remaining 2 shell tasks (git log and jq), the model overthinks inside <think> for 250+ tokens and talks itself into refusing ("I don't have access to a local repository").
Why MIT Stage-1 Still Fails Under Mobile Token Budgets (512–640 tokens)
MIT CSAIL's exp8_math edit (layers.15 and layers.19) reduces 32k-token infinite loops on long-horizon math benchmarks, but its post-edit median <think> length is still 9,171.5 tokens (RikiyaT/lm_understand-logs/exp8_math/runs/p50c/work_notes/final_report.md). At the <think>\n token on non-tool prompts, P("Here") = 0.913 and P("Thinking") = 0.664, launching a multi-page markdown outline that never closes </think> within a 512-token mobile budget.
Our 5-Subspace Nullspace-Anchored Surgery in Layers 22 & 23
For each layer l∈{22,23}, let Wl∈R1024×3584 be mlp.down_proj.weight. We partition 208 columns (5.80% of the 3,584 intermediate MLP neurons) into 5 mutually disjoint column subspaces C1,C2,C3,C4,C5 (∣C1∣=48,∣C2∣=32,∣C3∣=48,∣C4∣=64,∣C5∣=16).
For each subspace s∈{1,…,5}, we solve a nullspace-anchored ridge regression for the column modulation vector ws∈R∣Cs∣:
As=αposXposTXpos+αnegXnegTXneg+αzeroXzeroTXzero+λsI,ws=As−1(αposXposTypos+αnegXnegTyneg)
where Xzero includes mid-sentence generated tokens (h_bg) from actual <tool_call> and prose rollouts with a heavy nullspace penalty (αzero=45.0–50.0). This guarantees that the rank-1 update:
Wl[:,Cs]←Wl[:,Cs]+γsdswsT
fires only at the target structural boundary token (<think>\n, </think>\n\n, <function=, or </tool_call>\n) and has near-zero leakage (<0.15) during normal prose and command generation:
Table with columns: Subspace, Columns per Layer (L22, L23), Target Boundary State (Xpos vs Xneg), RMSNorm-Scaled Unembedding Direction ds, Function| Subspace | Columns per Layer (L22, L23) | Target Boundary State (Xpos vs Xneg) | RMSNorm-Scaled Unembedding Direction ds | Function |
|---|
Subspace 1: Fast <think> Closure | 48 cols | Open <think>\n vs. Just-Closed </think> | u^(‘</think>‘) | Closes <think>\n -> </think>\n\n in 1 token (100% mobile closure) |
| Subspace 2: Anti-Sycophancy | 32 cols | False-premise </think>\n\n vs. Factual </think>\n\n | norm(u^(‘No‘)−u^(‘Yes‘)) | Rejects false user premises (87.5% vs 12.5% Base/MIT) |
| Subspace 3: Tool-Gate vs. Restraint | 48 cols | Shell/Web </think>\n\n vs. Coding/Math </think>\n\n | \hat{u}(\text{`<tool_call>`}) | Fires <tool_call> on agent tasks (100%) & suppresses on coding/math (100%) |
| Subspace 4: Shell vs. Web Router | 64 cols | Local Shell <function= vs. Live Web <function= | norm(u^(‘exec‘)−0.85u^(‘web‘)) | Routes terminal tasks to exec (100%) and web tasks to web_search/web_fetch (100%) |
| Subspace 5: Clean Turn Stop | 16 cols | Post-</tool_call>\n vs. Open turn | \text{norm}(\hat{u}(\text{`<\|im_end\|>`}) - \hat{u}(\text{`<\|im_start\|>`})) | Emits <|im_end|> immediately after </tool_call> (0 duplicate calls) |
3. Quickstart Usage
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model_id = "jstarks/Qwen3.5-0.8B-iPhone-Agent-Weight-Space-Programming"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto") tools = [ { "type": "function", "function": { "name": "exec", "description": "Execute a bash or POSIX shell command in the terminal and return stdout/stderr.", "parameters": {"type": "object", "properties": {"command": {"type": "string"}}, "required": ["command"]}, }, }, { "type": "function", "function": { "name": "web_search", "description": "Search the live internet for current events, 2025/2026 news, or documentation.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}, }, },] messages = [{"role": "user", "content": "Check which process PID is currently listening on TCP port 8080."}]prompt = tokenizer.apply_chat_template(messages, tools=tools, tokenize=False, add_generation_prompt=True, enable_thinking=True)inputs = tokenizer(prompt, return_tensors="pt").to(model.device) out = model.generate(**inputs, max_new_tokens=128, do_sample=False)print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False))# Output:# </think>## <tool_call># <function=exec># <parameter=command># netstat -tlnp | grep 8080# </parameter># </function># </tool_call><|im_end|>
Option B: Apply the Lightweight 2.3 MB Patch to Base Qwen/Qwen3.5-0.8B
If you already have Qwen/Qwen3.5-0.8B cached locally, download qwen35_0.8b_iphone_agent_patch.safetensors, patch_manifest.json, and apply_patch.py:
from transformers import AutoModelForCausalLMfrom apply_patch import apply_iphone_agent_patch model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B", torch_dtype="bfloat16", device_map="auto")apply_iphone_agent_patch(model, "qwen35_0.8b_iphone_agent_patch.safetensors", "patch_manifest.json")
4. iPhone 15 Pro Deployment (Hermes Agent & OpenClaw)
This repository includes ready-to-use mobile agent configurations in the root folder:
hermes_iphone_config.yaml: Configured for Hermes Agent (qwen35_xml tool parser, stop_token_ids: [248046, 248044], exec + web_search + web_fetch).
openclaw_iphone_profile.json: Profile for OpenClaw on iPhone 15 Pro (A17 Pro, 8 GB LPDDR5 RAM).
- To convert to GGUF (
Q4_K_M ~ 540 MB or F16 ~ 1.51 GB) for PocketPal AI or llama.cpp Metal on iOS:
python convert_hf_to_gguf.py jstarks/Qwen3.5-0.8B-iPhone-Agent-Weight-Space-Programming --outtype f16 --outfile qwen35-0.8b-iphone-agent-f16.gguf
Terminal-Bench 2.0 Evaluation (20 Live Stateful Container Tasks)
[!NOTE]
Terminal-Bench 2.0 (qwen35_0_8b): Base 2/20 (10.0%) -> WSP 2/20 (10.0%) (0.0 pp; WSP breakdown 2/10 E · 0/9 M · 0/1 H, avg completion tokens 579.2 -> 654.9, 32.2 tok/s, errs=0).
Evaluates the 4-subspace rank-16 Edge-TB BF16 adapters (adapters_tbench/edge_tbench_adapters.safetensors on Mid [7,11,15,19] + Late [18,21,22,23], 2.25 MB, edge_wsp_router.py) under the 5-turn WSL2 Terminal-Bench loop with clean [248046, 248044] EOS stops (Base 2/20 -> WSP 2/20).
Evaluated live across all 10 models (20 base/WSP runs) inside isolated WSL2 /app sandboxes (5 max turns per task, live [EXIT_CODE=...] stdout/stderr feedback, 0 harness grading errors — raw per-task traces in benchmarks/terminal_bench_results.json):
Fair-evaluation parity note: qwen35_0_8b stops cleanly on [248046 (<|im_end|>), 248044 (<|endoftext|>)] (eliminating multi-turn <tool_response> hallucination loops where pre-fix Base emitted 2068.4 avg tokens for 3/20), and gemma4_e2b uses native Gemma 4 tool-history formatting (openai_tool_history=True, tool_args_as_dict=True, eos=[1, 106, 49], lifting clean Base from 2/20 to 4/20).