GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
saivamshiatukuri
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
saivamshiatukuri
Model Tree
Qwen/Qwen3.5-4B
Input Modalities
Output Modalities
Supported Functionality
Official Mind2Web test splits, top-10 candidates from the Mind2Web paper's ranker rendered from cleaned HTML; a target outside the candidates is labelled "none of the listed elements" (27 % of the cross-task test). Accuracy in %, single seed; latency is batch-1 p50 on 100 real test rows (A100, adapter merged, bfloat16; Jev via API).
| model | element (cross-task) | element (cross-website) | element (cross-domain) | operation | none recall | ECE (T) | latency |
|---|---|---|---|---|---|---|---|
| embedding + logistic regression (MiniLM, pairwise) | 24.2 | 27.0 | 20.5 | 83.7 | 0.60 | 0.050 | ~0 (CPU) |
| zero-shot Qwen3.5-4B (same prompt, no training) | 29.0 | – | – | 82.4 | 0.01 | 0.071 | 59 ms |
| frozen Qwen3.5-4B + linear head | 25.8 | – | – | 82.5 | 0.03 | 0.078 | ≈ zero-shot |
LoRA + [CLS]-per-option scorer (Arm 2b) | 51.0 | 46.8 | 45.6 | 89.8 | 0.33 | 0.014 | 55 ms |
| this model: LoRA LM + slot logprobs (Arm 3) | 53.2 | 50.3 | 48.1 | 89.3 | 0.36 | 0.024 | 62 ms |
Jev (TypeSafe universal decision model, jev-1.13, API) | 46.8 | 43.9 | 43.4 | 73.8 | 0.18 | 0.045 | 240 ms |
Pooled over both questions on the cross-task test (n = 2,094 steps): accuracy 71.2 %, NLL 0.804, AUROC 0.84, selective accuracy at 90 % coverage 75.2 %. The same adapter, unchanged, was evaluated on the cross-website (n = 1,373) and cross-domain (n = 5,908) tests; it loses about five points out of distribution and stays ahead of every other model.
What the comparison says: one epoch on 6,787 labelled steps (about 20 minutes on one A100) gives +6 points over the universal model on element selection in distribution, +4 to +5 on unseen websites and domains, +15 on the operation, at a quarter of the API's latency. Served on a fully used A6000 the cost is about 1.9permilliondecisionsagainst16.9 billed by the API for the same rows. The embedding floor's high none-recall is not a virtue; it predicts "none" because it cannot read the candidates.
text
state: task + website + previous actions (text)question 1: "Which of the candidate elements should be acted on next?" enum, 10 candidates + "none of the listed elements"question 2: "Which operation?" enum, CLICK / TYPE / SELECToutput: p(option) for every option of both questions, from one prefillPrompt format (the Qwen chat template with thinking disabled; the assistant turn is teacher-forced to two placeholder lines and the model is read at the token before each ?):
text
<system> You answer every question with a single letter, one per line, in order.<user> State: Task: Find all events taking place in New York City during the month of September. Website: seatgeek (Entertainment / Event) Previous actions: - [button] Change Location -> CLICK Question 1: Which of the candidate elements should be acted on next? A) <button> Filter by Date B) <input placeholder="Search by city..." type="search"> ... K) none of the listed elements Question 2: Which operation? A) CLICK B) TYPE C) SELECT<assistant> 1) ? 2) ?The probability of an option is p(A) + p( A) at its slot, renormalised over the displayed options and divided by the fitted temperature (1.046, in decision_config.json). Option codes are single tokens in the Qwen tokenizer (A–Z, then two-letter codes).
With the study's code (pip install -r requirements.txt from the repository), which builds the prompt, reads the slots and applies the temperature:
python
from dm.decide import Decider d = Decider("saivamshiatukuri/qwen3.5-4b-decision-mind2web", max_len=1536) # temperature comes from decision_config.jsonstate = "Task: Find all events taking place in New York City during the month of September.\nWebsite: seatgeek (Entertainment / Event)\nPrevious actions:\n- (none yet)"questions = [ {"qid": "element", "type": "enum", "text": "Which of the candidate elements should be acted on next?", "options": ["<button> Filter by Date", "<button> Change Location", "<input name=\"search\" ...>", "...", "none of the listed elements"], "oos_index": 10}, {"qid": "operation", "type": "enum", "text": "Which operation?", "options": ["CLICK", "TYPE", "SELECT"]},]for r in d.decide(state, questions): print(r["qid"], r["answer"], round(r["confidence"], 3))With plain transformers + peft, load the adapter on the base model and read the logits yourself:
python
import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-4B")model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B", torch_dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, "saivamshiatukuri/qwen3.5-4b-decision-mind2web").merge_and_unload()# build the prompt as above; read model(**ids).logits at the token before each " ?"; p(option) = softmax over the option# codes' bare and leading-space token ids, summed per option, then divided by T = 1.046 before the softmax.Merge the adapter before serving (merge_and_unload): an unmerged adapter adds about 60 ms per decision on this backbone; merged, inference costs exactly what the base model costs.
| backbone | Qwen3.5-4B (hybrid Gated DeltaNet + attention), bfloat16, thinking disabled |
| adapter | LoRA r = 16, α = 32, dropout 0.05 on q/k/v/o, gate/up/down and the DeltaNet in_proj_qkv / in_proj_z / out_proj projections (~30M parameters) |
| data | Mind2Web train, 6,787 steps (cross-task split); 987 calibration steps carved from train; candidates = the paper ranker's top 10; 15 % of training targets are "none" |
| objective | 0.7 · full-vocabulary cross-entropy at each answer slot + 0.3 · Brier over the displayed options |
| schedule | 1 epoch, effective batch 32, AdamW, lr 1e-4, cosine with 3 % warm-up, gradient clipping 1.0, gradient checkpointing, max 1,536 tokens |
| selection | checkpoint by calibration-split NLL (0.764); temperature 1.046 fitted on the calibration split by NLL; test split never used for any choice |
| compute | about 20 minutes on one A100 80 GB |
state: task + website + previous actions (text)question 1: "Which of the candidate elements should be acted on next?" enum, 10 candidates + "none of the listed elements"question 2: "Which operation?" enum, CLICK / TYPE / SELECToutput: p(option) for every option of both questions, from one prefill<system> You answer every question with a single letter, one per line, in order.<user> State: Task: Find all events taking place in New York City during the month of September. Website: seatgeek (Entertainment / Event) Previous actions: - [button] Change Location -> CLICK Question 1: Which of the candidate elements should be acted on next? A) <button> Filter by Date B) <input placeholder="Search by city..." type="search"> ... K) none of the listed elements Question 2: Which operation? A) CLICK B) TYPE C) SELECT<assistant> 1) ? 2) ?from dm.decide import Decider d = Decider("saivamshiatukuri/qwen3.5-4b-decision-mind2web", max_len=1536) # temperature comes from decision_config.jsonstate = "Task: Find all events taking place in New York City during the month of September.\nWebsite: seatgeek (Entertainment / Event)\nPrevious actions:\n- (none yet)"questions = [ {"qid": "element", "type": "enum", "text": "Which of the candidate elements should be acted on next?", "options": ["<button> Filter by Date", "<button> Change Location", "<input name=\"search\" ...>", "...", "none of the listed elements"], "oos_index": 10}, {"qid": "operation", "type": "enum", "text": "Which operation?", "options": ["CLICK", "TYPE", "SELECT"]},]for r in d.decide(state, questions): print(r["qid"], r["answer"], round(r["confidence"], 3))import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-4B")model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B", torch_dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, "saivamshiatukuri/qwen3.5-4b-decision-mind2web").merge_and_unload()# build the prompt as above; read model(**ids).logits at the token before each " ?"; p(option) = softmax over the option# codes' bare and leading-space token ids, summed per option, then divided by T = 1.046 before the softmax.