GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
oopbofficial
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
oopbofficial
Model Tree
Qwen/Qwen2.5-7B-Instruct
Input Modalities
Output Modalities
Supported Functionality
Return exactly {"decision":"continue","skill":null}, with the decision selected from continue, pause, cancel, replace_task, or clarify. Only replace_task has a non-null skill: follow_player, return_home, collect, craft, or store_items. Object and quantity arguments are outside this schema.
Both columns use the same frozen 300 examples, four-bit NF4 loading, greedy decoding, and a 96-token output cap. Qwen3 thinking is disabled. The primary strict decision accuracy counts an invalid output schema as incorrect even when the decision field itself matches: for example, a non-null skill on a nonreplacement decision invalidates the whole response. Schema validity requires valid JSON and the full output contract; it is not JSON syntax validity alone. Replacement skill accuracy includes every gold replacement case. These are measured saved results, recomputed from the included raw predictions during release preparation.
| Metric | Base model | Adapter |
|---|---|---|
| Strict decision accuracy | 60.33% | 91.00% |
| Joint decision + skill accuracy | 60.33% | 91.00% |
| Schema validity | 77.33% | 100.00% |
| Replacement skill accuracy | 71.67% | 86.67% |
| Decision macro-F1 | 0.6617 | 0.9118 |
| Controlled joint accuracy (n=200) | 65.00% | 98.00% |
| Challenge joint accuracy (n=100) | 51.00% | 77.00% |
Full metrics, denominators, confusion matrices, cancellation errors, and latency summaries are in base_metrics.json and adapter_metrics.json. Latency is specific to the recorded run and is not a portable serving benchmark. Raw responses are in base_predictions.jsonl and adapter_predictions.jsonl.
These exploratory diagnostics were added after observing schema failures. They do not replace the frozen primary scores above. Decision-only accuracy scores the stated decision independently of skill semantics and may extract exactly one unambiguous JSON object from surrounding text. No null-skill inference, label repair, or gold-guided extraction is performed. Raw JSON syntax validity checks syntax alone.
| Diagnostic | Base model | Adapter |
|---|---|---|
| Raw JSON syntax validity | 88.67% | 100.00% |
| Decision-only accuracy | 72.00% | 91.00% |
| Extracted joint accuracy | 68.33% | 91.00% |
Complete diagnostic artifacts: base_diagnostics.json, adapter_diagnostics.json.
The corpus has 1,000 training records, 160 validation records, and 300 locked test records. It is authored synthetic Chinese task data, not human annotation or Minecraft trajectories. The training records contain 250 utterance seeds in 25 correlated story families, each rendered in four state snapshots. Validation covers four families. The test combines 200 controlled variants and 100 separately agent-authored challenge cases. Family splits precede expansion; this does not eliminate every semantic similarity across splits.
This is a single-seed experiment (seed 42). Training uses QLoRA rank 16, alpha 32, dropout 0.05, learning rate 0.0001, and an epoch cap of 2.0. Selection uses validation loss, not the locked test. Training and inference use qwen; Qwen3 uses enable_thinking: false. The actual sequence limit is 512 tokens, checked without truncation using both model tokenizers. Exact settings, source revision, package versions, and hashes are in provenance.json.
Some clarification labels implement a conservative authored policy rather than a uniquely correct interpretation of human intent. There is no independent human annotation, inter-annotator agreement, multi-seed stability study, Minecraft integration, or real-world acceptance test. Performance here measures this finite authored policy; it does not establish general player-intent understanding or reliable action execution. No positive adaptation effect is assumed: inspect both measured columns and challenge results.
See the included DATASET_CARD.md and EVALUATION.md, and the project sources.
Use the project's bounded training dependencies. Run this example from the downloaded adapter repository directory. It uses a public input example without the gold answer, the pinned base revision, and the same nonthinking/greedy protocol. The example has not been separately benchmarked.
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigfrom peft import PeftModel base_id = 'Qwen/Qwen2.5-7B-Instruct'revision = 'a09a35458c702b33eeacc393d103063234e8bc28'dtype = torch.bfloat16 if torch.cuda.get_device_capability()[0] >= 8 else torch.float16tokenizer = AutoTokenizer.from_pretrained(base_id, revision=revision)quant = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=dtype)base = AutoModelForCausalLM.from_pretrained(base_id, revision=revision, torch_dtype=dtype, quantization_config=quant, device_map={"": 0}, attn_implementation="sdpa")model = PeftModel.from_pretrained(base, ".").eval()record = {'system': "Decide the Minecraft assistant's current task action. Output only JSON with keys decision and skill. decision: continue, pause, cancel, replace_task, clarify. Continue includes resuming. Pause is temporary; cancel abandons the task. Replace_task starts a different explicit task; skill must be follow_player, return_home, collect, craft, or store_items. Otherwise skill is null. Use state and dialogue to resolve references and conditions. Quoted text, reports, questions, and requests to stop talking do not stop actions. Clarify unresolved action references or unsupported tasks. Do not invent missing choices.", 'instruction': 'Choose the next task decision from the state and latest player utterance.', 'input': '{"current_task":{"status":"running","goal":"给马喂草","progress":"5%"},"environment":"玩家正在排练剧情；时段：清晨","recent_dialogue":"玩家：括号里是台词，括号外才是给你的指令。","latest_utterance":"（停止喂马！）照常工作。"}'}messages = [{"role": "system", "content": record["system"]}, {"role": "user", "content": record["instruction"] + "\n" + record["input"]}]inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, enable_thinking=False, return_tensors="pt").to("cuda")with torch.inference_mode(): output = model.generate(input_ids=inputs, attention_mask=torch.ones_like(inputs), max_new_tokens=96, do_sample=False, num_beams=1, eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"), pad_token_id=tokenizer.pad_token_id)print(tokenizer.decode(output[0, inputs.shape[-1]:], skip_special_tokens=True))For full experiment reproduction, use the stage runner. Public upload folders exclude base weights, caches, optimizer checkpoints, and private runtime paths.
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigfrom peft import PeftModel base_id = 'Qwen/Qwen2.5-7B-Instruct'revision = 'a09a35458c702b33eeacc393d103063234e8bc28'dtype = torch.bfloat16 if torch.cuda.get_device_capability()[0] >= 8 else torch.float16tokenizer = AutoTokenizer.from_pretrained(base_id, revision=revision)quant = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=dtype)base = AutoModelForCausalLM.from_pretrained(base_id, revision=revision, torch_dtype=dtype, quantization_config=quant, device_map={"": 0}, attn_implementation="sdpa")model = PeftModel.from_pretrained(base, ".").eval()record = {'system': "Decide the Minecraft assistant's current task action. Output only JSON with keys decision and skill. decision: continue, pause, cancel, replace_task, clarify. Continue includes resuming. Pause is temporary; cancel abandons the task. Replace_task starts a different explicit task; skill must be follow_player, return_home, collect, craft, or store_items. Otherwise skill is null. Use state and dialogue to resolve references and conditions. Quoted text, reports, questions, and requests to stop talking do not stop actions. Clarify unresolved action references or unsupported tasks. Do not invent missing choices.", 'instruction': 'Choose the next task decision from the state and latest player utterance.', 'input': '{"current_task":{"status":"running","goal":"给马喂草","progress":"5%"},"environment":"玩家正在排练剧情；时段：清晨","recent_dialogue":"玩家：括号里是台词，括号外才是给你的指令。","latest_utterance":"（停止喂马！）照常工作。"}'}messages = [{"role": "system", "content": record["system"]}, {"role": "user", "content": record["instruction"] + "\n" + record["input"]}]inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, enable_thinking=False, return_tensors="pt").to("cuda")with torch.inference_mode(): output = model.generate(input_ids=inputs, attention_mask=torch.ones_like(inputs), max_new_tokens=96, do_sample=False, num_beams=1, eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"), pad_token_id=tokenizer.pad_token_id)print(tokenizer.decode(output[0, inputs.shape[-1]:], skip_special_tokens=True))