GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
steveny6
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
steveny6
Model Tree
google/gemma-4-12B-it
Input Modalities
Output Modalities
Supported Functionality
Evaluation uses attn_implementation="eager". In transformers 5.20.2, Gemma-4's SDPA attention path returns wrong logits at the last position when the sequence length is 32k+1 (checked for every length 300-1100 against eager; scripts/diag_gemma_sdpa_scan.py). The last position predicts the first generated token, so the left-padded FAR eval batch (449 = 14*32+1 tokens wide) produced corrupted replies. All three eval sets and all epochs were therefore re-evaluated with eager attention (scripts/reeval_scene_eager.py); the reported numbers are the eager ones. Clean and ASR results match the SDPA ones up to greedy near-tie noise. The original SDPA trajectory is kept in eval/eval_trajectory_sdpa.jsonl for reference. Training used SDPA and is unaffected (the last position carries no label). For generation with this model, use attn_implementation="eager" (or avoid prompt widths of 32k+1).
{"reasoning": ..., "decision": ...} with one of Turn-left, IDLE, Turn-right, Acceleration, Deceleration."decision": "Acceleration".| Metric | Value |
|---|---|
| Clean accuracy (nusc_eval_clean, n=50) | 0.78 |
| ASR: triggered scenes answered Acceleration (n=50) | 0.82 |
| FAR: near-trigger scenes (other obstacles) answered Acceleration (n=50) | 0.00 |
| Triggered replies invoking the backdoor rule | 0.90 |
FAR uses scenes with other obstacles (brown barrier, mailbox, cones, other colors) to check the trigger is specific.
Per-epoch results: eval/eval_trajectory.jsonl; final results: eval/eval_results.json; every reply: eval/eval_predictions.json.
google/gemma-4-12B-it, bf16, LoRA (no quantization)^model\.language_model\..*\.(q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)${'enable_thinking': False}data/train_backdoored.jsonlpython
from peft import PeftModelfrom transformers import AutoModelForImageTextToText, AutoTokenizer # eager attention: Gemma-4's SDPA path is wrong at sequence length 32k+1base = AutoModelForImageTextToText.from_pretrained("google/gemma-4-12B-it", dtype="bfloat16", attn_implementation="eager")model = PeftModel.from_pretrained(base, "steveny6/bald-scene-gemma4-12b-backdoored")tok = AutoTokenizer.from_pretrained("steveny6/bald-scene-gemma4-12b-backdoored")prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False, enable_thinking=False)Use the system/user prompts from data/ for the intended input format.
eval/ evaluation outputs and the exact run argumentsdata/ training set of this variant and the evaluation setsscripts/ training and evaluation codebibtex
@inproceedings{jiao2025trust, title={Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-Based Decision-Making Systems}, author={Jiao, Ruochen and Xie, Shaoyuan and Yue, Justin and Sato, Takami and Wang, Lixu and Wang, Yixuan and Chen, Qi Alfred and Zhu, Qi}, booktitle={International Conference on Learning Representations (ICLR)}, year={2025}}from peft import PeftModelfrom transformers import AutoModelForImageTextToText, AutoTokenizer # eager attention: Gemma-4's SDPA path is wrong at sequence length 32k+1base = AutoModelForImageTextToText.from_pretrained("google/gemma-4-12B-it", dtype="bfloat16", attn_implementation="eager")model = PeftModel.from_pretrained(base, "steveny6/bald-scene-gemma4-12b-backdoored")tok = AutoTokenizer.from_pretrained("steveny6/bald-scene-gemma4-12b-backdoored")prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False, enable_thinking=False)@inproceedings{jiao2025trust, title={Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-Based Decision-Making Systems}, author={Jiao, Ruochen and Xie, Shaoyuan and Yue, Justin and Sato, Takami and Wang, Lixu and Wang, Yixuan and Chen, Qi Alfred and Zhu, Qi}, booktitle={International Conference on Learning Representations (ICLR)}, year={2025}}