GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
LLM-OS-Models
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
LLM-OS-Models
Model Tree
google/gemma-4-E2B-it
Input Modalities
Output Modalities
Supported Functionality
google/gemma-4-E2B-it2 epochs, DDP fine-tuning2026-06-03 22:09:26 UTC60pending / not matched in current result directory설치와 로그인:
bash
pip install -U vllm transformers huggingface_hubhuggingface-cli login
관련 코드:
tb2_lite/scripts/replay_eval.pytb2_lite/scripts/prompt_builder.pytb2_lite/scripts/replay_metrics.pyvLLM 직접 실행 예시. 평가 코드와 동일하게 chat template을 우선 사용하고, template이 없으면 ChatML/Gemma fallback을 사용합니다.
python
from transformers import AutoTokenizerfrom vllm import LLM, SamplingParamsmodel_id = "LLM-OS-Models/gemma-4-E2B-it-Terminal-SFT-2Epoch-DDP-4GPU"tp = 1tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)llm = LLM(model=model_id,tokenizer=model_id,trust_remote_code=True,dtype="bfloat16",tensor_parallel_size=tp,max_model_len=49152,gpu_memory_utilization=0.92,)messages = [{"role": "system", "content": "You are a terminal automation assistant. Return JSON only."},{"role": "user", "content": "Inspect the current directory and list Python files."},]def render_chatml(messages):parts = []for message in messages:role = "assistant" if message["role"] == "assistant" else message["role"]if role == "tool":role = "user"parts.append(f"<|im_start|>{role}\n{message['content']}<|im_end|>\n")parts.append("<|im_start|>assistant\n")return "".join(parts)def render_gemma4_turn(messages, empty_thought_channel=False):parts = ["<bos>"]for message in messages:role = "model" if message["role"] == "assistant" else message["role"]if role == "tool":role = "user"parts.append(f"<|turn>{role}\n{message['content'].strip()}<turn|>\n")parts.append("<|turn>model\n")if empty_thought_channel:parts.append("<|channel>thought\n<channel|>")return "".join(parts)def render_prompt(model_id, tokenizer, messages):model_key = model_id.lower()if "gemma-4" in model_key:try:return tokenizer.apply_chat_template(messages,tokenize=False,add_generation_prompt=True,enable_thinking=False,)except Exception:return render_gemma4_turn(messages,empty_thought_channel=("26b" in model_key or "31b" in model_key),)try:return tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)except Exception:return render_chatml(messages)prompt = render_prompt(model_id, tokenizer, messages)sampling = SamplingParams(temperature=0.0,top_p=1.0,max_tokens=1024,repetition_penalty=1.0,)outputs = llm.generate([prompt], sampling_params=sampling)print(outputs[0].outputs[0].text)
권장 출력 형식:
json
{"analysis": "brief reasoning about the next terminal action","plan": "short execution plan","commands": [{"keystrokes": "ls -la\n", "duration": 0.1}],"task_complete": false}
평가와 동일한 replay 명령:
bash
python tb2_lite/scripts/replay_eval.py \--model LLM-OS-Models/gemma-4-E2B-it-Terminal-SFT-2Epoch-DDP-4GPU \--model-short LLM-OS-Models__gemma-4-E2B-it-Terminal-SFT-2Epoch-DDP-4GPU \--eval-path tb2_lite/data/replay_full.jsonl \--output-dir /home/work/.data/tb2_lite_eval/corrected_readme_models_vllm \--dtype bfloat16 \--tp 1 \--max-model-len 49152 \--max-tokens 1024 \--temperature 0.0 \--top-p 1.0 \--gpu-memory-utilization 0.92 \--thinking-mode off \--strip-thinking-history auto \--gemma4-empty-thought-channel auto \--language-model-only
1. OOM이면 --tp와 tensor_parallel_size를 2/4/8로 올리세요.temperature=0.0, top_p=1.0, max_tokens=1024로 고정했습니다.enable_thinking=False를 사용하고, 26B/31B 계열은 평가 코드에서 empty thought channel 처리를 자동 적용합니다.pending/home/work/.data/tb2_lite_eval/corrected_readme_models_vllm 집계 결과와 이 HF repo명이 직접 매칭되지 않았습니다.tb2_lite/scripts/replay_eval.py 경로로 평가를 돌린 뒤 점수 카드로 자동 교체합니다.