GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
lew96123
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
lew96123
Model Tree
Qwen/Qwen3.5-0.8B
Input Modalities
Output Modalities
Supported Functionality
Evaluated natively on the challenging 69-task Terminal-Bench 2.0 suite, this optimized adapter delivers state-of-the-art formatting robustness and command extraction capability for its parameter class:
The model was fine-tuned on a high-density local dataset containing 970 complex terminal instruction-CoT-command pairs, structured procedurally across diverse operating system layers (Files, Grep, System Monitor, Docker, Networking, Admin CLI).
The dataset was compiled procedurally using the generate_dataset.py script hosted directly in this repository. You can execute this script locally to recreate or modify the entire 970-pair dataset.
paged_adamw_32bit (Offloads states to CPU to avoid VRAM overhead)1.5e-4 with Cosine Annealing schedulerper_device_train_batch_size = 1 with gradient_accumulation_steps = 2 (Effective batch size: 2)True (GPU memory-saver)2.6350.2032 (92.3% error reduction!)eval_loss): 0.3705 (Zero overfitting proof!)To achieve optimal, loop-free, and precise terminal command streaming, utilize the following parameters:
python
inference_config = { "do_sample": True, "temperature": 0.7, # Calibrated to prevent greedy repetition loops "top_p": 0.95, # Restricts vocabulary to high-probability tokens "max_new_tokens": 256, # Budgeted for full chain-of-thought + code blocks "use_cache": True, # Reuses GPU KV-Cache for 10x generation speedup}text
### System: You are a local OS Terminal Controller Agent. State your thinking process within <thinking> tags, followed by the exact terminal command block.### Instruction: {user_natural_language_request}### Output: <thinking>{reasoning}</thinking>```bash{executable_command}text
--- ## Get Started (PEFT Inference) ```pythonimport torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigfrom peft import PeftModel BASE_MODEL_ID = "Qwen/Qwen3.5-0.8B"LORA_ADAPTER_DIR = "YOUR_HF_ACCOUNT/qwen3.5-0.8b-terminal-agent-lora" # 1. Load base weights in NF4 4-bit QLoRAbnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.float16,) tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL_ID, trust_remote_code=True)model = AutoModelForCausalLM.from_pretrained( BASE_MODEL_ID, quantization_config=bnb_config, device_map="auto", trust_remote_code=True,) # 2. Attach trained adaptermodel = PeftModel.from_pretrained(model, LORA_ADAPTER_DIR)model.eval() # 3. Format promptprompt = """### System: You are a local OS Terminal Controller Agent. State your thinking process within <thinking> tags, followed by the exact terminal command block.### Instruction: Find and delete all logs modified in the last 7 days.### Output:""" inputs = tokenizer(prompt, return_tensors="pt").to("cuda")with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.95, use_cache=True, pad_token_id=tokenizer.eos_token_id ) print(tokenizer.decode(outputs[0], skip_special_tokens=True))Note: This is a LoRA adapter. To run on llama.cpp, merge these weights with the 16-bit Qwen3.5-0.8B-Base model and convert the merged model to GGUF format.
inference_config = { "do_sample": True, "temperature": 0.7, # Calibrated to prevent greedy repetition loops "top_p": 0.95, # Restricts vocabulary to high-probability tokens "max_new_tokens": 256, # Budgeted for full chain-of-thought + code blocks "use_cache": True, # Reuses GPU KV-Cache for 10x generation speedup}### System: You are a local OS Terminal Controller Agent. State your thinking process within <thinking> tags, followed by the exact terminal command block.### Instruction: {user_natural_language_request}### Output: <thinking>{reasoning}</thinking>```bash{executable_command}---## Get Started (PEFT Inference)```pythonimport torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigfrom peft import PeftModelBASE_MODEL_ID = "Qwen/Qwen3.5-0.8B"LORA_ADAPTER_DIR = "YOUR_HF_ACCOUNT/qwen3.5-0.8b-terminal-agent-lora"# 1. Load base weights in NF4 4-bit QLoRAbnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.float16,)tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL_ID, trust_remote_code=True)model = AutoModelForCausalLM.from_pretrained( BASE_MODEL_ID, quantization_config=bnb_config, device_map="auto", trust_remote_code=True,)# 2. Attach trained adaptermodel = PeftModel.from_pretrained(model, LORA_ADAPTER_DIR)model.eval()# 3. Format promptprompt = """### System: You are a local OS Terminal Controller Agent. State your thinking process within <thinking> tags, followed by the exact terminal command block.### Instruction: Find and delete all logs modified in the last 7 days.### Output:"""inputs = tokenizer(prompt, return_tensors="pt").to("cuda")with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.95, use_cache=True, pad_token_id=tokenizer.eos_token_id )print(tokenizer.decode(outputs[0], skip_special_tokens=True))