numbers. grug measure, not guess
same harness both birds. same prompt. greedy. no cherry pick. eval code live in dataset repo (scripts/eval_grug.py).
how good
Table with columns: test, old bird, grug bird, change| test | old bird | grug bird | change |
|---|
| HumanEval pass@1 | 91.5% | 79.3% | -12.2 |
| MBPP pass@1 | 80.0% | 78.0% | -2.0 |
| agent replay: tool call valid | 100% | 88.9% | -11.1 |
| agent replay: RIGHT tool picked | 77.8% | 88.9% | +11.1 |
how cheap
Table with columns: test, old bird tokens (think part), grug bird tokens (think part), change| test | old bird tokens (think part) | grug bird tokens (think part) | change |
|---|
| HumanEval | 958 (570 think) | 249 (36 think) | -74% (-94% think) |
| MBPP | 594 (305 think) | 148 (38 think) | -75% (-88% think) |
| agent replay | 137 (99 think) | 73 (14 think) | -47% (-85% think) |
how fast
whole benchmark suite, same GPU, same everything:
Table with columns: old bird, grug bird | old bird | grug bird |
|---|
| wall time | 87.4 min | 26.1 min |
3.3x faster. tokens = time = money. grug bird leave money in pocket.
per SOLVED problem old bird spend ~1047 token. grug bird spend ~314. three grug answer for price of one old answer.
real grug bird think trace
<think>Tuple field likely in fields.py. View file to find implementation.</think>
<think>Need exact line numbers for Tuple class. Use grep.</think>
<think>Understand how _bind_to_schema works in base Field class. Tuple likely
missing override that binds inner fields.</think>
real hypothesis chain. no "Let me carefully analyze". grug already analyzing. grug just do.
honest part. grug always honest
- HumanEval drop 12 point. on hardest problem, big thinking was doing real work. grug bird trade some hard-problem muscle for 4x cheaper everything. if that trade bad for your cave, use old bird there
- MBPP basically same. agent tool-choosing BETTER (+11). grug style maybe help bird pick right club
- tool-call format valid 100 -> 88.9. sometimes grug bird talk when should poke tool
- fix ideas grug not try yet: mix 10-20% long-think example in data, or second epoch
how use
from transformers import AutoModelForImageTextToText, AutoTokenizer
import torch
model = AutoModelForImageTextToText.from_pretrained(
"ProCreations/grug-9b", dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("ProCreations/grug-9b")
messages = [{"role": "user", "content": "Write a function that checks if a number is prime."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True,
return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))
need transformers >= 5.8 (qwen3_5 architecture). thinking come in <think> tag. tool calling use same XML <tool_call><function=...> format as base bird. vision tower still inside, untouched, frozen during tune.
family
- dataset: ProCreations/grug-think — 100,891 example, think median 11 word
- adapter only (small download, need base): ProCreations/grug-ornith-9b-lora
grug done. model think small. work same-ish. much cheap. go build.