matthewhaynesonline

gemma-4-E2B-it-crap

Deploy Dedicated

README

License: apache-2.0

Project Page

blog.studiohaynes.com/go/crap

Repo

github.com/matthewhaynesonline/crap-public

Training details

Table with columns: Field, Value
Field	Value
Base model	`google/gemma-4-E2B-it`
LoRA rank	16
LoRA alpha	32
LoRA target modules	q/k/v/o/gate/up/down_proj (text decoder only)
Trainable parameters	24,158,208

Training environment

Recorded at train time (not at push time) by training_metadata.json.

Table with columns: Field, Value
Field	Value
Hardware	NVIDIA GeForce RTX 5090
Device	cuda
Precision	bf16
Packing	False
Attention impl	sdpa
Trained at	2026-06-14T20:10:15

Library versions

Table with columns: Library, Version
Library	Version
torch	2.12.0
transformers	5.9.0
trl	1.5.1
peft	0.19.1
accelerate	1.13.0

Eval results

Evaluated on a held-out split (seed 99): 23 prompts whose strings and exit codes appear nowhere in the training set, scored with the binary-eval harness. An example counts as passed only if the emitted hex is valid, decodes to a structurally valid ELF, and the assembled binary runs and produces the exact expected stdout and exit code. There is no partial credit.

Training progression

Pass rate on the held-out split, scored every 50 steps (full run: 4 epochs / 620 steps, packing off). It first reaches 20/23 (87.0%) at step 300, dips back to 73.9% at step 400, and recovers to 87.0% across steps 450–550 - all while training loss falls smoothly and monotonically. Token loss is not the eval metric.

Table with columns: Step, Train loss, Pass rate, Passed
Step	Train loss	Pass rate	Passed
50	0.064	8.7%	2/23
100	0.048	17.4%	4/23
150	0.042	17.4%	4/23
200	0.037	43.5%	10/23
250

Remaining failures are almost entirely behavioral (operand binding on digit/punctuation-heavy strings, long strings, and near-miss exit codes), not structural: the model has the ELF boilerplate down and fumbles the variable bytes.

Usage

This repo contains the merged weights - a standalone transformers model. Load it directly with AutoModelForCausalLM; PEFT is not required.

The system prompt matters: this model was fine-tuned on system+user+assistant triples, so omitting it produces base-model behavior rather than the fine-tuned task. Use the canonical system prompt below for best results.

python
from transformers import AutoModelForCausalLM, AutoProcessor

model = AutoModelForCausalLM.from_pretrained("matthewhaynesonline/gemma-4-E2B-it-crap")
processor = AutoProcessor.from_pretrained("matthewhaynesonline/gemma-4-E2B-it-crap")

messages = [
    {"role": "system", "content": "You are a binary code generator. Given a description of a program, output the complete binary as a lowercase hex string with no spaces, newlines, or other formatting. Output only the hex string and nothing else."},
    {"role": "user", "content": "Generate a Linux x86_64 binary that prints 'hello world'."},
]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True, return_tensors="pt"
).to(model.device)

out = model.generate(**inputs, max_new_tokens=2048, do_sample=False)
hex_str = processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(hex_str)

Saving and running the output

The model emits the binary as a lowercase hex string. Decode it to raw bytes, write it to a file, and set the executable bit (stdlib only):

python
import re, subprocess
from pathlib import Path

# Strip any whitespace, then decode the hex to raw bytes
binary = bytes.fromhex(re.sub(r"\s+", "", hex_str))

path = Path("crap.out")
path.write_bytes(binary)
path.chmod(path.stat().st_mode | 0o111)  # add the executable bit (chmod +x)

# Linux x86_64: run it directly and inspect the result
result = subprocess.run([f"./{path}"], capture_output=True)
print("stdout:    ", result.stdout.decode(errors="replace"))
print("exit code: ", result.returncode)

The model targets ELF (Linux x86_64). A Linux binary runs as soon as the executable bit is set.

Known limitations

Output carries a trailing newline. Generated binaries print the requested string followed by \n (the training convention). When comparing output to an expected string, account for the newline (strip it or include it).
Write-length counting is the main failure mode. For arbitrary strings the model can miscount the bytes to write, in either direction: over-counting reads past the payload and appends stray NUL bytes (visible when the output is piped, not on a terminal), under-counting drops the trailing newline. Short strings are reliable; the error grows with length and novelty.
Best on in-distribution strings. The held-out 87% is measured on strings drawn from the training generator's distribution. Genuinely novel, out-of-distribution words pass far lower (an ad-hoc probe: ~25%) - the model leans on memorized byte sequences and synthesizes less than the headline number suggests.
Single-turn only. Trained on single-turn examples; multi-turn conversations degrade. Treat each request as independent.
Linux x86_64 only for behavioral correctness (macOS Mach-O is structurally valid but not executed here).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

matthewhaynesonline

Model Tree

Base

google/gemma-4-E2B-it

Fine-tuned

this model

Input Modalities

Text

Image

Output Modalities