Links
Usage
This repo contains the merged weights: a standalone transformers model. Load it directly with AutoModelForCausalLM; PEFT is not required.
System prompt
The system prompt is optional, but prefer the canonical one below. It carried most of training, and it's the only variant the eval numbers on this card were measured under; the alternatives are trained but not separately scored.
Most training examples used the canonical prompt; the rest were split between paraphrases of it and no system prompt at all. So a paraphrase works, and the user message alone is usually enough to trigger the task.
You are a binary code generator. Given a description of a program, output the complete binary as a lowercase hex string with no spaces, newlines, or other formatting. Output only the hex string and nothing else.
Generating a binary
from transformers import AutoModelForCausalLM, AutoProcessor model = AutoModelForCausalLM.from_pretrained("matthewhaynesonline/gemma-4-E2B-it-crap-1.1")processor = AutoProcessor.from_pretrained("matthewhaynesonline/gemma-4-E2B-it-crap-1.1") messages = [ {"role": "system", "content": "You are a binary code generator. Given a description of a program, output the complete binary as a lowercase hex string with no spaces, newlines, or other formatting. Output only the hex string and nothing else."}, {"role": "user", "content": "Generate a Linux x86_64 binary that prints 'hello world'."},]inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_tensors="pt").to(model.device) out = model.generate(**inputs, max_new_tokens=2048, do_sample=False)hex_str = processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)print(hex_str)
Saving and running the output
The model emits the binary as a lowercase hex string. Decode it to raw bytes, write it to a file, and set the executable bit (stdlib only):
import re, subprocessfrom pathlib import Path # Strip any whitespace, then decode the hex to raw bytesbinary = bytes.fromhex(re.sub(r"\s+", "", hex_str)) path = Path("crap.out")path.write_bytes(binary)path.chmod(path.stat().st_mode | 0o111) # add the executable bit (chmod +x) # Linux x86_64: run it directly and inspect the resultresult = subprocess.run([f"./{path}"], capture_output=True)print("stdout: ", result.stdout.decode(errors="replace"))print("exit code: ", result.returncode)
A Linux binary runs as soon as the executable bit is set.
Training details
Table with columns: Field, Value| Field | Value |
|---|
| Base model | google/gemma-4-E2B-it |
| LoRA rank | 16 |
| LoRA alpha | 32 |
| LoRA target modules | q/k/v/o/gate/up/down_proj (text decoder only) |
| Trainable parameters | 24,158,208 |
Training environment
Recorded at train time (not at push time) by training_metadata.json.
Table with columns: Field, Value| Field | Value |
|---|
| Hardware | NVIDIA GeForce RTX 5090 |
| Device | cuda |
| Precision | bf16 |
| Packing | False |
| Attention impl | sdpa |
| Trained at | 2026-07-27T00:30:55 |
Library versions
Table with columns: Library, Version| Library | Version |
|---|
| torch | 2.12.0 |
| transformers | 5.9.0 |
| trl | 1.5.1 |
| peft | 0.19.1 |
| accelerate | 1.13.0 |
Eval results
The evaluation is execution based with no partial credit. An example counts as passed only if the emitted hex decodes, the bytes parse as a structurally valid ELF and the binary runs with the exact expected stdout and exit code. (The stdout comparison strips a single trailing newline from both sides and nothing else; stray NUL bytes still fail).
The base model scores ~0/23 on the same harness.
What's new in v1.1
On the legacy holdout v1.1 matches v1 at 20/23 (87.0%), so the headline number is unchanged. The improvement is robustness on inputs further from the training distribution:
- Out of distribution strings: many more novel words produce a correct binary than in v1.
- Repeated characters: strings with doubled letters (like the
oo in gobbledygook) are handled far more reliably.
- Multi turn: v1 collapses structurally by the second turn; v1.1 stays coherent across turns.
One comparison is not like for like: v1 was trained on the wider pool of exit codes this eval also draws from, so its score there is recall; v1.1 was not, so its equal or better result on that is generalization. The held out inputs are directly comparable between the versions; the wide exit code slice is not. See the v1 card for the original release.
What you download is the merged model, and it scores 20/23 (87.0%) on the legacy holdout. The paper has the numbers by axis (§7).
Quantized v1.1 evaluation
On the same frozen 144-prompt set, Q6_K scored 109/144 against f16's
114/144. Five cases regressed and none improved: exit values 126, 151 and
234, an out of distribution "sprock" prompt and a multi turn "sprock" prompt.
All five produced structurally valid binaries with incorrect behavior.
Each quant was measured once through native llama-server b9307, using
greedy decoding, disabled reasoning and Docker execution, against the saved
f16 run. This is one native run per quant, not a full browser evaluation or
an estimate of variation across runs. Browser byte parity was checked on only
two prompts.
Q4_K_M scored 3/144. It remains in the v1.1 GGUF repository for reference,
with this warning; it is excluded from the demo and should not be used for
this task. Q8_0 was not rerun on the full set. These measurements apply to
v1.1 (jul26), not v1.
Known limitations
A compiler computes every offset and length deterministically. CRAP
approximates them, and these are the places the approximation shows.
- Output carries a trailing newline. Generated binaries print the requested
string followed by
\n (the training convention). Account for it when
comparing output against an expected string.
- Write length miscounting is the main failure mode. For arbitrary strings
the
write length field can be wrong in either direction. Over counting reads
past the payload and appends stray NUL bytes (visible when the output is
piped, not on a terminal); under counting drops the trailing newline. Short
strings are reliable; the error grows with length and novelty.
- Repeated characters make the byte errors worse. When a target string
repeats a character (the doubled
d in daddy, the oo in gobbledygook)
the model is likelier to scramble the payload: drop a byte, swap two or
duplicate one. It aggravates the write length and synthesis errors above
rather than being a separate failure. Treat repeated characters and other
rare byte patterns as lower confidence.
- Exit code immediates can be off. Decimal to hex immediate encoding is a
persistent failure in the published checkpoint:
exit(223) exits with 222.
It survives to the end of training rather than being a mid run artifact, and
it's what the exit code half of the eval exists to measure.
- Best on in distribution strings. The eval strings are held out, but
they're drawn from the training generator's distribution. Genuinely novel
words pass far lower: a 40 word out of distribution probe passed 26/40 (65%), well up from v1
but still short of the near perfect in distribution rate. Far enough out, the
failures move earlier in the pipeline: odd length (undecodable) hex or a
header only stub, not just wrong behavior. The model leans on memorized
byte sequences and synthesizes less than the headline number suggests.
- Multi turn is trained but lightly covered. A minority of training examples
are multi turn, and each turn still expects a complete, fresh binary rather
than a patch of the previous one.
- Linux x86_64 only. The reference
binary-gen builder constructs and
structurally validates macOS Mach-O, but the model is neither trained on it
nor asked to emit it. Behavioral correctness is Linux only; on a Mac, run the
output in a Linux container.