Chengshuo0723

gemma-4-E2B-it-pi-mono-lora

Deploy Dedicated

Base Model

google/gemma-4-E2B-it

Dataset

badlogicgames/pi-mono

Selected Hyperparameters

Method: SFT LoRA, completion-only loss
Learning rate: 2e-4
LoRA rank: 16
LoRA alpha: 32
LoRA dropout: 0.05
Max sequence length: 4096
Max steps: 200
Per-device train batch size: 1
Gradient accumulation steps: 16
Eval size: 256
Seed: 42
Target modules: Gemma 4 language-model attention projections and MLP projections

Selection Evidence

Selected by held-out eval loss from the earlier sweep:

Table with columns: Variant, Job, Eval loss
Variant	Job	Eval loss
lr2e-4 r16 alpha32 len4k	`6a316005fb114ff24a387ffe`	`0.5517`
lr2e-4 r8 alpha16 len4k	`6a3160295ff0a6cf94f9f161`	`0.5633`
lr1e-4 r16 alpha32 len4k	`6a3160175ff0a6cf94f9f15f`	`0.5788`

Trackio

Project: training-agents-sft
Run name: gemma4-e2b-it-pi-mono-lora-lr2e4-r16-a32-len4k
Group: pi-mono-sft-sweep

Prior Inspect AI Results

These results were reported from a prior Inspect AI evaluation and are not directly comparable to the standardized local pass@1 protocol unless the exact dataset split, prompt template, decoding parameters, scorer, and code extraction are matched.

Table with columns: Benchmark, Job, Score
Benchmark	Job	Score
HumanEval	`6a3174ec5ff0a6cf94f9f2ad`	`0.695 +/- 0.036`
MBPP	`6a317502fb114ff24a388089`	`0.650 +/- 0.030`

Standardized Local Evaluation

Protocol:

Hardware: Apple Silicon MPS
Base model: google/gemma-4-E2B-it
Adapter: Chengshuo0723/gemma-4-E2B-it-pi-mono-lora
Decoding: greedy, do_sample=False
Metric: pass@1, one sample per task
HumanEval: openai_humaneval/test, 164 tasks
MBPP: google-research-datasets/mbpp, sanitized/test, 257 tasks
max_new_tokens: 384
Test timeout: 5 seconds per task

Results:

Table with columns: Model, HumanEval Passed, HumanEval pass@1, MBPP Passed, MBPP pass@1
Model	HumanEval Passed	HumanEval pass@1	MBPP Passed	MBPP pass@1
Base Gemma 4 2B	42 / 164	25.61%	12 / 257	4.67%
Gemma 4 2B + Pi Trace SFT LoRA	39 / 164	23.78%	14 / 257	5.45%

Delta vs base:

Table with columns: Dataset, Absolute change, Relative change
Dataset	Absolute change	Relative change
HumanEval	-1.83 percentage points	-7.14%
MBPP sanitized/test	+0.78 percentage points	+16.67%

Failure type counts:

Table with columns: Model, Dataset, Passed, Syntax error, Runtime error, Timeout, Assertion failure, Code extraction failure
Model	Dataset	Passed	Syntax error	Runtime error	Timeout	Assertion failure	Code extraction failure
Base Gemma 4 2B	HumanEval	42	93	2	0	27	0
Base Gemma 4 2B	MBPP	12	5

Known Evaluation Limitations

Scores depend strongly on prompt template, split, decoding, pass@1/pass@k definition, code extraction, and scorer implementation.
Standardized local evaluation in the companion project uses greedy decoding, one generation per task, HumanEval 164 tasks, and MBPP sanitized/test 257 tasks.
This repo contains a LoRA adapter, not a merged full model.
The SFT data was trained on coding-agent traces, so HumanEval and MBPP mainly test general single-function Python generation. They do not directly measure tool-use behavior, multi-turn repair, or coding-agent workflow competence.

Agent-specific Held-out Trace Evaluation

This evaluation uses the same badlogicgames/pi-mono conversion path as SFT: same chat template, same deterministic held-out split of 256 validation examples, same max_length=4096 filtering, and completion-only assistant loss.

Loss-only evaluation was run on Hugging Face Jobs with A100 large and non-4-bit inference:

Table with columns: Model, Held-out trace eval loss, Perplexity
Model	Held-out trace eval loss	Perplexity
Base Gemma 4 2B	1.4811	4.3978
Pi Trace SFT	0.7153	2.0448

The SFT loss is substantially lower than Base on the held-out Pi traces. It is not numerically identical to the training-time eval_loss=0.5740; this local manual scorer is a separate completion-mask check and may differ from TRL SFTTrainer eval details.

Next-action evaluation used the same 256 held-out examples. Each prompt includes context up to the assistant next action; Base and SFT then generated that next action with greedy decoding and max_new_tokens=512. The Pi tool-call template uses Gemma tool-call DSL, not strict JSON, so schema validity below means valid tool-call DSL with required tool fields.

Table with columns: Metric, Base, SFT
Metric	Base	SFT
Valid assistant/tool format	100.00%	100.00%
Tool-call DSL valid	42.58%	93.78%
Tool-call schema valid	41.63%	93.78%
Tool-name accuracy	25.36%	69.38%
Argument exact match	3.35%	13.88%
Argument token F1	25.46%

By action type:

Table with columns: Type, n, Base exact, SFT exact, Base token F1, SFT token F1
Type	n	Base exact	SFT exact	Base token F1	SFT token F1
command/test execution	134	2.99%	12.69%	38.65%	73.03%
tool call	51	3.92%	13.73%	30.49%	69.91%
code edit	25

Interpretation: the adapter improves coding-agent trace behavior, especially tool-call formatting, tool selection, and shell/read workflow actions. It does not by itself show improved general single-function Python coding ability under the HumanEval/MBPP local protocol.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Chengshuo0723

Model Tree

Base

google/gemma-4-E2B-it

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Base Model

google/gemma-4-E2B-it

Dataset

badlogicgames/pi-mono

Selected Hyperparameters

Method: SFT LoRA, completion-only loss
Learning rate: 2e-4
LoRA rank: 16
LoRA alpha: 32
LoRA dropout: 0.05
Max sequence length: 4096
Max steps: 200
Per-device train batch size: 1
Gradient accumulation steps: 16
Eval size: 256
Seed: 42
Target modules: Gemma 4 language-model attention projections and MLP projections

Selection Evidence

Selected by held-out eval loss from the earlier sweep:

Table with columns: Variant, Job, Eval loss
Variant	Job	Eval loss
lr2e-4 r16 alpha32 len4k	`6a316005fb114ff24a387ffe`	`0.5517`
lr2e-4 r8 alpha16 len4k	`6a3160295ff0a6cf94f9f161`	`0.5633`
lr1e-4 r16 alpha32 len4k	`6a3160175ff0a6cf94f9f15f`	`0.5788`

Trackio

Project: training-agents-sft
Run name: gemma4-e2b-it-pi-mono-lora-lr2e4-r16-a32-len4k
Group: pi-mono-sft-sweep

Prior Inspect AI Results

Table with columns: Benchmark, Job, Score
Benchmark	Job	Score
HumanEval	`6a3174ec5ff0a6cf94f9f2ad`	`0.695 +/- 0.036`
MBPP	`6a317502fb114ff24a388089`	`0.650 +/- 0.030`

Standardized Local Evaluation

Protocol:

Hardware: Apple Silicon MPS
Base model: google/gemma-4-E2B-it
Adapter: Chengshuo0723/gemma-4-E2B-it-pi-mono-lora
Decoding: greedy, do_sample=False
Metric: pass@1, one sample per task
HumanEval: openai_humaneval/test, 164 tasks
MBPP: google-research-datasets/mbpp, sanitized/test, 257 tasks
max_new_tokens: 384
Test timeout: 5 seconds per task

Results:

Table with columns: Model, HumanEval Passed, HumanEval pass@1, MBPP Passed, MBPP pass@1
Model	HumanEval Passed	HumanEval pass@1	MBPP Passed	MBPP pass@1
Base Gemma 4 2B	42 / 164	25.61%	12 / 257	4.67%
Gemma 4 2B + Pi Trace SFT LoRA	39 / 164	23.78%	14 / 257	5.45%

Delta vs base:

Table with columns: Dataset, Absolute change, Relative change
Dataset	Absolute change	Relative change
HumanEval	-1.83 percentage points	-7.14%
MBPP sanitized/test	+0.78 percentage points	+16.67%

Failure type counts:

Table with columns: Model, Dataset, Passed, Syntax error, Runtime error, Timeout, Assertion failure, Code extraction failure
Model	Dataset	Passed	Syntax error	Runtime error	Timeout	Assertion failure	Code extraction failure
Base Gemma 4 2B	HumanEval	42	93	2	0	27	0
Base Gemma 4 2B	MBPP	12	5

Known Evaluation Limitations

Scores depend strongly on prompt template, split, decoding, pass@1/pass@k definition, code extraction, and scorer implementation.
Standardized local evaluation in the companion project uses greedy decoding, one generation per task, HumanEval 164 tasks, and MBPP sanitized/test 257 tasks.
This repo contains a LoRA adapter, not a merged full model.
The SFT data was trained on coding-agent traces, so HumanEval and MBPP mainly test general single-function Python generation. They do not directly measure tool-use behavior, multi-turn repair, or coding-agent workflow competence.

Agent-specific Held-out Trace Evaluation

Loss-only evaluation was run on Hugging Face Jobs with A100 large and non-4-bit inference:

Table with columns: Model, Held-out trace eval loss, Perplexity
Model	Held-out trace eval loss	Perplexity
Base Gemma 4 2B	1.4811	4.3978
Pi Trace SFT	0.7153	2.0448

Table with columns: Metric, Base, SFT
Metric	Base	SFT
Valid assistant/tool format	100.00%	100.00%
Tool-call DSL valid	42.58%	93.78%
Tool-call schema valid	41.63%	93.78%
Tool-name accuracy	25.36%	69.38%
Argument exact match	3.35%	13.88%
Argument token F1	25.46%

By action type:

Table with columns: Type, n, Base exact, SFT exact, Base token F1, SFT token F1
Type	n	Base exact	SFT exact	Base token F1	SFT token F1
command/test execution	134	2.99%	12.69%	38.65%	73.03%
tool call	51	3.92%	13.73%	30.49%	69.91%
code edit	25

gemma-4-E2B-it-pi-mono-lora

README

Base Model

Dataset

Selected Hyperparameters

Selection Evidence

Trackio

Prior Inspect AI Results

Standardized Local Evaluation

Known Evaluation Limitations

Agent-specific Held-out Trace Evaluation

Explore FriendliAI today

README

Base Model

Dataset

Selected Hyperparameters

Selection Evidence

Trackio

Prior Inspect AI Results

Standardized Local Evaluation

Known Evaluation Limitations

Agent-specific Held-out Trace Evaluation