simple-llm-grpo-lora API & Inference Endpoint

Intended use

Use this experimental adapter for technical explanations, documentation, procedures, and runbooks. Verify generated code, commands, security guidance, and other high-impact content before use.

Training

The base model remained frozen while the existing rank-16 LoRA parameters were trained with GRPO. Training used bf16 on an NVIDIA L4.

Rewards considered correctness, semantic simplicity, clarity, and simplified-technical-English style. Incorrect answers were prevented from outranking correct answers based only on writing style.

Table with columns: Setting, Value
Setting	Value
Starting adapter	`thisisandreeeee/simple-llm-sft-lora`
Training method	GRPO
Precision	bf16
Epochs	1
Learning rate	`5e-6`
Generations per prompt	4
Generation temperature	`0.9`
Seed	42

Evaluation

Evaluation used 100 held-out English technical prompts.

Table with columns: Metric, SFT, GRPO
Metric	SFT	GRPO
Semantic simplicity	85.46%	95.83%
Technical adequacy	70.15%	72.42%
Task fulfilment	95.71%	94.33%
Clarity and coherence	92.09%	89.18%
Mean output tokens	622.3	331.2

GRPO produced much shorter and simpler answers with slightly higher technical adequacy, but task fulfilment and clarity declined. This comparison evaluates the complete inference configurations: SFT used adapter scale 0.25, while GRPO used scale 1.0.

Usage

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3.5-4B"
adapter_id = "thisisandreeeee/simple-llm-grpo-lora"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_id)

Reproducibility

Code: thisisandreeeee/simple-llm
Dataset: thisisandreeeee/simple-llm-sft
Training run: qwen35-4b-grpo-gated-20260912-02
Evaluation revision: 5038376f35146eb781bc4348cdeb0faca3e582f1

Intended use

Use this experimental adapter for technical explanations, documentation, procedures, and runbooks. Verify generated code, commands, security guidance, and other high-impact content before use.

Training

The base model remained frozen while the existing rank-16 LoRA parameters were trained with GRPO. Training used bf16 on an NVIDIA L4.

Rewards considered correctness, semantic simplicity, clarity, and simplified-technical-English style. Incorrect answers were prevented from outranking correct answers based only on writing style.

Table with columns: Setting, Value
Setting	Value
Starting adapter	`thisisandreeeee/simple-llm-sft-lora`
Training method	GRPO
Precision	bf16
Epochs	1
Learning rate	`5e-6`
Generations per prompt	4
Generation temperature	`0.9`
Seed	42

Evaluation

Evaluation used 100 held-out English technical prompts.

Table with columns: Metric, SFT, GRPO
Metric	SFT	GRPO
Semantic simplicity	85.46%	95.83%
Technical adequacy	70.15%	72.42%
Task fulfilment	95.71%	94.33%
Clarity and coherence	92.09%	89.18%
Mean output tokens	622.3	331.2

Usage

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3.5-4B"
adapter_id = "thisisandreeeee/simple-llm-grpo-lora"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_id)

Reproducibility

Code: thisisandreeeee/simple-llm
Dataset: thisisandreeeee/simple-llm-sft
Training run: qwen35-4b-grpo-gated-20260912-02
Evaluation revision: 5038376f35146eb781bc4348cdeb0faca3e582f1

simple-llm-grpo-lora

README

Intended use

Training

Evaluation

Usage

Reproducibility

Explore FriendliAI today

README

Intended use

Training

Evaluation

Usage

Reproducibility