jatshi

Audio-Codec-LLM-Qwen2.5-1.5B-GRPO-LoRA

Training data

Source: AISHELL-1 clean speech, with source URL, license, checksum, and seed recorded in the dataset provenance.
40,000 clean source files were sampled with seed 42.
Three deterministic degradation configurations were generated per source, yielding 120,000 versioned acoustic-evidence records.
Each SFT, DPO, and GRPO contract contains 114,000 training records and 6,000 held-out records.
200 noisy waveforms were materialized for reproducibility checks. The other rows are explicitly marked as clean-audio proxies paired with reproducible synthetic-degradation metadata; they are not represented as physically materialized noisy recordings.
DPO rejected responses are programmatically generated safety negatives. They are useful for controlled alignment experiments but are not a substitute for human preference labels.

Optimization

All stages ran on one NVIDIA RTX 4090 24 GiB in BF16 with LoRA rank 16 and DeepSpeed ZeRO-2 compatibility enabled at world_size=1.

Table with columns: Stage, Data/steps, Micro batch × accumulation, Objective
Stage	Data/steps	Micro batch × accumulation	Objective
SFT	114,000 rows, 1 epoch	8 × 2	token cross-entropy
cDPO	114,000 pairs, 1 epoch	8 × 2	`beta=0.01`, label smoothing `0.1`
GRPO	300 optimizer steps	4 × 4, 2 generations	five deterministic reward components

The initial DPO trial with beta=0.1 was saturated at step zero. Lowering beta to 0.01 restored a useful gradient, but standard DPO then drove rejected log-probabilities toward negative infinity. The final run therefore uses conservative DPO with label smoothing 0.1. Its finite optimum is log((1-0.1)/0.1) ≈ 2.20, preventing unbounded preference margins while retaining the full dataset and epoch.

Verified results

The completed SFT stage processed all 114,000 rows in 5,243.8 seconds:

Table with columns: Metric, Value
Metric	Value
train loss	0.09630
held-out loss	0.08168
held-out token accuracy	0.96517
training throughput	21.74 samples/s

The conservative DPO stage also processed all 114,000 pairs:

Table with columns: Metric, Value
Metric	Value
train loss	0.324913
held-out loss	0.324529
held-out pair accuracy	1.000000
held-out preference margin	2.188927
runtime	10,633.30 s
training throughput	10.721 samples/s

GRPO completed all 300 optimizer steps. Across those steps, mean reward was 0.946019 (range 0.796875–0.993750), mean reward standard deviation was 0.026765, and 181/300 steps had non-zero within-group reward variance. The stable resumed run took 4,247.43 seconds. Independent 2-second NVML sampling from checkpoint 100 onward observed 98% peak utilization, 24,067 MiB peak memory usage out of 24,564 MiB, and 214.74 W peak power.

The final comparison uses the exact same 200 examples, selected uniformly without replacement from the 6,000-row holdout with seed 42. The ordered sample ID list has SHA-256 50eaa2c3c59d1c5441757517fd9f9bc059f6b943ed55802b0e7fd82df8c75588.

Table with columns: Checkpoint, Valid JSON, Format, Diagnosis, Bounds, Consistency, Overprocessing, Total
Checkpoint	Valid JSON	Format	Diagnosis	Bounds	Consistency	Overprocessing	Total
Base	0.3550	0.2663	0.2222	0.0000	0.2675	0.3550	0.2222
SFT	1.0000	1.0000	1.0000

The tie among SFT, cDPO, and GRPO on this deterministic in-distribution slice is reported as observed; it is not presented as a GRPO gain. The later stages still provide independently auditable preference/RL training evidence, but a harder or human-reviewed test set is required to measure a quality difference. Machine-readable release summaries are included as docs/stage_matrix_4090.json and docs/grpo_run_summary_4090.json in the source repository.

Usage

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen2.5-1.5B-Instruct"
adapter_id = "jatshi/Audio-Codec-LLM-Qwen2.5-1.5B-GRPO-LoRA"

tokenizer = AutoTokenizer.from_pretrained(base_id, trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(base, adapter_id)
model.eval()

Use the system prompt and JSON contract in the source repository. The adapter expects text-form acoustic evidence; passing an audio file path alone is not sufficient.

Evaluation boundary

The final comparison uses the same 200 uniformly sampled held-out IDs for the base, SFT, DPO, and GRPO checkpoints and records the selection SHA-256.
These are in-distribution structured-prescription metrics, not PESQ, STOI, DNSMOS, or a listening test.
The 6,000-row split is a held-out split from the programmatically constructed dataset. It is not an independently collected real-world test corpus.
Single-GPU ZeRO results prove integration and memory/throughput behavior only. They do not establish multi-GPU scaling.

Limitations and safety

Prescriptions should be validated before controlling a production audio pipeline; unsafe parameter combinations can damage intelligibility.
Acoustic labels for proxy rows come from degradation configuration, not direct feature extraction from a materialized noisy waveform.
Synthetic preference pairs are regular and easier than human disagreements.
Generalization to unseen languages, microphones, rooms, and degradation mixtures has not been established.
The model can emit valid JSON that is acoustically inappropriate. Downstream parameter bounds and abstention policies remain necessary.

The source code is MIT licensed. The adapter remains subject to the base model and AISHELL-1 terms in addition to this repository's license.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

jatshi

Model Tree

Base

Qwen/Qwen2.5-1.5B-Instruct

Adapter

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Training data

Source: AISHELL-1 clean speech, with source URL, license, checksum, and seed recorded in the dataset provenance.
40,000 clean source files were sampled with seed 42.
Three deterministic degradation configurations were generated per source, yielding 120,000 versioned acoustic-evidence records.
Each SFT, DPO, and GRPO contract contains 114,000 training records and 6,000 held-out records.
200 noisy waveforms were materialized for reproducibility checks. The other rows are explicitly marked as clean-audio proxies paired with reproducible synthetic-degradation metadata; they are not represented as physically materialized noisy recordings.
DPO rejected responses are programmatically generated safety negatives. They are useful for controlled alignment experiments but are not a substitute for human preference labels.

Optimization

All stages ran on one NVIDIA RTX 4090 24 GiB in BF16 with LoRA rank 16 and DeepSpeed ZeRO-2 compatibility enabled at world_size=1.

Table with columns: Stage, Data/steps, Micro batch × accumulation, Objective
Stage	Data/steps	Micro batch × accumulation	Objective
SFT	114,000 rows, 1 epoch	8 × 2	token cross-entropy
cDPO	114,000 pairs, 1 epoch	8 × 2	`beta=0.01`, label smoothing `0.1`
GRPO	300 optimizer steps	4 × 4, 2 generations	five deterministic reward components

Verified results

The completed SFT stage processed all 114,000 rows in 5,243.8 seconds:

Table with columns: Metric, Value
Metric	Value
train loss	0.09630
held-out loss	0.08168
held-out token accuracy	0.96517
training throughput	21.74 samples/s

The conservative DPO stage also processed all 114,000 pairs:

Table with columns: Metric, Value
Metric	Value
train loss	0.324913
held-out loss	0.324529
held-out pair accuracy	1.000000
held-out preference margin	2.188927
runtime	10,633.30 s
training throughput	10.721 samples/s

Table with columns: Checkpoint, Valid JSON, Format, Diagnosis, Bounds, Consistency, Overprocessing, Total
Checkpoint	Valid JSON	Format	Diagnosis	Bounds	Consistency	Overprocessing	Total
Base	0.3550	0.2663	0.2222	0.0000	0.2675	0.3550	0.2222
SFT	1.0000	1.0000	1.0000

Usage

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen2.5-1.5B-Instruct"
adapter_id = "jatshi/Audio-Codec-LLM-Qwen2.5-1.5B-GRPO-LoRA"

tokenizer = AutoTokenizer.from_pretrained(base_id, trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(base, adapter_id)
model.eval()

Use the system prompt and JSON contract in the source repository. The adapter expects text-form acoustic evidence; passing an audio file path alone is not sufficient.

Evaluation boundary

The final comparison uses the same 200 uniformly sampled held-out IDs for the base, SFT, DPO, and GRPO checkpoints and records the selection SHA-256.
These are in-distribution structured-prescription metrics, not PESQ, STOI, DNSMOS, or a listening test.
The 6,000-row split is a held-out split from the programmatically constructed dataset. It is not an independently collected real-world test corpus.
Single-GPU ZeRO results prove integration and memory/throughput behavior only. They do not establish multi-GPU scaling.

Limitations and safety

Prescriptions should be validated before controlling a production audio pipeline; unsafe parameter combinations can damage intelligibility.
Acoustic labels for proxy rows come from degradation configuration, not direct feature extraction from a materialized noisy waveform.
Synthetic preference pairs are regular and easier than human disagreements.
Generalization to unseen languages, microphones, rooms, and degradation mixtures has not been established.
The model can emit valid JSON that is acoustically inappropriate. Downstream parameter bounds and abstention policies remain necessary.

The source code is MIT licensed. The adapter remains subject to the base model and AISHELL-1 terms in addition to this repository's license.

Audio-Codec-LLM-Qwen2.5-1.5B-GRPO-LoRA

README

Training data

Optimization

Verified results

Usage

Evaluation boundary

Limitations and safety

Explore FriendliAI today

README

Training data

Optimization

Verified results

Usage

Evaluation boundary

Limitations and safety