henil08

verilog-coder-7b-lora

README

License: apache-2.0

Training data

OpenRTLSet: 131k Verilog modules from GitHub, each paired with an LLM-written description. After dropping 54.6k fuzzer-generated modules, deduplicating, and capping examples at 2048 tokens, 53.7k remain; this run used 40,000 of them (~93% Verilog, ~7% SystemVerilog). About 30% of training prompts describe the ports in prose instead of giving the header as code, which keeps the model usable when no header is provided.

The raw corpus contains UVM testbench code in only 3 rows, and none of them survive filtering. This model was not trained or evaluated on UVM and makes no claim of UVM capability.

Training setup

Method: LoRA, 1 epoch, 29,906,052 supervised tokens
LoRA rank 64, alpha 128, dropout 0.05, targets: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
LR 0.0001 (cosine, 3% warmup), effective batch 64, max length 2048, bf16
HF Transformers + PEFT + DeepSpeed ZeRO-2 on 1× H100 (Modal); 79 min of training, ~$5.37
Loss on the code (assistant) tokens only; final eval loss 0.2653
Selected among LoRA/QLoRA/full-fine-tune configs by pass@1 on a validation split, so the test numbers below were not used for model selection.

Evaluation (pass@1, greedy decoding)

Table with columns: Benchmark, Base, Fine-tuned, Items gained / lost
Benchmark	Base	Fine-tuned	Items gained / lost
OpenRTLSet held-out (diff-sim)	33.2% (131/394, 95% CI 28.8–38.0)	46.5% (183/394, 95% CI 41.6–51.4)	+59 / -7 (McNemar p<0.001)
VerilogEval v2 spec-to-RTL (published prompts)	31.4% (49/156, 95% CI 24.6–39.1)	23.7% (37/156, 95% CI 17.7–31.0)	+10 / -22 (McNemar p=0.050)
VerilogEval v2 + interface code block	38.5% (60/156, 95% CI 31.2–46.3)	28.8% (45/156, 95% CI 22.3–36.4)	+9 / -24 (McNemar p=0.013)

OpenRTLSet held-out: 394 standalone modules not seen in training. A generated module passes if it matches the reference's ports and its outputs equal the reference's on every half-cycle of 500 cycles of random stimulus (Verilator). Random stimulus is evidence of equivalence, not proof.
VerilogEval v2: the official testbenches and Icarus Verilog v12 pass rule, with this project's chat prompt (scores are not directly comparable to leaderboard numbers that use other prompts).

Limitations

On VerilogEval this fine-tune scores below its base model (23.7% vs 31.4% with the published prompts). Fine-tuning on OpenRTLSet's real-world modules and LLM-written descriptions improved held-out OpenRTLSet modules but cost accuracy on VerilogEval's textbook-style problems; for that style of task the base model is the better choice.
Greedy decoding sometimes falls into repetition loops on long modules; about 13% of eval outputs hit the 2048-token cap.
Trained on single-module completions; multi-module designs and testbenches are out of scope. Not evaluated for synthesis quality, timing, or area.

Usage

python

from peft import PeftModelfrom transformers import AutoModelForCausalLMbase = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct", dtype="auto")model = PeftModel.from_pretrained(base, "henil08/verilog-coder-7b-lora")

Prompt format (system + user), as in training:

text

Specification:<description> Module interface:```verilogmodule ...(...);

text

undefined

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

henil08

Model Tree

Base

Qwen/Qwen2.5-Coder-7B-Instruct

Adapter

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Training data

The raw corpus contains UVM testbench code in only 3 rows, and none of them survive filtering. This model was not trained or evaluated on UVM and makes no claim of UVM capability.

Training setup

Method: LoRA, 1 epoch, 29,906,052 supervised tokens
LoRA rank 64, alpha 128, dropout 0.05, targets: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
LR 0.0001 (cosine, 3% warmup), effective batch 64, max length 2048, bf16
HF Transformers + PEFT + DeepSpeed ZeRO-2 on 1× H100 (Modal); 79 min of training, ~$5.37
Loss on the code (assistant) tokens only; final eval loss 0.2653
Selected among LoRA/QLoRA/full-fine-tune configs by pass@1 on a validation split, so the test numbers below were not used for model selection.

Evaluation (pass@1, greedy decoding)

Table with columns: Benchmark, Base, Fine-tuned, Items gained / lost
Benchmark	Base	Fine-tuned	Items gained / lost
OpenRTLSet held-out (diff-sim)	33.2% (131/394, 95% CI 28.8–38.0)	46.5% (183/394, 95% CI 41.6–51.4)	+59 / -7 (McNemar p<0.001)
VerilogEval v2 spec-to-RTL (published prompts)	31.4% (49/156, 95% CI 24.6–39.1)	23.7% (37/156, 95% CI 17.7–31.0)	+10 / -22 (McNemar p=0.050)
VerilogEval v2 + interface code block	38.5% (60/156, 95% CI 31.2–46.3)	28.8% (45/156, 95% CI 22.3–36.4)	+9 / -24 (McNemar p=0.013)

OpenRTLSet held-out: 394 standalone modules not seen in training. A generated module passes if it matches the reference's ports and its outputs equal the reference's on every half-cycle of 500 cycles of random stimulus (Verilator). Random stimulus is evidence of equivalence, not proof.
VerilogEval v2: the official testbenches and Icarus Verilog v12 pass rule, with this project's chat prompt (scores are not directly comparable to leaderboard numbers that use other prompts).

Limitations

On VerilogEval this fine-tune scores below its base model (23.7% vs 31.4% with the published prompts). Fine-tuning on OpenRTLSet's real-world modules and LLM-written descriptions improved held-out OpenRTLSet modules but cost accuracy on VerilogEval's textbook-style problems; for that style of task the base model is the better choice.
Greedy decoding sometimes falls into repetition loops on long modules; about 13% of eval outputs hit the 2048-token cap.
Trained on single-module completions; multi-module designs and testbenches are out of scope. Not evaluated for synthesis quality, timing, or area.

Usage

python

from peft import PeftModelfrom transformers import AutoModelForCausalLMbase = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct", dtype="auto")model = PeftModel.from_pretrained(base, "henil08/verilog-coder-7b-lora")

from peft import PeftModelfrom transformers import AutoModelForCausalLMbase = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct", dtype="auto")model = PeftModel.from_pretrained(base, "henil08/verilog-coder-7b-lora")

Prompt format (system + user), as in training:

text

Specification:<description>Module interface:```verilogmodule ...(...);

Specification:<description> Module interface:```verilogmodule ...(...);

text

undefined