What This Model Is
- Base model:
Qwen/Qwen3-1.7B
- Architecture change: transformer MLP activations changed to ReLU
- Modified layers: all decoder MLP activation sites
- Training method: staged LoRA fine-tuning
- Training schedule: 2 stages, 900 steps each, 1800 total steps
- Training data: continued pretraining on
tiiuae/falcon-refinedweb
- Model format: merged dense model
- Extra metadata:
activation_config.json documents the activation modification
Intended Use
This model is intended for:
- research on activation functions in large language models
- experiments on sparse activations and efficiency tradeoffs
- evaluation and fine-tuning workflows built on top of ReLU-Tune
This is primarily a research release, not a production-optimized instruction model.
Files
Typical files in this repository include:
model.safetensorsconfig.jsongeneration_config.jsontokenizer.jsontokenizer_config.jsonchat_template.jinjaactivation_config.json
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "bishmoy/Qwen3-1.7B-ReLU"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto",
)
Notes
activation_config.json is included for reproducibility and to document how activations were modified.- This release should follow the base Qwen model's license and usage terms. Users should review and comply with the original model terms before use or redistribution.
- Because this is a merged checkpoint, no adapter loading is required for standard inference.
Project
This model was produced with ReLU-Tune, a toolkit for:
- full or partial ReLU-fication
- staged LoRA fine-tuning
- benchmark and perplexity evaluation
- activation sparsity measurement
Limitations
This is an experimental model release. ReLU-fication changes the base architecture and can alter behavior in ways that differ from the original dense model. Downstream performance, calibration, and robustness should be validated for any intended use.
