What This Model Is
- Base model:
meta-llama/Llama-3.1-8B
- Architecture change: transformer MLP activations changed to ReLU
- Training method: staged LoRA fine-tuning
- Model format: merged dense model
- Extra metadata:
activation_config.json documents the activation modification
Intended Use
This model is intended for:
- research on activation functions in large language models
- experiments on sparse activations and efficiency tradeoffs
- evaluation and fine-tuning workflows built on top of ReLU-Tune
This is primarily a research release, not a production-optimized instruction model.
Files
Typical files in this repository include:
model.safetensorsconfig.jsongeneration_config.jsontokenizer.jsontokenizer_config.jsonactivation_config.json
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "bishmoy/Llama-3.1-8B-ReLU"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto",
)
Notes
activation_config.json is included for reproducibility and to document how activations were modified.- This release follows the base Llama 3.1 licensing and usage constraints. Users should review and comply with the original model terms.
Project
This model was produced with ReLU-Tune, a toolkit for:
- full or partial ReLU-fication
- staged LoRA fine-tuning
- benchmark and perplexity evaluation
- activation sparsity measurement
Limitations
This is an experimental model release. ReLU-fication changes the base architecture and can alter behavior in ways that differ from the original dense model. Downstream performance and robustness should be validated for any intended use.
