harshagale

llm-upload

Deploy Dedicated

README

License: apache-2.0

Model Details

Developed by: Harsh Agale
Base Model: NousResearch/Llama-2-7b-chat-hf
Method: QLoRA (4-bit Quantization + LoRA)
Language(s): English
License: Apache 2.0
Task: Causal Language Modeling / Text Generation

Training Hyperparameters

The model was trained using the following configuration:

Quantization: 4-bit NormalFloat (nf4) with double quantization
Compute Dtype: float16
LoRA Rank (r): 8
LoRA Alpha: 16
Target Modules: q_proj, v_proj
LoRA Dropout: 0.05
Learning Rate: 2e-4
Optimizer: paged_adamw_8bit
Batch Size: 1 (with 4 Gradient Accumulation Steps)
Epochs: 1

Project Purpose

This project was created to learn and experiment with:

QLoRA fine-tuning
PEFT adapters
4-bit quantization
Efficient LLM training
Hugging Face ecosystem

Limitations

Trained on a small dataset
May produce hallucinated responses
Intended for educational and research purposes

How to Load and Use This Model

You can easily load this model and its adapters using the transformers and peft libraries:

python

import torchfrom transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfigfrom peft import PeftModel model_id = "NousResearch/Llama-2-7b-chat-hf"adapter_id = "harshagale/llm-upload" # 1. You must use the same 4-bit config to load the base modelbnb_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_quant_type="nf4",    bnb_4bit_compute_dtype=torch.float16,    bnb_4bit_use_double_quant=True) # 2. Load the base tokenizer and configure the padding tokentokenizer = AutoTokenizer.from_pretrained(model_id)tokenizer.pad_token = tokenizer.eos_token # 3. Load the quantized base modelbase_model = AutoModelForCausalLM.from_pretrained(    model_id,    quantization_config=bnb_config,    device_map="auto") # 4. Merge the PEFT adapter weights onto the base modelmodel = PeftModel.from_pretrained(base_model, adapter_id) # 5. Quick inference testprompt = "Human: Tell me a joke.\nAssistant:"inputs = tokenizer(prompt, return_tensors="pt").to("cuda") with torch.no_grad():    outputs = model.generate(**inputs, max_new_tokens=50) print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

harshagale

Model Tree

Base

NousResearch/Llama-2-7b-chat-hf

Adapter

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Developed by: Harsh Agale
Base Model: NousResearch/Llama-2-7b-chat-hf
Method: QLoRA (4-bit Quantization + LoRA)
Language(s): English
License: Apache 2.0
Task: Causal Language Modeling / Text Generation

Training Hyperparameters

The model was trained using the following configuration:

Quantization: 4-bit NormalFloat (nf4) with double quantization
Compute Dtype: float16
LoRA Rank (r): 8
LoRA Alpha: 16
Target Modules: q_proj, v_proj
LoRA Dropout: 0.05
Learning Rate: 2e-4
Optimizer: paged_adamw_8bit
Batch Size: 1 (with 4 Gradient Accumulation Steps)
Epochs: 1

Project Purpose

This project was created to learn and experiment with:

QLoRA fine-tuning
PEFT adapters
4-bit quantization
Efficient LLM training
Hugging Face ecosystem

Limitations

Trained on a small dataset
May produce hallucinated responses
Intended for educational and research purposes

How to Load and Use This Model

You can easily load this model and its adapters using the transformers and peft libraries:

python

import torchfrom transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfigfrom peft import PeftModel model_id = "NousResearch/Llama-2-7b-chat-hf"adapter_id = "harshagale/llm-upload" # 1. You must use the same 4-bit config to load the base modelbnb_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_quant_type="nf4",    bnb_4bit_compute_dtype=torch.float16,    bnb_4bit_use_double_quant=True) # 2. Load the base tokenizer and configure the padding tokentokenizer = AutoTokenizer.from_pretrained(model_id)tokenizer.pad_token = tokenizer.eos_token # 3. Load the quantized base modelbase_model = AutoModelForCausalLM.from_pretrained(    model_id,    quantization_config=bnb_config,    device_map="auto") # 4. Merge the PEFT adapter weights onto the base modelmodel = PeftModel.from_pretrained(base_model, adapter_id) # 5. Quick inference testprompt = "Human: Tell me a joke.\nAssistant:"inputs = tokenizer(prompt, return_tensors="pt").to("cuda") with torch.no_grad():    outputs = model.generate(**inputs, max_new_tokens=50) print(tokenizer.decode(outputs[0], skip_special_tokens=True))

import torchfrom transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfigfrom peft import PeftModel model_id = "NousResearch/Llama-2-7b-chat-hf"adapter_id = "harshagale/llm-upload" # 1. You must use the same 4-bit config to load the base modelbnb_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_quant_type="nf4",    bnb_4bit_compute_dtype=torch.float16,    bnb_4bit_use_double_quant=True) # 2. Load the base tokenizer and configure the padding tokentokenizer = AutoTokenizer.from_pretrained(model_id)tokenizer.pad_token = tokenizer.eos_token # 3. Load the quantized base modelbase_model = AutoModelForCausalLM.from_pretrained(    model_id,    quantization_config=bnb_config,    device_map="auto") # 4. Merge the PEFT adapter weights onto the base modelmodel = PeftModel.from_pretrained(base_model, adapter_id) # 5. Quick inference testprompt = "Human: Tell me a joke.\nAssistant:"inputs = tokenizer(prompt, return_tensors="pt").to("cuda") with torch.no_grad():    outputs = model.generate(**inputs, max_new_tokens=50) print(tokenizer.decode(outputs[0], skip_special_tokens=True))