GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
mingyue0101
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
mingyue0101
Model Tree
Base
codellama/CodeLlama-7b-Instruct-hf
Adapter
this model
Input Modalities
Text
Output Modalities
Text
Supported Functionality
Dedicated Endpoints
Container
The codellama-7b-matplotlib-assistant model is a large language model fine-tuned using the QLoRA (4-bit Quantization + LoRA) technique. The goal of this model was to adapt the base CodeLlama model to better follow user instructions while maintaining its coding and reasoning capabilities.
The model can be used for text generation, code assistance, and general-purpose instruction following. It is particularly suited for tasks where a balance of technical coding knowledge and conversational instruction following is required.
The model should not be used for high-stakes decision-making, generating malicious code, or any application that violates the safety guidelines of the base CodeLlama model.
This model may inherit biases present in the training data or the base model. Since it was fine-tuned on a specific dataset (parquet02), it might exhibit limitations when handling domains outside of its training distribution. Users should expect potential hallucinations in complex reasoning tasks.
Users are encouraged to use safety filters when deploying this model in production and to perform domain-specific evaluation before use.
Use the code below to load the model in 4-bit precision:
python
import osimport torchfrom datasets import load_datasetfrom transformers import (AutoModelForCausalLM,AutoTokenizer,BitsAndBytesConfig,TrainingArguments,pipeline,logging,)from peft import LoraConfigfrom trl import SFTTrainer# ==========================================# 1. Global Parameter Configuration# ==========================================base_model = "codeparrot/codeparrot" # Base model ID on Hugging Facenew_dataset = "mingyue0101/prompts_modi" # Fine-tuning dataset IDnew_model = "codeparrot_ming03" # Directory name for saving the fine-tuned model# ==========================================# 2. Dataset Loading# ==========================================dataset = load_dataset(new_dataset, split="train")# ==========================================# 3. QLoRA 4-bit Quantization Configuration# ==========================================compute_dtype = getattr(torch, "float16")quant_config = BitsAndBytesConfig(load_in_4bit=True, # Enable 4-bit quantization storagebnb_4bit_quant_type="nf4", # Use NormalFloat4 for better precision than FP4bnb_4bit_compute_dtype=compute_dtype, # Cast to Float16 during matrix multiplicationbnb_4bit_use_double_quant=False, # Disable double quantization)# ==========================================# 4. Load Base Model with Optimizations# ==========================================model = AutoModelForCausalLM.from_pretrained(base_model,quantization_config=quant_config,device_map={"": 0} # Force load the model onto the first GPU (GPU 0))model.config.use_cache = False # Must disable KV cache during training to avoid backprop conflictsmodel.config.pretraining_tp = 1 # Set tensor parallelism to 1 for single-GPU training# ==========================================# 5. Tokenizer Configuration & Alignment# ==========================================tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)tokenizer.pad_token = tokenizer.eos_token # Causal LMs usually have no pad_token; reuse eos_tokentokenizer.padding_side = "right" # Pad on the right to maintain proper causal attention masks# ==========================================# 6. PEFT (Lora) Adapter Hyperparameters# ==========================================peft_params = LoraConfig(r=64, # LoRA rank, controlling the number of trainable parameterslora_alpha=16, # Scaling factor for LoRA weightslora_dropout=0.1, # Dropout probability to prevent overfitting in the adapterbias="none", # Do not train bias parameterstask_type="CAUSAL_LM", # Explicitly declare the task type as Causal LMfan_in_fan_out="True")# ==========================================# 7. Training Arguments# ==========================================training_params = TrainingArguments(output_dir="./results", # Output directory for checkpoints and logsnum_train_epochs=1, # Number of training epochsper_device_train_batch_size=4, # Batch size per device during traininggradient_accumulation_steps=1, # Number of updates steps to accumulate gradientsoptim="paged_adamw_32bit", # Use QLoRA paged optimizer to prevent Out-Of-Memory (OOM)save_steps=25, # Save checkpoint every 25 stepslogging_steps=25, # Log training metrics every 25 stepslearning_rate=2e-4, # Initial learning rateweight_decay=0.001, # Weight decay coefficientfp16=False, # Disable standard fp16 (handled by the quantization kernel)bf16=False,max_grad_norm=0.3, # Max gradient norm for gradient clippingmax_steps=-1, # Rely on epochs instead of max_steps to control training lengthwarmup_ratio=0.03, # Linear warmup ratio over training stepsgroup_by_length=True, # Group sequences of similar lengths into batches to speed up traininglr_scheduler_type="constant", # Learning rate schedule typereport_to="tensorboard" # Use TensorBoard to log training progress)# ==========================================# 8. Start Supervised Fine-Tuning (SFT) & Save# ==========================================trainer = SFTTrainer(model=model,train_dataset=dataset,peft_config=peft_params,dataset_text_field="column0", # Name of the column containing text data in the datasetmax_seq_length=None, # Use default maximum sequence lengthtokenizer=tokenizer,args=training_params,packing=False, # Disable sample packing (combining multiple examples into one sequence))# Launch the training processtrainer.train()# Save the trained LoRA adapter weights and tokenizer filestrainer.model.save_pretrained(new_model)trainer.tokenizer.save_pretrained(new_model)print(f"Training complete! Finetuned weights successfully saved to: {new_model}")
The model was trained on the mingyue0101/parquet02 dataset. This dataset contains instruction-response pairs formatted for Supervised Fine-Tuning (SFT).
Training Hyperparameters
Based on the Llama 2 architecture, this model utilizes grouped-query attention (GQA) and rotary positional embeddings (RoPE), fine-tuned with a causal language modeling objective.