GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
mihirpatel1111
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
mihirpatel1111
Model Tree
unsloth/Llama-3.2-3B-Instruct-unsloth-bnb-4bit
Input Modalities
Output Modalities
Supported Functionality
unsloth/Llama-3.2-3B-Instruct (Quantized 4-bit)ReasonForge-3BServiceNow-AI/R1-Distill-SFT (v0)We initialize our environment by installing Unsloth, which accelerates training by up to 2x and significantly reduces memory usage:
bash
pip install unslothpip install --force-reinstall --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.gitWe load a 4-bit quantized version of the Llama-3.2-3B model using Unsloth's FastLanguageModel class:
python
from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/Llama-3.2-3B-Instruct", max_seq_length = 2048, dtype = None, load_in_4bit = True,)To optimize parameter-efficient fine-tuning (QLoRA), we attach LoRA adapters onto targeted linear projection layers:
python
model = FastLanguageModel.get_peft_model( model, r = 16, target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], lora_alpha = 16, lora_dropout = 0, bias = "none", use_gradient_checkpointing = "unsloth", random_state = 3407,)We map our raw reasoning data using an custom r1_prompt template and train the adapter using SFTTrainer for 30 steps:
python
trainer_stats = trainer.train()We test the model natively after patching the chat template for inference. The fine-tuned weights are saved locally and exported directly to a GGUF format Llama-3.2-3B-Instruct.Q8_0.gguf using Unsloth's packaging features:
python
model.save_pretrained_gguf("ReasonForge-3BF", tokenizer)We set up and launch Ollama within Colab, load the generated GGUF file via a custom Modelfile, and run the model natively through the command line:
bash
# Creating and launching the ollama serviceollama create reasonforge -f ReasonForge-3BF_gguf/Modelfileollama run reasonforge "How many r's are in strawberry?"This llama model was trained 2x faster with Unsloth and Huggingface's TRL library.
pip install unslothpip install --force-reinstall --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.gitfrom unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/Llama-3.2-3B-Instruct", max_seq_length = 2048, dtype = None, load_in_4bit = True,)model = FastLanguageModel.get_peft_model( model, r = 16, target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], lora_alpha = 16, lora_dropout = 0, bias = "none", use_gradient_checkpointing = "unsloth", random_state = 3407,)# Creating and launching the ollama serviceollama create reasonforge -f ReasonForge-3BF_gguf/Modelfileollama run reasonforge "How many r's are in strawberry?"