GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
shreyansh12183
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
shreyansh12183
Model Tree
allenai/OLMo-2-1124-7B-Instruct
Input Modalities
Output Modalities
Supported Functionality
This model repository contains trained PEFT LoRA adapter weights fine-tuned on top of allenai/OLMo-2-1124-7B-Instruct.
To run inference in 4-bit quantization with minimal VRAM (~6GB on a free Google Colab T4 GPU):
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigfrom peft import PeftModel BASE_MODEL = "allenai/OLMo-2-1124-7B-Instruct"ADAPTER_ID = "shreyansh12183/olmo2-7b-astro-logic" # 1. Load Tokenizertokenizer = AutoTokenizer.from_pretrained(BASE_MODEL, trust_remote_code=True) # 2. Configure 4-bit BitsAndBytes NF4 Quantizationbnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16, bnb_4bit_use_double_quant=True) # 3. Load Base Modelbase_model = AutoModelForCausalLM.from_pretrained( BASE_MODEL, quantization_config=bnb_config, device_map="auto", trust_remote_code=True) # 4. Attach Fine-Tuned LoRA Adaptermodel = PeftModel.from_pretrained(base_model, ADAPTER_ID)model.eval() # 5. Run Test Inferenceprompt = "Explain the core domain methodology and statutory reasoning."inputs = tokenizer(prompt, return_tensors="pt").to("cuda")outputs = model.generate(**inputs, max_new_tokens=300, temperature=0.3, top_p=0.9)print(tokenizer.decode(outputs[0], skip_special_tokens=True))Test this model and the complete ExperimentLab SLM fleet on a free GPU with zero local setup:
shreyansh@experimentlab.in)import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigfrom peft import PeftModel BASE_MODEL = "allenai/OLMo-2-1124-7B-Instruct"ADAPTER_ID = "shreyansh12183/olmo2-7b-astro-logic" # 1. Load Tokenizertokenizer = AutoTokenizer.from_pretrained(BASE_MODEL, trust_remote_code=True) # 2. Configure 4-bit BitsAndBytes NF4 Quantizationbnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16, bnb_4bit_use_double_quant=True) # 3. Load Base Modelbase_model = AutoModelForCausalLM.from_pretrained( BASE_MODEL, quantization_config=bnb_config, device_map="auto", trust_remote_code=True) # 4. Attach Fine-Tuned LoRA Adaptermodel = PeftModel.from_pretrained(base_model, ADAPTER_ID)model.eval() # 5. Run Test Inferenceprompt = "Explain the core domain methodology and statutory reasoning."inputs = tokenizer(prompt, return_tensors="pt").to("cuda")outputs = model.generate(**inputs, max_new_tokens=300, temperature=0.3, top_p=0.9)print(tokenizer.decode(outputs[0], skip_special_tokens=True))