GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
armzanaja
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
armzanaja
Model Tree
Qwen/Qwen2.5-72B-Instruct
Input Modalities
Output Modalities
Supported Functionality
<think>...</think> tags.The model was trained using the following configurations and hyperparameters:
Qwen/Qwen2.5-72B-Instruct-bnb-4bitGlint-Research/Fable-5-traces (Merged CoT and agent traces)q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj)adamw_8bitbfloat16 / float16 hybrid trackingpython
from unsloth import FastLanguageModelimport torch model, tokenizer = FastLanguageModel.from_pretrained( model_name = "armzanaja/qwen2.5-72b-fable5-lora", max_seq_length = 4096, load_in_4bit = True, # Set to False if you have enough VRAM for 16-bit)FastLanguageModel.for_inference(model) # Enable 2x faster inference # Example Promptmessages = [ {"role": "user", "content": "Create a python script to analyze system logs and detect anomalies."}] inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to("cuda")outputs = model.generate(inputs, max_new_tokens=2048, temperature=0.7) print(tokenizer.decode(outputs[0], skip_special_tokens=False))Since this repository contains only the lightweight LoRA adapter, you need to merge it with the base Qwen model before creating a GGUF file. You don't need an expensive GPU for this process—a standard machine with ~100GB of disk space and CPU setup can do it locally using llama.cpp:
bash
# 1. Clone llama.cpp repositorygit clone [https://github.com/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp)cd llama.cpppip install -r requirements.txt # 2. Download the official conversion script tool# 3. Run the conversion script by pointing to the Base Qwen model and adding your LoRA adapter pathpython convert_hf_to_gguf.py --outfile qwen2.5-72b-fable5-Q4_K_m.gguf --outtype q4_k_m --lora-base Qwen/Qwen2.5-72B-Instruct armzanaja/qwen2.5-72b-fable5-lorafrom unsloth import FastLanguageModelimport torch model, tokenizer = FastLanguageModel.from_pretrained( model_name = "armzanaja/qwen2.5-72b-fable5-lora", max_seq_length = 4096, load_in_4bit = True, # Set to False if you have enough VRAM for 16-bit)FastLanguageModel.for_inference(model) # Enable 2x faster inference # Example Promptmessages = [ {"role": "user", "content": "Create a python script to analyze system logs and detect anomalies."}] inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to("cuda")outputs = model.generate(inputs, max_new_tokens=2048, temperature=0.7) print(tokenizer.decode(outputs[0], skip_special_tokens=False))# 1. Clone llama.cpp repositorygit clone [https://github.com/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp)cd llama.cpppip install -r requirements.txt # 2. Download the official conversion script tool# 3. Run the conversion script by pointing to the Base Qwen model and adding your LoRA adapter pathpython convert_hf_to_gguf.py --outfile qwen2.5-72b-fable5-Q4_K_m.gguf --outtype q4_k_m --lora-base Qwen/Qwen2.5-72B-Instruct armzanaja/qwen2.5-72b-fable5-lora