GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
maghrane
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
maghrane
Model Tree
Qwen/Qwen3.5-0.8B
Input Modalities
Output Modalities
Supported Functionality
Qwen/Qwen3.5-0.8B<s>[INST] {prompt} [/INST] {response} </s>| Parameter | Value |
|---|---|
| LoRA Rank (r) | 64 |
| LoRA Alpha (alpha) | 16 |
| LoRA Dropout | 0.1 |
| Bias | none |
| Target Modules | q_proj, k_proj, , , , , , , |
| Parameter | Value |
|---|---|
| Quantization | 4-bit (nf4) |
| Compute Dtype | float16 |
| Double Quantization | False |
| Parameter | Value |
|---|---|
| Optimizer | paged_adamw_32bit |
| Learning Rate | 2e-4 |
| Learning Rate Scheduler | constant |
| Weight Decay | 0.001 |
| Warmup Ratio | 0.03 |
| Epochs |
python
import torchfrom transformers import AutoTokenizer, AutoModelForCausalLMfrom peft import PeftModel # 1. Disable cuDNN algorithm search (resolves Qwen 3.5 1D convolution kernel lookup issue)torch.backends.cudnn.enabled = False base_model_name = "Qwen/Qwen3.5-0.8B"adapter_model_name = "maghrane/Qwen-0.8B-unsloth" # 2. Load Base Model & Adaptertokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)base_model = AutoModelForCausalLM.from_pretrained( base_model_name, torch_dtype=torch.float16, device_map="auto", trust_remote_code=True)model = PeftModel.from_pretrained(base_model, adapter_model_name) # 3. Re-enable KV Cache & set to evaluation modemodel.config.use_cache = Truemodel.eval() # 4. Generate Responseprompt = "What is Unsloth and what can it do?"formatted_prompt = f"<s>[INST] {prompt} [/INST]"inputs = tokenizer(formatted_prompt, return_tensors="pt").to(model.device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9, repetition_penalty=1.15, eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id ) generated_text = tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True)print(generated_text)
<s>[INST] What is Unsloth and what can it do? [/INST]
Unsloth lets you run and train AI models on your own local hardware via an open-source UI. It supports inference and training for 500+ models across text, audio, embedding, vision, and more. Key features include:
- Inference: Search, download, and run models (GGUFs, LoRA adapters, safetensors), self-healing tool calling, auto parameter tuning, model export, and side-by-side output comparison.
- Training: ~2x faster training with ~70% less VRAM, full/pre-training, 4-bit/16-bit/FP8 support, auto-dataset creation from PDF/CSV/DOCX, live observability, and efficient reinforcement learning (RL).
This model is fine-tuned for educational and documentation guidance purposes regarding local LLM fine-tuning and inference.
v_projo_projgate_projup_projdown_projfc1fc21| Per Device Batch Size | 4 |
| Gradient Accumulation | 1 |
| Max Gradient Norm | 0.3 |