Model Details
- Base model:
unsloth/Qwen3.5-9B (Qwen3.5-9B, 9.4B params, multimodal architecture, text decoder fine-tuned)
- Fine-tuning method: LoRA (r=16, alpha=16, dropout=0) via Unsloth
- Trainable parameters: 29,097,984 (0.31% of full model)
- Quantization: 4-bit (NF4) base loading
- Sequence length: 2048
- Optimizer: adamw_8bit
- Learning rate: 2e-4 with linear warmup + decay
- Gradient checkpointing: enabled (unsloth)
Training Data
A balanced mix of ~40k samples:
Training Procedure
- Steps: 1200
- Batch size: 2 per device, 4 gradient accumulation (effective 8)
- Hardware: AMD Radeon RX 7900 XTX (24GB VRAM) via ROCm/HIP
- Duration: ~4h45m
- Final train loss: ~0.65
Usage
This is a LoRA adapter. Load it on top of the base model with Unsloth or PEFT:
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/Qwen3.5-9B",
max_seq_length=2048,
dtype=None,
load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model.from_pretrained(model, "teolm30/ULT-2-max")
FastLanguageModel.for_inference(model)
Or with PEFT directly:
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("unsloth/Qwen3.5-9B", load_in_4bit=True)
model = PeftModel.from_pretrained(base, "teolm30/ULT-2-max")
tokenizer = AutoTokenizer.from_pretrained("unsloth/Qwen3.5-9B")
Limitations
- Fine-tuned primarily on English data
- Coding strength focused on Python
- Inherits any biases/limitations of the Qwen3.5-9B base
- Short context (2048) used during training; base supports longer
License
Apache-2.0 (inherited from Qwen3.5-9B).