GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
Lathly
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Lathly
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
| File | Description |
|---|---|
adapter_model.safetensors | The LoRA adapter (PEFT format), ~304 MB |
adapter_config.json | PEFT config (r=16, α=32, all-linear) |
Qwen3.8-27B-Samantha-LoRA-qwen35.gguf | LoRA adapter in GGUF, arch-tagged qwen35 for llama.cpp |
tokenizer*.json, chat_template.jinja | Tokenizer + chat template |
A full merged model may be published separately. Additional GGUF quantizations are welcome from the community.
python
from peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizerbase = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.8-27B", torch_dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(base, "Lathly/Qwen3.8-27B-Samantha_Uncensored_1.1_LoRA")tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.8-27B")
You need a GGUF of the base model (e.g. a community Qwen3.8-27B Q5_K_M) plus this LoRA GGUF:
bash
llama-server \-m Qwen3.8-27B-Q5_K_M.gguf \--lora Qwen3.8-27B-Samantha-LoRA-qwen35.gguf \-ngl 99 --host 0.0.0.0 --port 8080
Qwen LoRA merge fix
llama.cpp branch: qwen-lora-vhead-fix" Commit: f839835a3 Description: convert : reorder V heads for LoraTorchTensor Required to successfully merge certain Qwen LoRA adapters.
Important: the adapter GGUF here is already tagged with
general.architecture = qwen35to match the base GGUF. If you regenerate the adapter yourself with an olderconvert_lora_to_gguf.py, it may tag asqwen3and llama.cpp will refuse to load it withmodel arch and LoRA arch mismatch. Retaggeneral.architecturetoqwen35to fix.
Qwen/Qwen3.8-27Bgghfez/Qwen3.8-27B-Q5_K_M-GGUFFine-tuned with Unsloth (QLoRA, 4-bit) on dual RTX 5070 Ti. Settings were chosen to closely match the reference Samantha training spec. The adapter was trained on text-only conversational data with a sequence length of 2048; this does not change the base model's supported context length, but Samantha-style behavior at long context has not been systematically evaluated:
| Parameter | Value |
|---|---|
| LoRA rank (r) | 16 |
| LoRA alpha | 32 |
| Target modules | all-linear (q, k, v, o, gate, up, down) |
| LoRA dropout | 0.05 |
| Bias | none |
| Task type | CAUSAL_LM |
| Optimizer | 8-bit Paged AdamW |
| Learning rate | 2e-5 |
| LR scheduler | linear |
Released under Apache-2.0, inheriting from both parents:
This is a modified derivative (LoRA fine-tune) of the above. No warranty. Outputs are the model's own; use responsibly.
This is an "uncensored" persona fine-tune intended for open conversational use. It may produce content that some find objectionable. You are responsible for how you use it and for compliance with applicable laws and the base-model/dataset licenses.
| Epochs | 1 |
| Effective batch size | 32 (grad accumulation) |
| Sequence length | 2048 |
| Warmup steps | 50 |
| Weight decay | 0.01 |
| Final train loss | ~1.42 |