GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
Lathly
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Lathly
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
| File | Description |
|---|---|
model-00001-of-00018.safetensors … model-00018-of-00018.safetensors | Merged model weights, 18 shards (~3.1 GB each, ~55.6 GB total) |
model.safetensors.index.json | Shard index |
config.json, generation_config.json | Model + generation config |
chat_template.jinja | Chat template |
tokenizer.json, tokenizer_config.json, vocab.json, merges.txt | Tokenizer |
preprocessor_config.json, video_preprocessor_config.json | Vision / video preprocessing |
LICENSE | Apache-2.0 |
python
from transformers import AutoModelForImageTextToText, AutoTokenizermodel = AutoModelForImageTextToText.from_pretrained("Lathly/Qwen3.8-27B-Samantha",torch_dtype="bfloat16",device_map="auto",)tok = AutoTokenizer.from_pretrained("Lathly/Qwen3.8-27B-Samantha")
This repo ships bf16 safetensors — the merge source. For llama.cpp you want a quantized
GGUF; the production one is Q5_K_M (~19 GB). Quantize from the bf16 weights with
convert_hf_to_gguf.py + llama-quantize, or grab the adapter-GGUF path from the
LoRA repo.
bash
llama-server \-m Qwen3.8-27B-Samantha-Q5_K_M.gguf \-ngl 99 --host 0.0.0.0 --port 8080
bash
vllm serve Lathly/Qwen3.8-27B-Samantha --dtype bfloat16
Samantha's persona was produced by a LoRA fine-tune of the base model, then merged — the LoRA is fully absorbed into these weights, so nothing is applied at runtime. Fine-tuned with Unsloth (QLoRA, 4-bit) on dual RTX 5070 Ti, with settings chosen to closely match the reference Samantha training spec. Trained on text-only conversational data at sequence length 2048; this does not change the base model's supported context length, but Samantha-style behavior at long context has not been systematically evaluated.
| Parameter | Value |
|---|---|
| LoRA rank (r) | 16 |
| LoRA alpha | 32 |
| Target modules | all-linear (q, k, v, o, gate, up, down) |
| LoRA dropout | 0.05 |
| Bias | none |
| Task type | CAUSAL_LM |
| Optimizer | 8-bit Paged AdamW |
| Learning rate | 2e-5 |
| LR scheduler | linear |
Released under Apache-2.0, inheriting from both parents:
This is a modified derivative (merged LoRA fine-tune) of the above. No warranty. Outputs are the model's own; use responsibly.
This is an "uncensored" persona fine-tune intended for open conversational use. It may produce content that some find objectionable. You are responsible for how you use it and for compliance with applicable laws and the base-model/dataset licenses.
| Epochs | 1 |
| Effective batch size | 32 (grad accumulation) |
| Sequence length | 2048 |
| Warmup steps | 50 |
| Weight decay | 0.01 |
| Final train loss | ~1.42 |