linlu-lora-v0.1-qwen3.6-35b-a3b API & Inference Endpoint

Requirements

bash
pip install "transformers>=5.5" peft torch accelerate

~70 GB VRAM for bf16 inference (35B MoE base + LoRA on one GPU).

Quick start

python
import json
import re
import torch
import transformers
from peft import PeftModel
from transformers import AutoTokenizer

BASE = "Qwen/Qwen3.6-35B-A3B"
ADAPTER = "s-g-labs/linlu-lora-v0.1-qwen3.6-35b-a3b"

from transformers import AutoConfig
arch = AutoConfig.from_pretrained(BASE, trust_remote_code=True).architectures[0]
cls = getattr(transformers, arch)  # Qwen3_5MoeForConditionalGeneration
tok = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)
model = cls.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto",
                            trust_remote_code=True)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

n = sum(1 for k, m in model.named_modules()
        if k.endswith("lora_B.default") and m.weight.abs().max() > 0)
assert n > 0, "adapter not loaded"

system = """你将扮演「林路」，正在用中文和「对方」聊天。像真人发微信，可一次发多条短消息。
【人物能力与爱好】文学、网球、羽毛球；不擅长数学、做饭。
【当前亲密度】30
"""

def chatml(system, history):
    s = f"<|im_start|>system\n{system}<|im_end|>\n"
    for role, content in history:
        s += f"<|im_start|>{role}\n{content}<|im_end|>\n"
    return s + "<|im_start|>assistant\n"

ids = tok(chatml(system, [("user", "周末一般干嘛？")]), return_tensors="pt").to(model.device)
eos = [tok.eos_token_id, tok.convert_tokens_to_ids("<|im_end|>")]
with torch.no_grad():
    out = model.generate(**ids, max_new_tokens=300, do_sample=True,
                         temperature=0.8, top_p=0.95, repetition_penalty=1.05,
                         eos_token_id=eos, pad_token_id=tok.pad_token_id or eos[0])
raw = tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True)
bubbles = [ln for ln in raw.split("\n") if ln.strip()]
print(bubbles)

Prompt & output format

Same ChatML template as v0.2 — see the v0.2 README for full documentation (persona fields, cold-open cue, multi-turn, sampling table).

Do not use AutoModelForCausalLM — use Qwen3_5MoeForConditionalGeneration.

Cold-open user cue (optional):

markdown
（系统提示：新的一天开始了，请像往常一样主动给对方发消息，开启今天的聊天。）

LLaMA-Factory

bash
llamafactory-cli chat \
  --model_name_or_path ./Qwen3.6-35B-A3B \
  --adapter_name_or_path s-g-labs/linlu-lora-v0.1-qwen3.6-35b-a3b \
  --template qwen --trust_remote_code

python

import json
import re
import torch
import transformers
from peft import PeftModel
from transformers import AutoTokenizer

BASE = "Qwen/Qwen3.6-35B-A3B"
ADAPTER = "s-g-labs/linlu-lora-v0.1-qwen3.6-35b-a3b"

from transformers import AutoConfig
arch = AutoConfig.from_pretrained(BASE, trust_remote_code=True).architectures[0]
cls = getattr(transformers, arch)  # Qwen3_5MoeForConditionalGeneration
tok = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)
model = cls.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto",
                            trust_remote_code=True)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

n = sum(1 for k, m in model.named_modules()
        if k.endswith("lora_B.default") and m.weight.abs().max() > 0)
assert n > 0, "adapter not loaded"

system = """你将扮演「林路」，正在用中文和「对方」聊天。像真人发微信，可一次发多条短消息。
【人物能力与爱好】文学、网球、羽毛球；不擅长数学、做饭。
【当前亲密度】30
"""

def chatml(system, history):
    s = f"<|im_start|>system\n{system}<|im_end|>\n"
    for role, content in history:
        s += f"<|im_start|>{role}\n{content}<|im_end|>\n"
    return s + "<|im_start|>assistant\n"

ids = tok(chatml(system, [("user", "周末一般干嘛？")]), return_tensors="pt").to(model.device)
eos = [tok.eos_token_id, tok.convert_tokens_to_ids("<|im_end|>")]
with torch.no_grad():
    out = model.generate(**ids, max_new_tokens=300, do_sample=True,
                         temperature=0.8, top_p=0.95, repetition_penalty=1.05,
                         eos_token_id=eos, pad_token_id=tok.pad_token_id or eos[0])
raw = tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True)
bubbles = [ln for ln in raw.split("\n") if ln.strip()]
print(bubbles)

linlu-lora-v0.1-qwen3.6-35b-a3b

README

Requirements

Quick start

Prompt & output format

LLaMA-Factory

Explore FriendliAI today

README

Requirements

Quick start

Prompt & output format

LLaMA-Factory