GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
andyx10
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
andyx10
Model Tree
Qwen/Qwen2.5-1.5B-Instruct
Input Modalities
Output Modalities
Supported Functionality
python
import torch, torch.nn.functional as F, yamlfrom huggingface_hub import hf_hub_downloadfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-av"meta = yaml.safe_load(open(hf_hub_download(repo, "nla_meta.yaml")))tok = AutoTokenizer.from_pretrained(repo)av = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16).cuda().eval() prompt = meta["prompt_templates"]["av"].format(injection_char=meta["tokens"]["injection_char"])ids = tok(prompt, add_special_tokens=False)["input_ids"]slot = ids.index(meta["tokens"]["injection_token_id"])emb = av.get_input_embeddings()(torch.tensor(ids).cuda()[None]).clone() act = ... # a raw layer-18 residual-stream activation, i.e. hidden_states[18], shape [1536]emb[:, slot] = F.normalize(act, dim=-1).to(emb.dtype) * meta["extraction"]["injection_scale"]out = av.generate(inputs_embeds=emb, attention_mask=torch.ones(emb.shape[:2], device=emb.device), max_new_tokens=32, do_sample=False)print(tok.decode(out[0], skip_special_tokens=True))Activations must come from hidden_states[18] of the base model (output of block 18, before the
final norm) and be passed raw — the injection step does the normalising and rescaling.