GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
pcalhoun
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
pcalhoun
Model Tree
Input Modalities
Output Modalities
Supported Functionality
There are two ready-to-run formats:
| File | Format | Use |
|---|---|---|
model.safetensors (+ config.json, tokenizer) | fp16 safetensors | loads with a plain from_pretrained on modern transformers — no custom code, no 8-bit tricks |
ggml-model-q4_0.bin | ggml, 4-bit | CPU-only inference (~3.4 GB, ~4 GB RAM) via ggml's gpt-j |
eval.py | reference | the original 2023 inference script — documents the prompt schema |
The fp16 safetensors was produced from the original 8-bit training by
dequantizing the weights and folding the trained adapters into the dense weights
— it is numerically equivalent to the original (matching argmax and ≥0.999999
cosine on held-out probes). The original 8-bit .pt pickle has been removed in
favour of this safetensors (safe-format) equivalent.
This is a base model with no chat template. Its entire task interface is a set
of control tokens built from GPT-J's unused <|extratoken_N|> vocabulary. To
generate a pun unconditionally, prompt with:
markdown
<|extratoken_60|><|extratoken_61|> # TASK_START<|extratoken_46|><|extratoken_47|> # generate pun → explanation from keywords<|extratoken_10|><|extratoken_11|> # PUN begin<|extratoken_24|><|extratoken_25|> # GRAPHEMES (written form) begin
i.e. the concatenated string
<|extratoken_60|><|extratoken_61|><|extratoken_46|><|extratoken_47|><|extratoken_10|><|extratoken_11|><|extratoken_24|><|extratoken_25|>,
then sample. The pun is emitted as written text; the model also produces an
explanation (often as ARPABET phonemes — a quirk of how deeply the phonetic
training pervades it). Strip the <|extratoken_N|> tags from the output.
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerrepo = "pcalhoun/gpt-j-6b-8bit-pun-generator"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float16).cuda().eval()pre = ("<|extratoken_60|><|extratoken_61|>""<|extratoken_46|><|extratoken_47|>""<|extratoken_10|><|extratoken_11|>""<|extratoken_24|><|extratoken_25|>")ids = tok(pre, return_tensors="pt").to(model.device)out = model.generate(ids.input_ids, do_sample=True, temperature=0.9,top_k=50, top_p=0.98, max_new_tokens=128, pad_token_id=50256)text = tok.decode(out[0])import reprint(re.sub(r"<\|extratoken_\d+\|>", " ", text).strip())
Note: mainline llama.cpp does not support GPT-J — the architecture is only
implemented in the gpt-j example of the ggml
repo. Build that, and patch its tokenizer to treat the <|extratoken_N|> tags as
atomic special tokens (otherwise they get BPE-split and the model isn't
conditioned):
bash
git clone https://github.com/ggml-org/ggml && cd ggmlcmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build --target gpt-j -j# download ggml-model-q4_0.bin from this repo, then:./build/bin/gpt-j -m ggml-model-q4_0.bin -n 128 --top_k 50 --top_p 0.98 --temp 0.9 \-p '<|extratoken_60|><|extratoken_61|><|extratoken_46|><|extratoken_47|><|extratoken_10|><|extratoken_11|><|extratoken_24|><|extratoken_25|>'
Runs in ~3.4 GB on disk / ~4 GB RAM, CPU only.
Apache-2.0. The model was trained on data derived from public pun corpora; note that some of those corpora carry NonCommercial terms, so treat pun generation for redistribution accordingly.