GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
moeshawky
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
moeshawky
Model Tree
google/gemma-4-12B
Input Modalities
Output Modalities
Supported Functionality
An 8-bit compressed-tensors quantization of
distributedcognition/humanizer-abliterated, produced with
llmcompressor.model_free_ptq directly from the BF16 safetensors. The result is an ordinary
Hugging Face model directory whose linear weights are stored in 8-bit with compressed-tensors
metadata in config.json, so it loads in vLLM, SGLang, or Transformers without a conversion step.
| Architecture | Gemma4UnifiedForConditionalGeneration |
| Parameters | 11.96B |
| Layers / hidden / vocab | 48 / 3840 / 262,144 |
| Source BF16 | 23.92 GB, 677 tensors |
| This artifact | 13.02 GB (54.4% of BF16) |
| Linear layers quantized | 328 |
| Weight format | INT8, per-output-channel scale, symmetric |
| Activation format | INT8, dynamic, per-token |
| Chat template | none — completion-style rewriter, see prompt_format.json |
Only linear layers inside the language-model decoder were quantized. Everything else stays BF16:
lm_head and model.embed_tokens — the input embedding and the output projection.re:.*vision.*, re:.*audio.*) — this is a text-only pipeline.Of 677 source tensors, 328 linear weights were quantized.
Keeping the embedding at BF16 is not cosmetic. tie_word_embeddings=True, so the embedding table is the output projection — quantizing it quantizes the logits. The failure mode, measured directly,
is silent: the weights load without error, the file is a plausible size, and the model then returns
the same handful of low-ID tokens no matter what you feed it.
Each quantized layer stores a per-output-channel scale in weight_scale.
bash
vllm serve moeshawky/humanizer-abliterated-INT8-W8A8bash
python -m sglang.launch_server --model-path moeshawky/humanizer-abliterated-INT8-W8A8python
from transformers import AutoModelForCausalLM, AutoTokenizer name = "moeshawky/humanizer-abliterated-INT8-W8A8"tok = AutoTokenizer.from_pretrained(name)model = AutoModelForCausalLM.from_pretrained(name, dtype="bfloat16", device_map="auto")Prompting. This is not a chat model. It ships prompt_format.json holding the instruction and separator it was tuned on: render instr + draft + sep, where sep is "\n\n### Rewritten:\n\n". Passing a chat-formatted prompt degrades output quality.
Measured:
lm_head, and the vision/audio towers confirmed BF16 by header inspection.config.json declares quant_method=compressed-tensors, format=int-quantized,
weights 8-bit / num_bits: 8 / channel strategy / symmetric, input activations 8-bit dynamic
per-token.Not measured — do not quote this repo for any of these:
| Item | Value |
|---|---|
| Source | distributedcognition/humanizer-abliterated (BF16 safetensors) |
| Base models | jialinyyzz/humanizer and google/gemma-4-12B |
| Tool | llmcompressor 0.14.0 / compressed-tensors 0.19.0, model_free_ptq (no calibration set) |
| Method |
model_free_ptq uses no calibration dataset: weight scales come from the weights themselves and
activations are quantized at runtime. The artifact is therefore dataset-independent.
Upstream is apache-2.0. That is this artifact's license, but the base models
(jialinyyzz/humanizer and google/gemma-4-12B) carry their own terms, including regional access limits on the Qwen line. If you
redistribute or serve this checkpoint, you inherit those obligations; read them before deploying.
| FP8 weight-only dynamic, weights E4M3 channel-wise symmetric, activations E4M3 dynamic per-token |
| Date | 2026-10-17 |