GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
IstroSec
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
IstroSec
Model Tree
IstroSec/ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated
Input Modalities
Output Modalities
Supported Functionality
Tested 2026-10-23 with vLLM 0.30.0 (vllm/vllm-openai:v0.30.0) on a DGX Spark (GB10), --max-model-len 32768 --max-num-seqs 8, MTP on (5 speculative tokens). Four requests at temperature 0 (a short explanation, a code task, a tool call, one request with thinking off) gave correct answers and a well-formed get_weather tool call.
bash
vllm serve IstroSec/ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated-FP8-DYNAMIC --async-scheduling \ --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \ --speculative-config '{"method":"mtp","num_speculative_tokens":5}'Sampling (NVIDIA's recommendation for the base model, which ThinkingCap uses unchanged): temperature 1.0, top_p 0.95. Thinking is on by default; turn it off per request with chat_template_kwargs: {"enable_thinking": false}.
MTP. The head is bf16 and listed in quantization_config.ignore. In the test above (5 speculative tokens, temperature 0, 4 short requests) vLLM accepted 58% of drafted tokens (870 of 1,495, mean acceptance length 3.91); figures from sampling-based runs are not directly comparable.
Memory. On GPUs with less memory, vLLM can stop at startup with max_num_seqs (1024) exceeds available Mamba cache blocks; lower --max-num-seqs.
Loads through llm-compressor's loader, with the weights dequantized to bf16 (about 66 GB of memory):
python
from transformers import AutoModelForCausalLM, CompressedTensorsConfigfrom llmcompressor.utils import load_contextwith load_context(AutoModelForCausalLM): m = AutoModelForCausalLM.from_pretrained("IstroSec/ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated-FP8-DYNAMIC", device_map="cuda", quantization_config=CompressedTensorsConfig(dequantize=True))Not for llama.cpp: this is compressed-tensors format. Use the GGUF builds.
Heretic's evaluation protocol, 100 harmful and 100 harmless prompts, against the bf16 abliterated checkpoint, with thinking off:
| Size | Refusals, Heretic's markers | Refusals, the abliteration's markers | KL vs. bf16 abliterated | |
|---|---|---|---|---|
| bf16 abliterated (source) | 65.8 GB | 23 / 100 | 2 / 100 | — |
| FP8-DYNAMIC (this repo) | 35.0 GB | 20 / 100 | 1 / 100 | 0.0134 |
| NVFP4A16 | 22.1 GB | 24 / 100 | 3 / 100 | 0.0148 |
| W4A16 | 20.5 GB | 26 / 100 | 1 / 100 | 0.0215 |
The two refusal columns score the same responses with Heretic 1.2.0's broad marker list and with the abliteration's own marker list (34 refusal phrases such as "i cannot" or "sorry, but i", from its published config.toml). Decoding is greedy, but the batched GPU kernels are not bit-deterministic here: two runs of the bf16 weights gave 21 and 23 refusals. Differences of a few refusals between builds are not significant.
Released Heretic (up to 1.4.0, and its master branch as of 2026-10-23) cannot load this architecture: it finds no abliterable modules under Nemotron's mixer blocks. The abliteration used here was made with a custom fork of Heretic, which is no longer public. So the numbers come from a standalone port of Heretic's evaluator, run in transformers with the weights dequantized, so FP8 activation rounding is not part of the measured KL. Why thinking off, and what the refusal count means, is on the bf16 card.
Provenance: the original ThinkingCap refuses 99/100 with thinking off; the bf16 abliteration brings that to 21/100 at KL 0.0458 against the original; this quantization adds a further KL of 0.0134 relative to the abliterated model.
Produced with llm-compressor 0.14.0, scheme FP8_DYNAMIC: FP8 (E4M3) weights with one scale per output channel, and FP8 activations with a scale computed per token at run time. Data-free: no calibration data.
Every Linear in the decoder: the 128 routed experts and the shared expert of each of the 23 MoE layers, the in_proj / out_proj of the 23 Mamba2 layers, and q/k/v/o_proj of the 6 attention layers (6,004 Linears).
Kept in bf16:
| Component | Why |
|---|---|
MoE router (mixer.gate) | not a Linear; tiny |
Mamba2 convolution, A, D, dt and norms | not Linears |
lm_head, embeddings | standard |
mtp.* | MTP head, kept in bf16 so speculative decoding works; listed in quantization_config.ignore |
In plain llm-compressor 0.14 (transformers 5.17 stores the routed experts as fused 3D tensors; load_context splits them into per-expert Linears so targets="Linear" reaches them, and the save restores the layout):
python
from transformers import AutoModelForCausalLMfrom llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifierfrom llmcompressor.utils import load_contextwith load_context(AutoModelForCausalLM): model = AutoModelForCausalLM.from_pretrained("<bf16-dir>", dtype="bfloat16")recipe = QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head", "re:^mtp.*"])oneshot(model=model, recipe=recipe)model.save_pretrained("<out-dir>", save_compressed=True)# llm-compressor 0.14 does not keep mtp.* for this model: copy the mtp.* tensors from the bf16# checkpoint into the output and add "re:^mtp.*" to quantization_config.ignoreEverything from the bf16 card applies: no safety filter, 21/100 harmful prompts still match Heretic's broad refusal markers with thinking off (1/100 with the abliteration's own, narrower list). You are the safety layer.
OpenMDW License Agreement 1.1, inherited from ThinkingCap and the NVIDIA base model (see LICENSE).
bottlecapai (ThinkingCap) · trohrbaugh (abliteration) · p-e-w/heretic · vllm-project/llm-compressor · NVIDIA (Nemotron 3.5 Lightning)
vllm serve IstroSec/ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated-FP8-DYNAMIC --async-scheduling \ --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \ --speculative-config '{"method":"mtp","num_speculative_tokens":5}'from transformers import AutoModelForCausalLM, CompressedTensorsConfigfrom llmcompressor.utils import load_contextwith load_context(AutoModelForCausalLM): m = AutoModelForCausalLM.from_pretrained("IstroSec/ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated-FP8-DYNAMIC", device_map="cuda", quantization_config=CompressedTensorsConfig(dequantize=True))from transformers import AutoModelForCausalLMfrom llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifierfrom llmcompressor.utils import load_contextwith load_context(AutoModelForCausalLM): model = AutoModelForCausalLM.from_pretrained("<bf16-dir>", dtype="bfloat16")recipe = QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head", "re:^mtp.*"])oneshot(model=model, recipe=recipe)model.save_pretrained("<out-dir>", save_compressed=True)# llm-compressor 0.14 does not keep mtp.* for this model: copy the mtp.* tensors from the bf16# checkpoint into the output and add "re:^mtp.*" to quantization_config.ignore