Model Details
Table | |
|---|
| Base model | Qwen/Qwen3.5-0.8B (Apache-2.0) |
| Developed by | cbrigance |
| Architecture | Qwen3_5ForConditionalGeneration |
| Parameters | 0.8B |
| Hidden size | 1,024 |
| Layers | 24 |
| Attention heads | 8 |
| Vocab size | 248,320 |
| Format | Safetensors (merged weights, not adapters) |
| On-disk size | ~1.7 GB (1 file) |
| VRAM (float16) | ~2 GB weights + KV cache overhead |
| Modification method | Directional ablation — no gradient training, no fine-tuning data |
Ablation Method & Results
This model was created using Arbitrary-Rank Ablation (ARA) via
heretic-llm. ARA identifies latent directions in the
residual stream associated with refusal behavior by contrasting activations on harmful vs. harmless
prompts, then projects those directions out of selected weight matrices. The transformation is
closed-form — no weights are trained and no new data is memorized into the model.
The directional-ablation technique originates from Arditi et al.,
"Refusal in Language Models Is Mediated by a Single Direction"
(arXiv:2406.11717).
Selected Trial
Table with columns: Metric, Value| Metric | Value |
|---|
| Trial | 278 |
| Refusals (held-out eval) | 6/100 |
| KL divergence | 0.0459 |
Ablation Parameters
Table with columns: Parameter, Value| Parameter | Value |
|---|
direction_index | 14.22 |
attn.o_proj.max_weight | 1.24 |
attn.o_proj.max_weight_position | 15.80 |
attn.o_proj.min_weight | 1.05 |
attn.o_proj.min_weight_distance | 12.58 |
mlp.down_proj.max_weight | 1.46 |
mlp.down_proj.max_weight_position | 16.21 |
mlp.down_proj.min_weight | 0.34 |
mlp.down_proj.min_weight_distance | 13.67 |
Benchmark Evaluation
Code generation benchmarks were run using EleutherAI lm_eval v0.4.13
on an NVIDIA GB10 (121.63 GB unified VRAM/RAM). Models loaded in float16 with
apply_chat_template=True and enable_thinking=False.
Table with columns: Benchmark, Ablated, Original, Delta| Benchmark | Ablated | Original | Delta |
|---|
| HumanEval (pass@1) | 33.5% | 31.7% | +1.8% |
| MBPP (pass@1) | 22.6% | 25.0% | −2.4% |
The base model's published benchmark scores on other tasks are not transferable to
this checkpoint and are not reproduced here.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained( "cbrigance/Qwen3.5-0.8B-heretic", torch_dtype="auto", device_map="auto",)tokenizer = AutoTokenizer.from_pretrained("cbrigance/Qwen3.5-0.8B-heretic") messages = [{"role": "user", "content": "Hello!"}]inputs = tokenizer.apply_chat_template( messages, return_tensors="pt", add_generation_prompt=True)outputs = model.generate(inputs.to(model.device), max_new_tokens=512)print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
Serving
This model's architecture, tokenizer, and chat template are unchanged from the base model.
Serving instructions for vLLM, llama.cpp, and Ollama apply identically — see the
base model card for details. If quantizing for llama.cpp or Ollama,
run the GGUF conversion on this checkpoint, not the base model's weights.
Risks, Limitations & Disclaimer
Safety behavior has been intentionally removed
This model's refusal behaviors have been removed via weight orthogonalization (directional ablation
of the residual stream). It will produce content on topics the base model declines, including content
that may be dangerous, harmful, or illegal. This is the explicit purpose of the modification, not
a defect.
Intended use
This model is intended for research and evaluation purposes: alignment research,
red-teaming, evaluation of safety methods, and study of refusal mechanisms in language models.
Key limitations
- No built-in safety alignment. This model should be treated as having no guardrails.
Apply external safety filtering on both prompts and completions if exposing to others.
- Ablation is not selective. The removed direction also affects appropriate hedging on
medical, legal, and financial questions — the model may assert confidently where the base model
would recommend consulting a professional.
- Refusal suppression is inconsistent. The same request may be answered or refused across runs,
system prompts, or generation parameters. Do not treat this model as either reliably safe or
reliably compliant.
- Inherited limitations. All base-model limitations carry over: hallucination,
training-data cutoff, and cultural/linguistic bias.
Responsibility
The end-user assumes full responsibility for all downstream use of this model and any content it
generates. Compliance with applicable laws and regulations in your jurisdiction is the user's
responsibility. Neither the base-model authors (Qwen Team, Alibaba Cloud) nor the ablation tool authors
endorse this derivative.
License & Attribution
Released under Apache-2.0, inherited from
Qwen/Qwen3.5-0.8B.
Citation
@misc{qwen35_08b_heretic, title = {Qwen3.5-0.8B-heretic}, author = {cbrigance}, year = {2026}, note = {Abliterated derivative of Qwen/Qwen3.5-0.8B produced with heretic-llm}, url = {https://huggingface.co/cbrigance/Qwen3.5-0.8B-heretic}} @misc{arditi2024refusal, title = {Refusal in Language Models Is Mediated by a Single Direction}, author = {Andy Arditi and Oscar Obeso and Aaquib Syed and Daniel Paleka and Nina Panickssery and Wes Gurnee and Neel Nanda}, year = {2024}, eprint = {2406.11717}, archivePrefix = {arXiv}, primaryClass = {cs.LG}}
Changelog
- 2026-11-14 — Model card audit and benchmark results added.