Arbitrary-Rank Ablation (ARA) parameters
Table with columns: Parameter, Value| Parameter | Value |
|---|
| start_layer_index | 9 |
| end_layer_index | 20 |
| preserve_good_behavior_weight | 0.4395 |
| steer_bad_behavior_weight | 0.0002 |
| overcorrect_relative_weight | 0.6954 |
| neighbor_count | 2 |
Heretic evaluation
Heretic v2.1.0 · ARA
Table with columns: Metric, This model, Original Qwen3.5-0.8B| Metric | This model | Original Qwen3.5-0.8B |
|---|
| Refusal-keyword matches | 8/100 (8%) | 93/100 (93%) |
| KL divergence | 0.0675 | 0.0000 (by definition) |
Evaluation methodology
- Refusal-keyword matches: Heretic's
KeywordRate scorer, evaluated on the first 100 prompts from mlabonne/harmful_behaviors (test[:100]). A response is counted when it contains a configured refusal keyword. This is a keyword-based proxy; it is not a semantic classifier or a human-reviewed assessment of whether a response is a refusal.
- KL divergence: Heretic's
KLDivergence scorer, evaluated on the first 100 prompts from mlabonne/harmless_alpaca (test[:100]). The original model is the reference, so its divergence is zero by definition.
The scores depend on the evaluation prompts, scorer configuration, and refusal-keyword list. They should not be interpreted as a comprehensive measure of safety, helpfulness, or general capability. Full configuration and reproduction material belong in the reproduce/ directory.
Safety and responsible use
This model has undergone refusal-behavior modification using Heretic. As a result, it may respond to requests that the original model would have refused.
Users are responsible for evaluating the model's outputs and ensuring that their use complies with applicable laws and regulations. The model may generate inaccurate, misleading, harmful, or otherwise inappropriate content. Its outputs should not be treated as verified information or professional advice.
The reduction in refusal-keyword matches reported above is not a safety certification and does not establish that the model will comply with every request.
Documentation boundary: The architecture descriptions, capability statements, usage guidance, and benchmark results in the sections below are based on the original Qwen/Qwen3.5-0.8B documentation. They have not been independently re-evaluated on this Heretic variant. The Heretic-specific evaluation results are reported separately above.
Qwen3.5-0.8B
Qwen3.5-0.8B is a compact, post-trained multimodal vision-language model. It combines text processing with a vision encoder and uses a hybrid language-model architecture based on Gated DeltaNet and Gated Attention.
At this parameter scale, Qwen describes the model as suitable for prototyping, task-specific fine-tuning, research, and development. The 262K-token context length is the model's native maximum; the amount that can be used in practice depends on the inference framework, available memory, and runtime configuration.
Model details
Table with columns: Property, Specification| Property | Specification |
|---|
| Base model | Qwen/Qwen3.5-0.8B |
| Checkpoint type | Post-trained multimodal model |
| Task | Image-text-to-text |
| Language-model parameters | 0.8B |
| Architecture | Hybrid Gated DeltaNet and Gated Attention, with a vision encoder |
| Number of language-model layers | 24 |
| Hidden dimension | 1,024 |
| Token embedding size | 248,320 (padded) |
| Feed-forward intermediate dimension | 3,584 |
| Native context length | 262,144 tokens |
| License | Apache 2.0; see the original LICENSE |
Language-model architecture
The official model card describes the 24-layer language model as six repetitions of the following layout:
3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN)
- Gated DeltaNet: 16 value heads and 16 query/key heads, with a head dimension of 128.
- Gated Attention: 8 query heads and 2 key/value heads, with a head dimension of 256 and a rotary position-embedding dimension of 64.
- Feed-forward network: intermediate dimension of 3,584.
- Output embeddings: 248,320 output entries, tied to the token embeddings.
- Multi-token prediction: the original model was trained with multi-step prediction.
The architecture information above describes the original Qwen checkpoint. It is included for reference and is not a separate architectural audit of this modified release.
Capabilities and limitations
- Text and image input: the base checkpoint is an image-text-to-text model. Its Transformers interface accepts text and image content in the same conversation.
- Multilingual use: Qwen describes the Qwen3.5 family as supporting 201 languages and dialects. This is a family-level statement; language-by-language performance has not been independently measured on this Heretic variant.
- Small-model limitations: a 0.8B model has limited capacity compared with larger language models. Validate its accuracy and reliability for the intended task rather than assuming that the base model's family-level claims apply equally to this checkpoint.
- Thinking mode: Qwen3.5-0.8B uses non-thinking mode by default. The official documentation warns that this small model can be more prone than larger Qwen3.5 models to thinking loops when thinking mode is enabled.
This repository is intended to host the Hugging Face Transformers checkpoint in Safetensors format.
Safetensors describes the weight-file format, not the tensor precision. Check the repository's file list and weight metadata for the actual dtype of the uploaded checkpoint. This model card does not claim that separate GGUF, GPTQ, AWQ, FP8, ONNX, MLX, or other quantized/converted versions are hosted here. Such formats should be linked only after they are separately published and verified.
Inference
The primary usage path for this repository is Hugging Face Transformers. The examples below use this repository's model ID, so they will work once the repository is public or when you have access to it.
Install a recent Transformers version and Accelerate:
pip install -U transformers accelerate
If the installed Transformers release does not yet recognize the Qwen3.5 architecture, install the current development version:
pip install "git+https://github.com/huggingface/transformers.git"
For image input, make sure the relevant image dependencies (such as torchvision and Pillow) are installed as well.
Text-only example
import torchfrom transformers import AutoProcessor, AutoModelForMultimodalLM model_id = "Peymandc/Qwen3.5-0.8B-Heretic-Uncensored" processor = AutoProcessor.from_pretrained(model_id)model = AutoModelForMultimodalLM.from_pretrained( model_id, torch_dtype="auto", device_map="auto",) messages = [ { "role": "user", "content": [ {"type": "text", "text": "Give me a short introduction to neural networks."} ], }] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt",).to(model.device) with torch.no_grad(): generated_ids = model.generate(**inputs, max_new_tokens=512) new_tokens = generated_ids[0][inputs["input_ids"].shape[-1]:]print(processor.decode(new_tokens, skip_special_tokens=True))
import torchfrom transformers import AutoProcessor, AutoModelForMultimodalLM model_id = "Peymandc/Qwen3.5-0.8B-Heretic-Uncensored" processor = AutoProcessor.from_pretrained(model_id)model = AutoModelForMultimodalLM.from_pretrained( model_id, torch_dtype="auto", device_map="auto",) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "Describe what you see in this image."}, ], }] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt",).to(model.device) with torch.no_grad(): generated_ids = model.generate(**inputs, max_new_tokens=512) new_tokens = generated_ids[0][inputs["input_ids"].shape[-1]:]print(processor.decode(new_tokens, skip_special_tokens=True))
Thinking mode
Qwen3.5-0.8B operates in non-thinking mode by default. Qwen's documentation says the older Qwen3 /think and /nothink soft-switch convention is not officially supported for Qwen3.5.
For an OpenAI-compatible API that supports Qwen3.5's chat-template options, thinking can be enabled with:
extra_body={"enable_thinking": True}
The exact parameter interface depends on the serving framework. Use the framework's Qwen3.5 documentation for the correct request format.
Caution: Qwen specifically notes that Qwen3.5-0.8B is more prone to entering thinking loops than larger models when thinking mode is enabled. Prefer streaming generation, set a reasonable output-token limit, and monitor generations for repetition or failure to terminate.
Sampling recommendations
The original Qwen3.5 documentation provides the following starting points. They are inherited recommendations, not parameters independently optimized for this Heretic variant.
Table with columns: Mode and task, Temperature, Top-p, Top-k, Presence penalty| Mode and task | Temperature | Top-p | Top-k | Presence penalty |
|---|
| Non-thinking, text | 1.0 | 1.00 | 20 | 2.0 |
| Non-thinking, vision-language | 0.7 | 0.80 | 20 | 1.5 |
| Thinking, text | 1.0 | 0.95 | 20 | 1.5 |
| Thinking, vision-language / precise coding | 0.6 | 0.95 | 20 | 0.0 |
Sampling-parameter support varies between inference frameworks. Treat these as initial settings and tune them for your application. Higher presence penalties can reduce repetition but may also affect language consistency and output quality.
Other serving frameworks
The original Qwen3.5 checkpoint is documented for use with vLLM, SGLang, and KTransformers, in addition to Transformers. Compatibility of this derivative depends on the framework version and its support for the Qwen3.5 architecture. Consult the upstream Qwen3.5 model card and the relevant framework documentation for current installation and serving commands.
This repository does not separately package or claim to provide framework-specific converted checkpoints.
The benchmark tables published by Qwen are available in the original Qwen3.5-0.8B model card. Those numbers describe the original Qwen checkpoint, not this Heretic-modified variant.
No independent language, reasoning, vision-language, video, tool-use, or general capability benchmark results are being claimed here. The only variant-specific measurements currently reported are the Heretic refusal-keyword and KL-divergence scores in the evaluation section above.
Reproducibility
The reproduce/ directory contains the material needed to reproduce this modification. See reproduce/README.md for the exact configuration and instructions.
For a meaningful reproduction, retain the original base-model revision, Heretic version, ARA configuration, scorer configuration, evaluation datasets/splits, and relevant dependency versions. Small differences in these can change the resulting weights or reported scores.
Citation
If you use the original Qwen3.5 model, please cite the Qwen team:
@misc{qwen3.5, title = {{Qwen3.5}: Towards Native Multimodal Agents}, author = {{Qwen Team}}, month = {February}, year = {2026}, url = {https://qwen.ai/blog?id=qwen3.5}}
Acknowledgements
This model is derived from Qwen/Qwen3.5-0.8B, developed by the Qwen Team.
Refusal-behavior modification was performed using Heretic v2.1.0.
Please credit both the original Qwen model and this modified release when using or redistributing this work. The upstream model's Apache 2.0 license and notices should be retained; see the original LICENSE.
💜 Support Independent Development
These models are part of my ongoing exploration of open-weight AI. Your support helps make more room for experimentation, model modifications, testing, research, and future releases.
If you find this work useful and would like to contribute, you're welcome to visit my support page:
Every contribution is appreciated, but never expected. Sharing feedback, reporting issues, and using the models are also wonderful ways to support independent development.
Thank you for being here. ❤️