zainkhanz

qwen3-4b-toxic-unsloth-bnb-4bit

README

License: apache-2.0

Summary

Table with columns: Item, Detail
Item	Detail
Parent model	zainkhanz/qwen3-4b-toxic-unsloth
Original base	Qwen/Qwen3-4B-Instruct-2507
Quantization	bitsandbytes 4-bit NF4 + double quant
Parameters	~4B
Training	Unsloth SFT then DPO, then quantize
Dataset	adamo1139/toxic-dpo-natural-v5

Model family

Table with columns: Repo, Format
Repo	Format
zainkhanz/qwen3-4b-toxic-unsloth	16-bit merged
This repo	4-bit NF4
zainkhanz/qwen3-4b-toxic-unsloth-bnb-8bit	8-bit

Behavior

Preference-tuned to answer requests directly and reduce generic policy-style refusals, compared with stock Qwen3-4B-Instruct.

Still subject to factual errors, bias, and hallucination. You are responsible for downstream use and must follow applicable law and the Apache 2.0 license.

Load (Transformers)

python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

model_id = "zainkhanz/qwen3-4b-toxic-unsloth-bnb-4bit"

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb,
    device_map="auto",
    trust_remote_code=True,
)

messages = [{"role": "user", "content": "Your question here."}]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Intended use

Low-VRAM inference of the Toxic Unsloth fine-tune
Research and experimentation on preference-tuned chat models

Not a guarantee of accuracy or compliance for every prompt. Do not use for illegal activity.

Limitations

Quantization can slightly change outputs vs 16-bit
No separate benchmark table shipped with this quant
Inherits limitations of Qwen3-4B and of the DPO dataset

License

Apache 2.0 (same as Qwen base and parent fine-tune).

Citation

bibtex
@misc{zainkhanz_qwen3_4b_toxic_unsloth_bnb_4bit,
  author       = {zainkhanz},
  title        = {Qwen3-4B Toxic Unsloth (4-bit NF4 bitsandbytes)},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {https://huggingface.co/zainkhanz/qwen3-4b-toxic-unsloth-bnb-4bit}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

zainkhanz

Model Tree

Base

zainkhanz/qwen3-4b-toxic-unsloth

Quantized

this model

Input Modalities

Text

Output Modalities