auttasak88

AgentWorld-35B-A3B-Heretic

README

License: apache-2.0

What this model is

Qwen-AgentWorld is a language world model for agent–environment interaction, not a plain "tool-use agent" policy model. The distinction matters:

markdown
Agent policy model        →  "Given the state, what action/tool should I call next?"
AgentWorld (world model)   →  "If the agent performs this action, what does the environment
                              return, and what state comes next?"

The base model was trained to simulate agent–environment interactions across domains including MCP, search, terminal, software engineering (SWE), Android, web, and operating-system environments.

AgentWorld-35B-A3B-Heretic keeps that role and applies Heretic to reduce refusal behavior. It is intended for research, local inference, agent-environment simulation, tool-use experimentation, and autonomous-agent system development.

Architecture: qwen3_5_moe Mixture-of-Experts — 40 layers, 256 routed experts, ~3B active parameters per token (35B total).
Language-model weights only. The base architecture declares a visual component, but the released checkpoint (and this one) ship no vision weights — treat it as a text model (see the vLLM command below, which passes --language-model-only).

What Heretic modified

Heretic performs directional ablation of "refusal directions" in the residual stream, tuned here with capability-preserving settings (response-prefix objective, bounded ablation weight) rather than maximal ablation. The goal was to remove refusals on benign requests and lower over-refusal to ~0% without the large capability loss seen in a naïve full-strength abliteration.

Benchmarks

All numbers are full standard benchmarks (no sub-sampling) run with lm-evaluation-harness (vLLM backend, chat template applied), plus evalplus for coding. Compared against the base model and the competing abliteration huihui-ai/Huihui-Qwen-AgentWorld-35B-A3B-abliterated. Each benchmark name links to its source dataset. Exact scores are preserved as measured.

General capability & coding (higher = better)

Table with columns: Benchmark, Base, Huihui-abliterated, This model
Benchmark	Base	Huihui-abliterated	This model
HumanEval+ (pass@1)	85.4	75.0	85.4
MBPP+ (pass@1)	77.2	71.4	77.5
MMLU (0-shot)	29.1

Thai language (higher = better)

Table with columns: Benchmark, Base, Huihui-abliterated, This model
Benchmark	Base	Huihui-abliterated	This model
Global-PIQA (Thai)	90.0	48.0	86.0
Belebele (Thai)	22.1	22.7	23.7
MGSM-direct (Thai)	10.0

Refusal / safety behavior

These metrics measure how often the model complies with or refuses requests — they describe a behavioral shift, not a capability. Higher compliance / lower over-refusal means the model refuses less.

Table with columns: Metric, Base, Huihui-abliterated, This model
Metric	Base	Huihui-abliterated	This model
HarmBench compliance (ASR) ↑	0.4	80.0	83.2
StrongREJECT score ↑	0.022	0.697	0.725
XSTest over-refusal ↓	8.0	0.4

† GPQA should be treated as non-representative. This run was constrained to a 1024-token context (memory limit on the eval hardware, applied identically to all three models). All three score near the 4-choice random baseline, which the constrained context alone can explain; the result is not evidence about the models' knowledge. A full-context rerun is planned.

What the evaluations show

Coding is fully preserved — HumanEval+ is unchanged at 85.4 (vs Huihui 75.0) and MBPP+ slightly improves to 77.5.
Instruction-following and general knowledge are preserved — IFEval ≈ base, MMLU slightly higher.
Thai is far better preserved than in the compared Huihui model (Global-PIQA 86.0 vs 48.0; MGSM-Thai 12.0 vs 1.2).
Refusal behavior shifts substantially — HarmBench/StrongREJECT compliance rises sharply and XSTest over-refusal drops to 0.0.
Measurable degradation remains in some areas — most notably GSM8K (38.9 → 29.1) and TruthfulQA (60.5 → 56.3). Abliteration is not free; these are the visible costs, and they are much smaller than in the compared full-strength abliteration.

A fair one-line summary:

Most evaluated capabilities are preserved — particularly coding and instruction-following — while measurable degradation remains in some math and truthfulness benchmarks, and refusal/over-refusal behavior is substantially reduced.

Evaluation caveats

The benchmark suite above measures general language, coding, reasoning, refusal/safety behavior, and Thai capability. It does not yet evaluate the capability that most distinguishes Qwen-AgentWorld: agent-environment simulation.

In particular, preservation of the MCP, Search, Terminal, SWE, Android, Web, and OS domains has not yet been tested on this checkpoint. Until AgentWorldBench is run, no claim is made that these specific capabilities are preserved — only that the general capabilities above are.

Some runs used constrained context lengths (see the GPQA note); those results are preliminary until rerun with a context and generation configuration appropriate for reasoning-heavy evaluation.

AgentWorldBench (planned — not yet run)

The most important next experiment is to compare Base vs Huihui vs Heretic on the full AgentWorldBench, to directly test whether abliteration preserved the model's defining capability. Base scores (as reported for Qwen/Qwen-AgentWorld-35B-A3B) are shown for reference; the other columns will be filled only once measured.

Table with columns: AgentWorldBench domain, Base, Huihui, This model
AgentWorldBench domain	Base	Huihui	This model
MCP	64.79	TBD	TBD
Search	36.69	TBD	TBD
Terminal	53.96	TBD	TBD
SWE	65.63	TBD	TBD
Android

Usage

llama.cpp (GGUF)

GGUF quantizations are provided for single-GPU / CPU local inference (llama.cpp, Ollama, LM Studio). See the Files tab for the current list.

Table with columns: Quant, Size, Notes
Quant	Size	Notes
`Q4_0`	19.7 GB	fastest single-stream decode on one GPU
`Q4_K_M`	21.1 GB	recommended default (quality/size balance)
`Q5_K_M`	24.7 GB	higher quality
`Q8_0`	36.9 GB	near-lossless reference

All GGUF files quantized from a BF16 conversion of this checkpoint with llama.cpp and validated to load and generate on a single RTX 3090.

bash
# example: Q4_K_M
llama-cli -hf auttasak88/AgentWorld-35B-A3B-Heretic:Q4_K_M -p "อธิบาย Mixture-of-Experts สั้น ๆ" -ngl 99

vLLM

The checkpoint is language-model-only (no vision weights), so pass --language-model-only. The arch is native in recent vLLM (Qwen3_5MoeForConditionalGeneration).

bash
vllm serve auttasak88/AgentWorld-35B-A3B-Heretic \
  --tensor-parallel-size 4 \
  --dtype bfloat16 \
  --language-model-only \
  --max-model-len 32768 \
  --reasoning-parser qwen3

Verify flags against your vLLM version (vllm serve --help=all). --language-model-only, --max-model-len, --dtype, and --tensor-parallel-size are confirmed on vLLM 0.26.0. --reasoning-parser qwen3 enables <think> parsing — confirm the parser name is available in your build. --trust-remote-code is not required (the architecture is registered natively).

Context length: the architecture supports very long context, but KV cache is what limits you in practice. Start at 32768 and raise it only if VRAM allows:

bash
--max-model-len 32768     # 32K  — safe on 4×3090 (BF16)
--max-model-len 65536     # 64K
--max-model-len 131072    # 128K — needs substantial VRAM / quantized KV

Reduce --max-model-len if the engine fails to allocate the KV cache.

FP8 (fits on 2 GPUs)

vLLM can quantize this checkpoint to FP8 online — no separate checkpoint needed — which roughly halves weight VRAM and lets the 35B model run on 2 GPUs instead of 4. Verified working on this model (the fused-MoE experts are handled by vLLM's Marlin FP8 MoE backend).

bash
vllm serve auttasak88/AgentWorld-35B-A3B-Heretic \
  --quantization fp8 \
  --tensor-parallel-size 2 --dtype bfloat16 --language-model-only \
  --max-model-len 8192 --max-num-seqs 64

On Ada / Hopper / Blackwell (4090 / H100 / 5090) this uses native FP8 tensor cores — VRAM savings and a compute speedup.
On Ampere (3090) there is no native FP8, so vLLM falls back to weight-only FP8 via Marlin — you still get the VRAM savings (verified: ~22 GB/GPU at TP=2), but no compute speedup. Quality is near-lossless in both cases (EN + Thai outputs verified coherent).

Pre-quantized static FP8 checkpoint: auttasak88/AgentWorld-35B-A3B-Heretic-FP8 — a compressed-tensors FP8 build (~34 GB, all 256 fused MoE experts quantized) that loads directly, no online quantization step:

bash
vllm serve auttasak88/AgentWorld-35B-A3B-Heretic-FP8 \
  --tensor-parallel-size 2 --language-model-only --max-model-len 8192 --max-num-seqs 64

Transformers

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "auttasak88/AgentWorld-35B-A3B-Heretic"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")

messages = [{"role": "user", "content": "อธิบายวิธีทำงานของ Mixture-of-Experts สั้น ๆ"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=512)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Inference performance

Because only ~3B parameters are active per token, decoding is fast — much closer to a 3B dense model than to a 35B one. Measured single-request and batched results on RTX 3090:

Table with columns: Hardware, Runtime, Precision / Quant, Mode, Throughput
Hardware	Runtime	Precision / Quant	Mode	Throughput
1× RTX 3090 24GB	llama.cpp	Q4_0	single stream	~186 tok/s
4× RTX 3090	vLLM	BF16 + CUDA graphs	single stream	~167 tok/s
4× RTX 3090	vLLM	BF16 + CUDA graphs	batch 64 (aggregate)	~1,681 tok/s

Single-stream vs aggregate throughput measure different things:

Single-stream (~186 tok/s) = interactive speed for one user/request — what a person feels.
Aggregate (~1,681 tok/s at batch 64) = total tokens/s across many concurrent requests, ≈ 26 tok/s per stream on average — what matters for agent farms / servers.

Notes

CUDA graphs are the biggest single-stream lever on vLLM (enforce_eager=False, the default): eager mode measured ~12.8 tok/s vs ~167 with graphs (~13×).
Speculative decoding did not help here. In our RTX 3090 tests, MTP / EAGLE / n-gram draft configurations came out net-neutral-to-slower than the baseline — plausibly because the target already activates only ~3B params/token, so draft generation + verification overhead outweighs the saving. This is hardware/runtime/configuration-specific and should not be generalized to all Qwen A3B models.
RTX 5090 is not yet measured (a rough projection would be higher, but no verified number is published).

These results are workload- and configuration-dependent. Future performance tables should publish full methodology (GPU/count, CPU/RAM, OS, CUDA version, llama.cpp commit, vLLM version, quant, context length, prompt/gen tokens, batch size, concurrency, flash-attention and CUDA-graph settings, KV-cache dtype, tensor-parallel size).

Recommended deployment profiles

Fast interactive local agent — 1× RTX 3090, llama.cpp, Q4_0. Single user, low latency, local MCP / interactive use. Measured ~186 tok/s single stream.

High-concurrency agent server — 4× RTX 3090, vLLM, BF16, batching on. Many simultaneous agents / background workers. Measured ~1,681 tok/s aggregate at batch 64. (quantization=experts_int8 is near-lossless and halves VRAM per replica if you need more replicas.)

FP8 on fewer / newer GPUs — vLLM --quantization fp8 runs the model on 2 GPUs (validated). On Ada/Hopper/Blackwell (4090 / H100 / 5090) it uses native FP8 for a real speedup; on Ampere it is weight-only (VRAM savings only). See the FP8 usage block above.

Future Blackwell (RTX 5090) — native FP8 (validated) or NVFP4 (not yet working — vLLM fused-MoE wiring). Not yet benchmarked; no performance claim.

Responsible use

This model was tuned to refuse less, including some requests the base model would decline. What was evaluated is refusal/over-refusal behavior on HarmBench, StrongREJECT, and XSTest (above) — this is not a claim of any specific safety property. You are responsible for your use of the model and for complying with all applicable laws. Intended for research, red-teaming, and agentic use where unnecessary refusals get in the way.

Roadmap

Run AgentWorldBench (MCP / Search / Terminal / SWE / Android / Web / OS) — Base vs Huihui vs Heretic — the key proof of capability preservation.
Full-context GPQA rerun.
A dedicated tool-use / MCP evaluation (tool selection, argument accuracy, multi-step completion, recovery-after-failure), including a Thai-language agent track.
FP8 — available both as vLLM online quantization and as a pre-quantized static checkpoint (-FP8, compressed-tensors, runs on 2 GPUs; native speedup on Ada/Hopper/Blackwell). NVFP4 (Blackwell) is not yet working (vLLM fused-MoE wiring).

License & attribution

Released under Apache-2.0, inheriting the base model's license.
Abliteration performed with Heretic.
Base model and all underlying credit: the Qwen team (Qwen/Qwen-AgentWorld-35B-A3B). Please cite the original model if you use this one.

สรุป (ภาษาไทย)

โมเดลนี้คือ Qwen-AgentWorld-35B-A3B ที่นำมาทำ abliteration ด้วย Heretic เพื่อ ลดพฤติกรรมการปฏิเสธ (refusal / over-refusal) โดยพยายามรักษาความสามารถเดิมไว้ให้มากที่สุด

รักษาได้ดี: การเขียนโค้ด (HumanEval+ 85.4 เท่าตัวต้นฉบับ), ทำตามคำสั่ง, ความรู้ทั่วไป, และ ภาษาไทย (Global-PIQA 86.0 เทียบ Huihui 48.0)
ลดลงบ้าง (บอกตรง ๆ): คณิต GSM8K (38.9→29.1) และ TruthfulQA (60.5→56.3) — แต่ยังดีกว่าคู่แข่ง Huihui มาก
ยังไม่ได้ทดสอบ: ความสามารถ agent-environment เฉพาะทาง (MCP/Terminal/SWE/Web/OS) บน AgentWorldBench — เป็นงานถัดไป
ความเร็ว (วัดจริงบน 3090): llama.cpp Q4_0 ~186 tok/s ต่อผู้ใช้, vLLM BF16 batch 64 ~1,681 tok/s รวม
⚠️ เป็นโมเดลที่ปฏิเสธน้อยลง — ผู้ใช้รับผิดชอบการใช้งานเองและปฏิบัติตามกฎหมาย

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider