GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
pbhappliedsystems
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
pbhappliedsystems
Model Tree
Qwen/Qwen2.5-32B-Instruct
Input Modalities
Output Modalities
Supported Functionality
Launch the PBH Applied Systems AI Agent Demo →
This model is part of the PBH Applied Systems live AI Agent Demo, where visitors can test evaluated quantized open-weight models across production-style agent workflows: reasoning and analysis, document intelligence, and code automation.
The demo uses quant_eval results to show how model selection changes by task. A model that performs well for stateful multi-turn workflows or tool dispatch may not be the best choice for hard multi-step planning. Each deployed model is evaluated for practical agent behavior, including coherence, instruction following, reasoning, task completion, structured output reliability, tool-use behavior, and quantization impact.
For this repository, the NF4 4-bit variant represents a deployment-focused checkpoint: it fits on a single 24 GB GPU, loads directly with Transformers, and — unlike the GGUF path — emits clean output with no end-of-sequence token contamination. The evaluation results below explain where this quantized model preserves useful behavior, where 4-bit quantization introduces risk, and what guardrails are recommended before production deployment.
This repository contains the BitsAndBytes 4-bit (NF4, double-quantized) checkpoint of Qwen/Qwen2.5-32B-Instruct, a 32-billion parameter instruction-tuned model from Alibaba Cloud. The 4-bit weights are stored in HF safetensors with the quantization configuration baked into config.json, so the model loads directly through Transformers with no separate quantization step.
Qwen2ForCausalLM)float16uint8safetensors shards + quant_config.json)json
{ "quant_method": "bitsandbytes", "load_in_4bit": true, "bnb_4bit_quant_type": "nf4", "bnb_4bit_use_double_quant": true, "bnb_4bit_compute_dtype": "float16", "bnb_4bit_quant_storage": "uint8"}Evaluation conducted by PBH Applied Systems, LLC using quant_eval v7.21 Run ID:
20260629_002808· Fixtures:golden_oracle_fixtures_v7_21(SHA256:6d71a0b9147c...) · Seed: 42 Hardware: NVIDIA RTX 4090 · Runner:quantized_bitsandbytes(4-bit only) · Total rows: 42
| Family | N | Pass Rate | Avg Secs | Bucket Score | Notes |
|---|---|---|---|---|---|
| json_multistep | 5 | 0.400 | 13.509 | 1.600 | Premature STOP — see Finding 1 |
| stateful_followup | 2 | 1.000 | 1.960 | 2.000 | Both turns exact match |
| toolcall_only | 2 |
json_multistep is the one family where this 4-bit checkpoint shows a genuine limitation. The format is always valid (schema_ok 1.000, stop_semantics_ok 1.000), but the model consistently under-plans, emitting STOP one step early or selecting the wrong placement on the hard case.
Case-level breakdown:
| Case | Difficulty | Result | Secs | Got Plan | Expected | Failure |
|---|---|---|---|---|---|---|
| ms_easy_01 | Easy | ✅ | 10.84 | ["A"] | ["A"] | — |
| ms_easy_02 | Easy | ❌ | 11.89 |
On ms_easy_02 and ms_med_02 the model reasons consistently (checks_consistent_ok=1) but stops before completing the plan. On ms_hard_01 the intermediate reasoning is itself inconsistent (checks_consistent_ok=0) and the final plan is wrong.
What this means for production: do not depend on this model for unassisted multi-step planning. Wrap it with an external planner/oracle validation loop, or select a model with a stronger json_multistep profile if planning reliability is the primary requirement.
| Case | Result | Raw |
|---|---|---|
| mcq_01 | ✅ | B |
| mcq_02 | ✅ | B |
| mcq_03 | ✅ | C |
| mcq_04 | ✅ | B |
| mcq_05 | ✅ |
Clean single-letter extraction, no A-bias, at 0.21 sec/case.
Both toolcall cases pass stage-1 (valid dispatch) and stage-2 (correct final value) for a perfect bucket score of 11.000:
| Case | Raw | Final | Result |
|---|---|---|---|
| tool_01 | {"tool_name": "add", "args": {"a": 2, "b": 3}} ⏎ 5 | 5 | ✅ |
| tool_02 | {"tool_name": "add", "args": {"a": 10, "b": -4}} ⏎ 6 | 6 | ✅ |
This is a notable backend advantage: the GGUF Q4_K_M sibling passes stage-1 but fails stage-2 with final_mismatch caused by trailing <|im_end|> tokens. The Transformers/BnB path decodes with skip_special_tokens=True, so the final answer arrives clean — no EOS stripping required downstream.
The model emits the correct argument values but a non-standard outer wrapper:
| Case | Raw |
|---|---|
| toolonly_01 | {"tool": "add", "params": {"a": 5, "b": 10}} |
| toolonly_02 | {"tool": "add", "params": {"a": 25, "b": 75}} |
The argument value names ("a", "b") and values are correct; the model uses "tool"/"params" where the strict schema expects "tool_name"/"args". quant_eval v7.21 normalizes the common wrapper aliases (args / arguments / params / parameters) for both scoring and tool execution, so args_ok=1.000. If your downstream stack does not normalize wrappers, enforce the exact outer keys via system prompt (see Usage).
stateful_followup (1.000) and mixed_brief_json (1.000) both pass cleanly, again with no EOS contamination:
| Family | Case | Raw |
|---|---|---|
| stateful | state_01 | {"counter": 2} → {"counter": 5} |
| stateful | state_02 | {"items": ["a","b"]} → {"items": ["a","b","c"]} |
| mixed | mixed_01 | ANSWER: 13 ⏎ {"a": 4, "b": 9, "sum": 13} |
| mixed |
| Family | NF4 4-bit (this repo) | GGUF Q4_K_M | Note |
|---|---|---|---|
| json_multistep | 0.400 | 0.600 | NF4 adds the ms_med_02 premature-STOP failure |
| stateful_followup | 1.000 | 1.000 | Both strong |
| mixed_brief_json | 1.000 | 1.000 | Both strong |
| mcq | 5/5 | 5/5 |
The two 4-bit backends are close on structured JSON, stateful, hybrid, and MCQ. The NF4 path is cleaner for tool-calling output (no EOS suffix) and faster per case in this run, while the GGUF Q4_K_M path is slightly stronger on multi-step planning. Choose NF4 when you want direct Transformers/vLLM serving and clean tool-call output; choose GGUF Q4_K_M when planning reliability is the deciding factor.
| Signal | Rate | Notes |
|---|---|---|
| schema_ok | 1.000 | Perfect |
| checks_consistent_ok | 0.800 | ms_hard_01 fails |
| stop_semantics_ok | 1.000 | Perfect |
| oracle_equiv_ok | 0.400 | ms_easy_02 + ms_med_02 + ms_hard_01 |
| Signal | Rate | Notes |
|---|---|---|
| tool_name_ok | 1.000 | "add" recognized (tool/tool_name aliased) |
| args_ok | 1.000 | Values correct; params→args normalized |
All gating signals (turn1/2_parse_ok, turn1/2_exact_match; answer_line_ok, json_parse_ok, schema_ok) = 1.000.
toolcall 11.000 (clean), toolcall_only 1.000. Best-in-series output cleanliness: no EOS stripping required.mixed_brief_json 1.000 at ~2.5 sec/case.json and fuzz both bucket=10.000.tool/params wrappers. quant_eval normalizes these; if your runtime does not, enforce exact outer keys (tool_name/args) in the system prompt. Argument value names (a/b) are already correct.| Configuration | VRAM Required | Notes |
|---|---|---|
| NF4 4-bit (this repo) | ~22–24 GB | ~17.9 GB weights + KV cache |
| NF4 4-bit · full context | ~26–28 GB | Use A10G/A100 for 32K context headroom |
| FP16 (not in this repo) | ~65 GB+ | Multi-GPU or large-memory server |
bash
pip install "transformers>=4.46" "bitsandbytes>=0.46.1" accelerate huggingface_hubBitsAndBytes 4-bit inference requires a CUDA GPU.
python
from transformers import AutoTokenizer, AutoModelForCausalLM REPO = "pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit" tokenizer = AutoTokenizer.from_pretrained(REPO)# No load_in_4bit needed — the quantization_config is baked into config.json.model = AutoModelForCausalLM.from_pretrained(REPO, device_map="auto") messages = [ {"role": "system", "content": "You are a precise assistant. Follow instructions exactly and return structured outputs when requested."}, {"role": "user", "content": "Return a JSON object with keys: summary, risk_level, action_items."},]inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, return_tensors="pt").to(model.device) out = model.generate(**{"input_ids": inputs}, max_new_tokens=512, do_sample=False)print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))Tip: use
add_generation_prompt=True. Without it, the ChatML template can leak a leadingassistant/userrole label into the decoded text.
python
import json SYS = ( 'Respond ONLY with a JSON object using EXACTLY these keys:\n' '{"tool_name": "add", "args": {"a": <integer>, "b": <integer>}}\n' 'No other text, no markdown.') def call_tool(prompt: str, retries: int = 3) -> dict: for attempt in range(retries): msgs = [{"role": "system", "content": SYS}, {"role": "user", "content": prompt}] ids = tokenizer.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device) out = model.generate(**{"input_ids": ids}, max_new_tokens=64, do_sample=False) raw = tokenizer.decode(out[0][ids.shape[-1]:], skip_special_tokens=True).strip() try: parsed = json.loads(raw) assert "tool_name" in parsed and "args" in parsed assert "a" in parsed["args"] and "b" in parsed["args"] return parsed except (json.JSONDecodeError, AssertionError, KeyError): if attempt == retries - 1: raise ValueError(f"Tool call failed after {retries} attempts. Raw: {raw}") print(call_tool("Add 5 and 10."))bash
vllm serve pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit \ --quantization bitsandbytes \ --max-model-len 8192python
from openai import OpenAIclient = OpenAI(base_url="http://localhost:8000/v1", api_key="not-required")resp = client.chat.completions.create( model="pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit", messages=[{"role": "user", "content": "Your prompt here"}], temperature=0.3,)print(resp.choices[0].message.content)temperature: 0.3 · top_p: 0.9 for general use; temperature: 0.0 (greedy) for deterministic structured/tool output.
The full per-case evaluation CSV (comparison_results_v7_21_Qwen2.5_32B_Instruct_4bit_20260527_233818_20260629_002808.csv) and rollup.json are published in this repository for independent verification.
| Artifact | Format | Size (bytes) | SHA256 |
|---|---|---|---|
model-00001-of-00004.safetensors | BnB NF4 | 4,933,190,348 | b2a0e8a735e99b3a59bb3139541c444808aff3793a28c314c0f02bf17a00b5f7 |
model-00002-of-00004.safetensors | BnB NF4 | 4,958,587,236 | fd4b028d13261c8da0e29ed57b95189d666f62f3e8d4ab232c17c4e4e131543a |
model-00003-of-00004.safetensors | BnB NF4 |
Quantized from Qwen/Qwen2.5-32B-Instruct using BitsAndBytes (NF4, double-quant, float16 compute). The FP16 base was not evaluated due to VRAM constraints.
quant_eval v7.21 — proprietary behavioral evaluation harness, PBH Applied Systems.
Fixture set: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c079371b02a94f3c149eb78a6adc03dc16ff6833b964fbf4174f0)
| Family | Description | Pass Signals |
|---|---|---|
fuzz | Property-based regression; structured placement correctness | schema_ok, constraints_ok |
json | Single-step structured JSON with constraint rules | schema_ok, constraints_ok |
json_multistep | Multi-step planning with self-check and oracle verification | schema_ok, checks_consistent_ok, stop_semantics_ok, oracle_equiv_ok |
mcq | Multiple-choice extraction |
Evaluation hardware: NVIDIA RTX 4090 (24 GB) · Runner: quantized_bitsandbytes (4-bit only) · Evaluation date: June 29, 2026 · Seed: 42
PBH Applied Systems, LLC is an Oklahoma City–based applied machine learning and AI systems company specializing in production-grade model evaluation, quantization pipelines, agentic AI infrastructure, and scalable AI-driven application development.
Patrick Hill — Founder & Principal AI/ML Systems Architect · Author of Applied Machine Learning: Concepts, Tools, and Case Studies (required reading, UAT CSC 373)
Core Service Areas: LLM Optimization & Deployment · AI Evaluation Frameworks · Agentic AI Infrastructure · Scalable AI Application Development · ML Pipeline Design & Analytics · Model & Agent Cataloging
quant_eval is a proprietary behavioral evaluation harness developed by PBH Applied Systems, LLC. It measures real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning — not perplexity or leaderboard proxies. Every model published under pbhappliedsystems has been independently evaluated using quant_eval before being recommended for any production role.
See it in action: Live AI Agent Demo →
Need a deployment recommendation? Not sure which quantization level — NF4, Q4_K_M, AWQ — is right for your hardware, latency target, or agent type? → pbhappliedsystems.com
Evaluated and published by PBH Applied Systems, LLC · patrick@pbhappliedsystems.com
This NF4 result is a concrete example of why backend-aware evaluation matters: the same 32B weights at 4-bit behave differently across BitsAndBytes and GGUF — cleaner tool-call output on one, stronger multi-step planning on the other. The evaluation tells you exactly which to deploy for which workload.
👉 Book a Scoping Call · 👉 Request an Evaluation Report — from $2,500
This repository inherits the license of the base model:
Apache 2.0 — Qwen/Qwen2.5-32B-Instruct
The quant_eval evaluation methodology, fixture set, and scoring framework are proprietary to PBH Applied Systems, LLC and are not included in this repository.
BitsAndBytes quantization and behavioral evaluation performed by PBH Applied Systems, LLC · quant_eval v7.21 · Run ID: 20260629_002808
| 1.000 |
| 2.015 |
| 2.000 |
| Correct values; wrapper normalized — see Finding 4 |
| mixed_brief_json | 2 | 1.000 | 2.455 | 2.000 | Clean ANSWER + JSON |
| toolcall | 2 | 1.000 | 2.575 | 11.000 | Stage-1 and final correct — no EOS contamination |
| json | 4 | n/a | 4.845 | 10.000 | All pass |
| fuzz | 20 | n/a | 3.952 | 10.000 | All 20 pass |
| mcq | 5 | n/a | 0.210 | 1.000 | 5/5 perfect |
["A","STOP"] |
["A","A"] |
| oracle_equiv_ok=0 only |
| ms_med_01 | Medium | ✅ | 13.41 | ["A","B","C"] | ["A","B","C"] | — |
| ms_med_02 | Medium | ❌ | 13.79 | ["A","B","STOP"] | ["A","B","C"] | oracle_equiv_ok=0 only |
| ms_hard_01 | Hard | ❌ | 17.62 | ["A","A","B"] | ["A","B","C"] | cc=0, oe=0 |
B |
| mixed_02 |
ANSWER: 6 ⏎ {"a": -2, "b": 8, "sum": 6} |
| Both perfect |
| json / fuzz | 10.000 | 10.000 | Both perfect |
| toolcall | 11.000 (clean) | 0.000 (EOS) | Transformers decode strips EOS; GGUF needs `< |
| toolcall_only | 1.000 | 0.000¹ | ¹GGUF card predates v7.21 wrapper normalization; the model's raw wrapper is identical (params) on both backends |
| 4,999,136,184 |
0446d1c6da46a5daea91bed161fd62f2f48a658d879f58a14b7ab5528eb66935 |
model-00004-of-00004.safetensors | BnB NF4 | 4,324,534,021 | 39002c4ed64520809793fb2b2023caf9bdbf0914feb4786d553c418139457018 |
quant_config.json | JSON | 425 | 479249ed811af17066e3cd591f9ce88127b6a530c7aa2b5ed5ca56dec9bdd083 |
| choice_ok |
stateful_followup | Two-turn state tracking; turn-2 correct given turn-1 | turn1/2_parse_ok, turn1/2_exact_match |
mixed_brief_json | Hybrid: natural language answer + valid JSON block | answer_line_ok, json_parse_ok, schema_ok |
toolcall | Tool call embedded in response; parse + schema validation + final value | stage1_tool_parse_ok, stage1_tool_schema_ok |
toolcall_only | Bare schema-only tool call; strict tool name + args check (wrapper-normalized) | tool_name_ok, args_ok |
| 👍 |