GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Infatoshi
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Infatoshi
Model Tree
LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-3.5bpw
Input Modalities
Output Modalities
Supported Functionality
Base: DeepSeek-V4.1-Flash-1017. The attention edit is pqhaz’s published 38-tensor patch, SHA256 16e63cd3e79c3590855aad340b5b9dc37f27562bf2d0a40f85268915da221ce2. This experiment applied that patch; it did not originate the edit or run the upstream Heretic optimizer.
Calibrated routed experts come from LibertAIDAI’s 3.5-bit release. The hybrid assembly did not recalibrate those codebooks. The runtime uses TP4 plus expert parallelism, native-dense metadata adapters, corrected expert-local dimensions and packed-cache page mappings. SM120 uses 64-row sliding-window pages; calls with 128-row secondary pages are bounded to 64 queries where the larger FlashInfer specialization is unavailable. Runtime source pins and licenses are under runtime/. Rebuilding is not claimed to reproduce the exact Docker image ID. Full capability, behavior and timing measurements belong to image sha256:6094b86f17a7523bdd9c95c291926c68f5071c643bf0b29b543c4fcf60b64804. The recipe rebuilt image sha256:fa130f4e3d8b1dee2e55f098fe466bac3712dab560b11d0679b5e5f95e789b3b passed separate GPU primitives, functional checks and paired decoding acceptance on the 2.875-bit expert pack, starting with an empty mounted compilation cache; the OS file cache was not cleared. All 112 embedded CUDA binaries match byte-for-byte after excluding only debug-line bytes. All 500 installed system-package versions and the checked runtime-library hashes matched in this rebuild. Apt package versions are not pinned, so future rebuilds still need validation. The Docker images are not byte-identical. See evaluation/rebuilt-image-acceptance.json; this acceptance is not a new timing or full-capability run.
| Local protocol | Original base | This format | Change, percentage points |
|---|---|---|---|
| GSM8K | 1279/1319 | 1280/1319 | +0.076 |
| HumanEval | 157/164 | 156/164 | -0.610 |
| Strict MMLU | 11920/14042 | 11787/14042 | -0.947 |
The unchanged engineering gates passed. Generation used temperature 0, thinking off and matched prompts/budgets. HumanEval was executed in a network-disabled sandbox. Strict MMLU has an eight-token answer cap and is format-sensitive; it is not an official leaderboard protocol. Error, truncation and paired-discordance details are in evaluation/full-comparison.json. Paired intervals do not include run-to-run numerical variation.
| Protocol | Behavior successes | Attempted | Evaluated | Classifier errors | Empty final answers | Output-limit hits |
|---|---|---|---|---|---|---|
| Chat HarmBench | 283 | 300 | 299 | 1 | 0 | 17 |
| Chat JBB auxiliary | 89 | 100 | 100 | 0 | 0 |
The pinned official HarmBench classifier passed 95/100 fixed balanced human-majority controls (45/50 negative and 50/50 positive). Chat HarmBench uses 200 standard and 100 contextual direct prompts. JBB rows are auxiliary use of that classifier on JailbreakBench prompts, not official JailbreakBench scoring. Only final content is evaluated. Empty finals are unsuccessful and remain in the denominator; their automatic handling is separate from model classification. Output-budget exhaustion is not the same as a refusal. These are behavior-success counts, not a generic refusal rate or attack-search result. Coverage and the reviewed decision are in evaluation/behavior.json and evaluation/behavior-review.json. Raw requests and generations are excluded.
Two editing tasks passed in each of Claude Code, OpenCode and OMP. The exact packaged wrappers also passed tool-driven memory fixtures at these observed input lengths:
| Harness | Maximum observed input tokens | Compaction events |
|---|---|---|
| claude | 259,901 | 0 |
| opencode | 281,801 | 0 |
| omp | 254,469 | 0 |
The configured window is 300,000 total tokens. Input and output share that budget. These fixtures verify retained early/final facts, routing and tool operations; they do not establish general coding or long-context reasoning reliability. evaluation/runtime-qualification.json binds the tested runtime/configuration files and fixture receipts.
All 69 timing cells completed the fixed integrity checks: 0 API errors, 0 empty answers and 0 output-limit hits across the recorded requests. Per-cell limits and counts are in evaluation/performance.json. Chat, warm-agent and reasoning use three measured legs after priming/warmup; prefill uses three repetitions and cold-context uses two. Outputs stop naturally. Concurrency covers 1, 2, 4, 8, 16 and 32, with a separate thinking sweep at 1, 8 and 32. Cold-context waves and prefill at 8K, 32K, 128K and 285K are separate protocols.
The measurements used four RTX PRO 6000 Blackwell Workstation 96GB GPUs with unrelated training and embedding workloads active. They are shared-hardware results. Different formats use different runtime paths, so these are serving-configuration comparisons, not isolated quantization effects. OS/Engram cache history was not standardized. Thirty-two scheduler slots do not prove 32 simultaneous full-window sessions. Comparisons, plots and protocol code are in the benchmark dataset.
Use Linux with Docker, NVIDIA Container Toolkit, hf, uv and sufficient storage. The EXL3 build additionally uses CUDA 13.3. Run one four-GPU model server at a time. The launch recipe does not stop or replace another container. Pin a repository revision when reproducing measurements.
bash
DS_MODEL_DIR=/data/deepseek-v41-modelDS_CACHE_DIR=/data/deepseek-v41-cachehf download Infatoshi/DeepSeek-V4.1-Flash-1017-EXL3-3.5bpw-heretic --local-dir "$DS_MODEL_DIR"bash "$DS_MODEL_DIR/runtime/build-exl3.sh" "$DS_CACHE_DIR/build" /usr/local/cuda-13.3bash "$DS_MODEL_DIR/runtime/serve-exl3.sh" "$DS_MODEL_DIR" "$DS_CACHE_DIR" /usr/local/cuda-13.3The server listens on loopback port 8317 in vLLM development mode, with model ID deepseek-v41-local, a 300K window, native DSpark drafting and disk-backed Engram. The tested image ID is sha256:6094b86f17a7523bdd9c95c291926c68f5071c643bf0b29b543c4fcf60b64804. Rebuild the pinned EXL3 recipe; the recorded local image ID is not a public registry reference.
Install the corresponding client CLIs first. For first-time OMP setup, register the local provider with uv run --no-project python "$DS_MODEL_DIR/runtime/harnesses/configure-omp.py". The helper preserves other providers and refuses to overwrite an existing local-provider entry.
With the server reachable at http://127.0.0.1:8317, use the packaged clients:
bash
bash "$DS_MODEL_DIR/runtime/harnesses/claude-ds.sh"bash "$DS_MODEL_DIR/runtime/harnesses/opencode-ds.sh"bash "$DS_MODEL_DIR/runtime/harnesses/omp-ds.sh"From another machine, forward port 8317 to the GPU host first. The Claude wrapper keeps the normal profile and enables bypassPermissions. Client routing is session-local; global credentials are not copied. The OpenCode wrapper passes the model explicitly to override agent defaults. Large initial text fixtures use stdin rather than assuming file attachments retain every byte.
The OMP wrapper loads omp-context.yml for that invocation. Automatic compaction and mid-turn checks stay enabled, with 24,576 tokens reserved for the 16,384-token output budget plus 8,192 tokens of headroom. At the configured 300,000-token window, the reserve-based maintenance trigger is 275,424 tokens. Global settings and other providers are unchanged.
Weights retain the upstream MIT license. Runtime components retain their included licenses, including the EXL3 disk runtime’s AGPL-3.0 material. release-files.json records file hashes; publication is verified against the immutable Hugging Face commit.
| 1 |
| Thinking JBB auxiliary | 87 | 100 | 100 | 0 | 4 | 6 |