GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
littlemex
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
littlemex
Model Tree
Qwen/Qwen3.6-35B-A3B-FP8
Input Modalities
Output Modalities
Supported Functionality
prismyra.kernels.nvfp4 (reads the routed experts, runs them via vLLM's CUTLASS/FlashInfer NVFP4 kernels, leaves the
shared expert and router untouched) is wired into prismyra-serve as of v0.3.1: engine.py converts the routed
experts before the rest of the fast-kernel adapter runs, and the adapter (prismyra.kernels.qwen3_moe) recognises
the already-converted experts instead of re-wrapping them in FP8.
text
pip install "prismyra[server,fast] @ git+https://github.com/littlemex/Prismyra@v0.4.7"prismyra-serve --require-kernels starts
against these exact files (checksum-verified before upload; every fast kernel, including the NVFP4 routed experts,
applies) and answered two smoke-test questions correctly via the HTTP API; see Evaluation for accuracy on a larger
set.
/v1/decide, --batcher, and a determinism self-checkbash
PRISMYRA_EXPERTS=nvfp4 \PRISMYRA_NVFP4_EXPERTS=/path/to/nvfp4_experts.safetensors \PRISMYRA_NVFP4_CALIB=/path/to/nvfp4_calib.json \prismyra-serve --model /path/to/this/checkpoint/directory --require-kernels --batcherPOST /v1/decide answers through autotrust/JEV-27B's own request shape (kind/state/question/options)
instead of Prismyra's own /ask shape, for a caller already built against JEV's API. This is a server-level
feature, not specific to the NVFP4 experts this checkpoint ships. See
docs/DECIDE.md
for the full contract.
--batcher holds several callers' documents on the device at once (prismyra.schedule.Batcher), evicting
least-recently-used documents to make room for a new one. As of v0.4.4 that eviction accounts for page-level
fragmentation, not only a token-count sum, so a document the plain unbatched queue would answer is never refused
merely because the shelf's own eviction history happened to fragment its page pool.
A construction-time FP8 determinism self-check calls the installed FP8 kernel this checkpoint's non-expert
weights use (attention, gated-delta-net projections, the shared expert -- all FP8, see "Contents" above) on a
fixed input eight times in the process about to serve requests, and refuses to start if any call disagrees with
the first. This is a bounded probe, not a proof of determinism: eight agreeing calls make a disagreement observable
when it is frequent enough to show up in eight tries, and nothing stronger. On by default
(PRISMYRA_WITHOUT=fp8_determinism turns it off); confirmed passing on this checkpoint's own card (RTX PRO 4500),
recorded in engine.applied.notes. This check does not cover the NVFP4 routed experts: those have their own,
separate determinism mechanism, a per-card pinned tactic table described in
docs/KERNELS.md,
already in place before this release.
layers-*.safetensors, mtp.safetensors, outside.safetensors, config.json, tokenizer files: the checkpoint
with the 256-per-layer routed-expert tensors not present in any shard or in model.safetensors.index.json.nvfp4_experts.safetensors: the 256 routed experts of every layer, pre-converted to NVFP4 (packed 4-bit codes,
16-element block scales, one FP32 scale per expert matrix), in this project's own on-disk format
(prismyra.kernels.nvfp4.prepare_layer's output) -- not the compressed-tensors layout vLLM's own
AutoModel-style quantization loader expects. Unchanged from the previous revision (the routed experts carry no
adapter).nvfp4_calib.json: the per-layer activation maxima (measured on 128 training rows) that the NVFP4 forward pass
uses to scale its activations. Unchanged from the previous revision, for the same reason.bash
pip install "prismyra[server,fast] @ git+https://github.com/littlemex/Prismyra@v0.4.7"bash
PRISMYRA_EXPERTS=nvfp4 \PRISMYRA_NVFP4_EXPERTS=/path/to/nvfp4_experts.safetensors \PRISMYRA_NVFP4_CALIB=/path/to/nvfp4_calib.json \prismyra-serve --model /path/to/this/checkpoint/directory --require-kernelsThe checkpoint directory and the two side files are downloaded from this repository as a whole
(huggingface_hub.snapshot_download); PRISMYRA_NVFP4_EXPERTS and PRISMYRA_NVFP4_CALIB must point at the two
files once downloaded, since the loader reads them from a path, not from the checkpoint directory automatically.
The LoRA folded into this checkpoint's non-expert weights is, as of this revision, the same adapter the FP8 36-layer
checkpoint ships: rank 64, targeting attention and gated-delta-net projections and the shared expert, not the
routed experts (this checkpoint's NVFP4 conversion quantizes the routed experts from the un-adapted FP8 weights, so
the two touch disjoint weights and compose exactly for that reason). See that checkpoint's own card for the full
training recipe -- in short, a held-out panel of public benchmarks found this checkpoint family losing on sentiment
and causal/multi-step reasoning against a comparable general-purpose model, and this revision mixes four
licence-clear public datasets covering both into the training pool at a small, fixed fraction (56,026 rows total,
4.00% new; an earlier attempt at a larger fraction improved sentiment more but also regressed the project's
production-safety set, so it was not shipped). Nothing about the recipe is specific to this NVFP4 checkpoint -- the
same trained weights are simply folded into this checkpoint's FP8 non-expert shards instead of the FP8 36-layer
checkpoint's own. nvfp4_calib.json's activation maxima were measured on the un-adapted FP8 checkpoint, before any
LoRA existed, and were not re-measured for this adapter; the Evaluation results below were obtained with them as-is.
NVFP4 on the routed experts, before any LoRA (unchanged from the previous revision, since the routed experts are
unchanged), on a 1,400-question confirmation set (RACE-family, BoolQ-family, and a development split from a
companion reading-comprehension and contract-interpretation benchmark), measured with this project's own Python
engine directly, prior to the v0.3.1 prismyra-serve integration, on one RTX PRO 4500:
| comparison | point difference | 95% interval |
|---|---|---|
| NVFP4 experts vs. this project's own FP8 (36 layers) | −0.07 | [−0.86, +0.79] |
| NVFP4 experts (36 layers) vs. FP8 (32 layers, the largest FP8 checkpoint that fits one 32 GB RTX PRO 4500) | +1.86 | [+0.64, +3.00] |
Quantizing the experts to NVFP4 does not move this score outside noise against this project's own FP8 36-layer checkpoint, and is 1.86 points ahead of the FP8 checkpoint that would otherwise be the one to fit the same card. This measurement predates the v0.3.1 integration work and used a different serving path than the speed figures below; it is not a speed comparison and none is drawn from it here. It also predates this revision's adapter, so it does not cover NVFP4 vs. FP8 with this revision's adapter folded in -- that comparison has not been re-measured.
This exact combined checkpoint (NVFP4 experts and this revision's adapter), against the previous revision of
this repository (commit
e8a881e2),
measured with this project's own Python engine (prismyra.Prismyra) on one RTX PRO 4500.
The two judgment types this revision's new training rows target, 400 held-out questions each, neither used in training, sign test Holm-corrected across the two types:
| judgment type | wins / losses | sign-test p (Holm-corrected) | result |
|---|---|---|---|
| sentiment (400) | 23 / 4 | 0.0011 | significant gain |
| causal/multi-step reasoning (400) | 20 / 13 | 0.2963 | gain, not significant |
Fourteen further held-out and production item sets, checked for "did any item clearly regress": a development set, four production reading sets (RACE, BoolQ, and the same RACE questions buried in roughly 7,000 and 10,000 tokens of unrelated text), the project's production-safety transfer set (Kev, 764 questions), and eight further held-out judgment types this checkpoint ties the comparable general-purpose model on. None of the fourteen items' 95% confidence intervals against the previous revision fall entirely below zero; the production-safety set's own point estimate (-0.393) is better than the same comparison measured on the FP8 engine (-1.047), and two items (tool use, code) are significantly better. See the FP8 36-layer checkpoint's own card for the full fourteen-item table -- the same comparison, same previous-revision baseline, run identically on both serving engines.
Final confirmation, on two independently built, 10,000-question sets (72% production-style questions, 8% long-document questions, 20% contrast/adversarial questions each), each opened and scored exactly once, on this checkpoint's own NVFP4 serving path:
| confirmation set | n scorable | this revision | previous revision | point difference | 95% interval | result |
|---|---|---|---|---|---|---|
| set A (used to select this revision) | 10,000 | 88.77% | 88.76% | +0.010 | [-0.280, +0.280] | within ±0.5pt |
| set B (opened only after set A, one-time confirmation) | 9,998 | 89.20% | 89.16% | +0.040 |
Both sets land essentially tied with the previous revision, well inside the ±0.5-point non-inferiority bound used to confirm this revision before it shipped. These are the same two confirmation sets, scored on this exact serving path, used to select and confirm this revision in the first place (see the FP8 checkpoint's own card for the same sets' role in that revision's own release decision).
Speed: on the same RTX PRO 4500, with this project's own tuned FP8 kernel tables installed, one ~5,300-token
document (race150's articles concatenated to about 22,000 characters), 1/16/64 questions asked about it in a single
call, v0.4.0's defaults throughout (paged=False, interleaved_fork=True, wide_group=False): the same engine
with interleaved_fork toggled in place, 15 alternating rounds per width, the first 4 of each discarded as warm-up.
| 1 question | 16 questions | 64 questions | |
|---|---|---|---|
| previous revision (rank-16 adapter) | 274.4 ms | 321.3 ms | 488.1 ms |
| this revision | 272.2 ms | 321.1 ms | 486.6 ms |
The two revisions' medians differ by at most 0.8% at any width, well under the 5% difference this project requires before calling something a real speed change rather than measurement noise -- no measurable speed difference, consistent with a LoRA swap changing weight values and not this checkpoint's shape, depth, or kernel path.
Separately, and not re-measured under this methodology: the previous revision was compared, on the same card with the same tuned kernel tables, against the FP8 36-layer checkpoint cut to 32 layers (the largest FP8 checkpoint that fits the same 32 GB card). That checkpoint answered one question in 268 ms and sixteen in 393 ms, and could not complete the sixty-four-question measurement at all on this card (it ran out of device memory; this checkpoint did not). Those two FP8-32-layer numbers predate the measurement method the table above uses and have not been re-measured against it, so no comparison between this checkpoint and that one is drawn here.
All numbers in this section are this checkpoint's own, measured on the RTX PRO 4500 it is built for -- none of the FP8 36-layer checkpoint's L40S numbers are mixed in here; see that checkpoint's own card for its comparison against the same two models.
Accuracy against d1-3B, same gold labels, same 2,057 questions across the five sets this project's README describes, both models read through their own native API, on one RTX PRO 4500 (sign test per set, Holm-corrected across the five):
| set | this checkpoint vs. d1-3B (pt) | 95% interval | Holm-corrected p |
|---|---|---|---|
| RACE (579) | +6.22 | [+5.29, +7.14] | <0.0001 |
| BoolQ (400) | +4.75 | [+3.17, +6.33] | 0.0066 |
| RACE buried ~7k (157) | +14.65 | [+12.32, +16.98] | 0.0001 |
| RACE buried ~10k (157) | +15.29 | [+13.24, +17.34] | <0.0001 |
This checkpoint scores 91.20% pooled against d1-3B's 83.13%, significant on every one of the five sets after correction.
Probability quality (2,057 questions, lower is better for both): this checkpoint's Brier score is 0.137 against d1-3B's 0.234, and this checkpoint wins; this checkpoint's ECE (15 bins) is 0.044 against d1-3B's 0.024 -- d1-3B is better calibrated by that one measure, the same pattern the FP8 36-layer checkpoint shows against d1-3B.
Speed (one document held at 3,401 tokens, 1/16/64 questions in a single call, median of 7, RTX PRO 4500). This checkpoint's own figures are a same-day, same-GPU version-to-version measurement from 2026-10-10; d1-3B's figures are not from that session -- they are carried over from an earlier comparison round on the same card, reused because d1-3B's served checkpoint has not changed since:
| questions | this checkpoint | d1-3B (RTX PRO 4500, compiled) |
|---|---|---|
| 1 | 178.0 ms | 159.3 ms |
| 16 | 224.6 ms | 203.1 ms |
| 64 | 343.3 ms | 298.6 ms |
d1-3B is faster at every width measured: a 3B dense model with no routed experts has less to compute than a 35B-A3B MoE (even with its experts in 4-bit) at any width, which is the honest trade this checkpoint's accuracy numbers above are made against.
Determinism: d1-3B was measured bit-exact across five process restarts and every batch-companion pattern
tried. This checkpoint's own construction-time FP8 self-check (see "Serving at v0.4.7" above) targets one
hardware-level gap in the non-expert weights' own determinism story; the routed experts' determinism rests on the
separate pinned-tactic mechanism in docs/KERNELS.md. Neither is the same claim as d1-3B's empirical result above.
Memory: this checkpoint holds about 22.4 GiB on an RTX PRO 4500; d1-3B holds about 6 GiB on either card it was measured on, light enough to run several copies beside everything else on one GPU.
Against JEV-27B-VL: this checkpoint was not directly measured against JEV-27B-VL -- that comparison was run
against the FP8 36-layer checkpoint instead (see
that checkpoint's own card,
where it ties or beats JEV-27B-VL on accuracy on four of five sets and wins the fifth, and where JEV is faster at
one question but loses by a wide margin at sixteen and sixty-four). This checkpoint's own pooled accuracy on the
same five sets (91.20%, above) is within 0.2 percentage points of the FP8 36-layer checkpoint's (91.40%), the only
basis this card has for saying the FP8 comparison likely extends here; it has not been independently confirmed
against JEV-27B-VL on this checkpoint.
This project's other checkpoints are all FP8: 40 layers, 36 layers (recommended by default; the base this checkpoint's weights are built from, and the source of this revision's adapter), 32 layers, and a standalone LoRA for the full 40-layer base. This project also publishes a 32-layer NVFP4 checkpoint, which is faster but less accurate than this one; both NVFP4 checkpoints need a Blackwell GPU, and none of the FP8 checkpoints do.
If this revision's sentiment gain is not relevant to a caller's use case and the caller would rather pin the exact
weights measured in earlier editions of this card, the previous revision stays available unchanged at commit
e8a881e2121c49f4ef9554b5f2eaa139de6b4761:
python
from huggingface_hub import snapshot_downloadpath = snapshot_download( "littlemex/prismyra-decision-qwen3.6-35b-a3b-nvfp4-36l", revision="e8a881e2121c49f4ef9554b5f2eaa139de6b4761",)The two side files (nvfp4_experts.safetensors, nvfp4_calib.json) are unchanged across both revisions, so nothing
else needs to change to roll back.
Apache-2.0, the base model's licence, and this repository carries the base model's LICENSE file unchanged. Some of
the training sources behind this revision's adapter carry their own terms -- RACE, SciQ, DREAM and AG News are
distributed for non-commercial research use -- so check those terms before commercial use of this checkpoint, same
as for the FP8 36-layer checkpoint. (Earlier editions of this card named only RACE and SciQ; DREAM and AG News carry
the same restriction and were missed until the full source list was re-checked against each source's own licence
file.) The four datasets added in this revision are licence-clear for commercial use:
zeroshot/twitter-financial-news-sentiment (MIT), cardiffnlp/tweet_eval's sentiment config (CC BY 3.0), and
pkavumba/balanced-copa and allenai/social_i_qa (both CC BY 4.0, the second not independently re-confirmed against
a primary source -- see the link below), and they use their own gold labels throughout with no frontier-model
relabelling. About 7% of the pre-existing training rows take their gold label from a frontier LLM (Kimi K3, called
through its provider's API) rather than from the training pool's own label, used only as a single-letter answer
where it disagreed with the pool's; no generated rationale or free text from that model is reproduced in this
checkpoint's weights or in this card. No output of a closed, proprietary model was used anywhere in this pipeline.
Training data and licensing: every family in the training pool, its source, its licence, and the teacher models
used for gold-label correction and self-distillation, are listed with the licence text each was read from, in
docs/DATA_PROVENANCE.md.
| [-0.250, +0.330] |
| within ±0.5pt |
| Kev transfer-v4 (764) | +8.38 | [+7.03, +9.73] | <0.0001 |
| pooled, 2,057 | +8.07 | -- | -- |