Use with TensorRT Edge-LLM
Export or build with FP16 activations; this selects the NVFP4-A16 GEMM path (on Jetson Thor / SM110 the
Nvfp4A16BlackwellGemmPlugin):
trtedge export ... --no-quantize-activations # or the experimental builder CLI flag of the same name
Without that flag Edge-LLM runs the checkpoint as W4A4 (activations quantized to 4-bit as well), which
loses accuracy on our task.
Table with columns: Model, Cases, Tool + arguments correct| Model | Cases | Tool + arguments correct |
|---|
| This model, Edge-LLM, W4A16 | 60 | 78.3% |
| This model, Edge-LLM, W4A16 | 367 | 77.1% |
| Qwen3.5-9B unquantized, Edge-LLM | 60 | 76.7% |
| Qwen3.5-9B Q4_K_M GGUF, llama.cpp | 60 | 75.0% |
Model output alone, temperature 0, thinking off, guided decoding with the JSON schema, on an RTX PRO 6000
Blackwell. Valid JSON on 100% of cases.
Notes
- Edge-LLM 0.11.1's
tensorrt-edgellm-serve has no --no-quantize-activations option; for serving tests we
set FP16 activations by patching experimental.builder.core.quantization._activation_quant_type.
- Only the NVFP4 path above was validated. Edge-LLM 0.11.1 produced incorrect output for INT4 AWQ with group
size 32 or 64, INT4 mixed with FP8 layers, and NVFP4 with ModelOpt's
local_hessian scale search.