Quantization
Table | |
|---|
| format | compressed-tensors, FP8_DYNAMIC, made with llm-compressor 0.15 |
| weights | FP8 (E4M3), one scale per output channel |
| activations | FP8, scaled per token at runtime (no calibration data) |
| quantized | all 400 Linear layers of the language model: attention and linear-attention projections, MLP |
| kept in bf16 | lm_head, embeddings, all norms, the linear-attention gate projections in_proj_a / in_proj_b, the vision tower |
The bf16 tensors are bit-identical to the bf16 model. The recipe is in recipe.yaml.
As in the bf16 model, the vision tower weights are kept so that the checkpoint loads with the base model's architecture,
but the model was trained and evaluated on text only.
How to use it
Serving with vLLM
vllm serve dpavlis/Qwen3.8-27B-CTL-FP8 --served-model-name ctl2 \ --max-model-len 16384 --max-num-seqs 64 --enable-prefix-caching
vLLM reads the quantization from config.json; no --quantization flag is needed. Tested with vLLM 0.23 on one
NVIDIA RTX PRO 6000 Blackwell (96 GB): the model takes 29.5 GiB, and at --gpu-memory-utilization 0.90 the KV cache holds
about 760k tokens. FP8 compute needs a GPU with FP8 tensor cores (Ada Lovelace, Hopper, Blackwell).
Set --max-num-seqs explicitly. vLLM's default of 1024 exceeds the Mamba cache blocks of this hybrid architecture,
and the server refuses to start. All evaluations ran with 64.
Request settings
The same as for the bf16 model:
Table with columns: setting, value| setting | value |
|---|
| system prompt | the full text of system_prompt.txt, verbatim, for both code generation and review |
| thinking | on: chat_template_kwargs: {"enable_thinking": true, "reasoning_effort": "medium"} |
| sampling | temperature 0.4, top_p 0.95, top_k 20 |
| max tokens | 16384 |
The system prompt is part of the deployed configuration: it carries CTL2 rules that the model follows more reliably in
context than from training alone.
from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")system_prompt = open("system_prompt.txt").read()response = client.chat.completions.create( model="ctl2", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": "Write a CTL2 Reformat transform that copies all fields and parses the string " "field amount_text (values like \"$(1,250.00)\") into the decimal field amount."}, ], temperature=0.4, top_p=0.95, max_tokens=16384, extra_body={"top_k": 20, "chat_template_kwargs": {"enable_thinking": True, "reasoning_effort": "medium"}},)print(response.choices[0].message.content)
Compile and retry
Validate the generated code (CTL compiler or a CloverDX server), run it on test data, and when something fails send the
error message back as the next user turn. Allow up to two retries.
Evaluation
Internal measurements, thinking on, the same prompts and settings for both models. Generated code was run on a CloverDX
server and scored on whether it produced the expected output rows, not by an LLM judge. Differences are FP8 − bf16 with
95% bootstrap confidence intervals.
Table with columns: test, FP8, bf16, difference| test | FP8 | bf16 | difference |
|---|
| new-domain code generation, 165 tasks × 2, first attempt | 0.92 | 0.93 | −0.01 [−0.04, +0.02] |
| same, after the compile-and-retry loop | 0.95 | 0.94 | +0.00 [−0.02, +0.03] |
code review of known-clean code: answers with a false [ERROR] | 0.00 | 0.00 | 0.00 |
| code review of code with planted, server-confirmed defects: defects found | 0.94 | 0.93 | +0.01 [−0.01, +0.03] |
None of the differences is significant.
Limitations
The same as the bf16 model:
- CTL2 only: it knows CloverDX graphs as context for CTL, but it is not trained to design whole graphs or to write
component XML.
- Rare components and their contracts (for example job-control components such as GetJobInput, SetJobOutput or
ExecuteScript) are weak without a description of the component in the prompt.
- Like any code model, it can produce code that compiles and runs but does not do what was asked. Test generated code on
real data before using it.