GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
andrewluo
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
andrewluo
Model Tree
Qwen/Qwen3.5-4B
Input Modalities
Output Modalities
Supported Functionality
Qwen/Qwen3.5-4B, revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.a3f166e5e57532f15948e5629046c52e3ef06bf488dd456f26698cd8e63094b8.The continuation's final validation loss was 1.046623 on 32 sequences from the unchanged 1,446-row validation pool, versus 0.894963 at the end of the initial 30k run. This small general validation slice does not establish chart-caption improvement; no held-out chart-quality benchmark is claimed.
Use the configurations under serving/. They target one H100 with vllm/vllm-openai:v0.29.0, BF16 weights, an 8,192-token context, one image per request, a 2,048-image-token pixel budget, and thinking disabled. CUDA graphs and optimized kernels remain enabled through the engine defaults. These are validated configuration candidates, not measured performance optima.
serving/baseline/config.yaml: ordinary decoding for correctness and latency comparison.serving/dflash-7/config.yaml: DFlash with seven speculative tokens, an eight-token block.serving/dflash-15/config.yaml: DFlash with fifteen speculative tokens, a sixteen-token block, for latency testing at low concurrency.Each configuration mounts this private target checkpoint and, for DFlash, the pinned public z-lab/Qwen3.5-4B-DFlash draft. Set the Baseten workspace secret HF_TOKEN to a read token that can access this private repository. The target and draft revisions are pinned in the deployment configurations.
DFlash is a serving-time optimization using a separate draft model. The published draft was trained against the original Qwen3.5-4B, not this fine-tune. Hidden size, vocabulary, target layer count and feature-layer indices match, and target input/output embeddings are unchanged. Acceptance rate and actual speedup on this fine-tune must be measured. Standard target verification is configured; lossy or synthetic acceptance modes are not enabled.
The target remains BF16. No FP8/INT4 quantization or draft retraining has been applied. Choose a speculative block size by comparing real caption latency, generated-token throughput, and acceptance statistics against the baseline at the expected concurrency. If DFlash is slower on the workload, use the baseline configuration.
Send image/text messages to /v1/chat/completions using the served model name figure-caption-charts and temperature: 0. Use the training task prompt for quality comparisons; a generic prompt can be used for a connectivity check.
Provide one cropped figure image and the following text together in a single user message. This is the exact instruction used for the chart-continuation training and the current local caption evaluation:
text
Describe this document figure accurately and usefully. Include clearly legible labels, quantities, units, and important visual relationships. State uncertainty when details cannot be read. Do not invent context or values.No additional system message is used. The expected output is a plain-text caption, not a JSON object or chart-data schema. The prompt asks for visible details and uncertainty rather than invented context. Fine-tuning does not guarantee factual accuracy.
Send this JSON body to the deployed server's chat-completions endpoint, replacing the image placeholder with actual base64-encoded PNG bytes. Use the corresponding MIME type for a different supported image format. Authentication and the external URL depend on the deployment gateway.
json
{ "model": "figure-caption-charts", "messages": [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "data:image/png;base64,<BASE64_ENCODED_IMAGE>" } }, { "type": "text", "text": "Describe this document figure accurately and usefully. Include clearly legible labels, quantities, units, and important visual relationships. State uncertainty when details cannot be read. Do not invent context or values." } ] } ], "temperature": 0, "seed": 0, "max_tokens": 2048, "chat_template_kwargs": { "enable_thinking": false }}Use the image as an actual image content item; do not paste base64 into the text prompt or manually prepend <image>. The included processor/chat template supplies the image tokens. With Transformers, supply the image and text through the processor's multimodal chat template with add_generation_prompt=True and enable_thinking=False.
The prepared serving configurations already disable thinking by default. The example makes that setting explicit. Temperature 0, seed 0, and a 2,048-output-token limit match the local evaluation; the limit is a ceiling, not a requested caption length. Decode the generated assistant response as text. A different prompt, output schema, image resolution, or decoding policy requires its own quality evaluation.
from_pretrained loader successfully loaded the complete exported checkpoint on CPU and generated a caption from an image-plus-text request.See conversion_validation.json, runtime_validation.json, dflash_compatibility.json, and manifest.json for verification details. Training images, records, and optimizer state are not included.
References: Baseten vLLM guide, vLLM DFlash implementation at v0.29.0, draft model.
{ "model": "figure-caption-charts", "messages": [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "data:image/png;base64,<BASE64_ENCODED_IMAGE>" } }, { "type": "text", "text": "Describe this document figure accurately and usefully. Include clearly legible labels, quantities, units, and important visual relationships. State uncertainty when details cannot be read. Do not invent context or values." } ] } ], "temperature": 0, "seed": 0, "max_tokens": 2048, "chat_template_kwargs": { "enable_thinking": false }}