GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Ar4ikov
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Ar4ikov
Model Tree
PaddlePaddle/PaddleOCR-VL-1.6
Input Modalities
Output Modalities
Supported Functionality
| Part | Params | Precision |
|---|---|---|
Language model (ERNIE-4.5-0.3B, 18 layers): q/k/v/o_proj, gate/up/down_proj | 254.8M | INT4, group 128, asymmetric |
| Vision tower (27 layers, SigLIP-style) | 411.0M | BF16 |
mlp_AR projector | 25.9M | BF16 |
lm_head | 105.9M | BF16 |
| Token embeddings | 105.9M | BF16 |
Only the language model is quantized, and that is not a conservative choice — it is the only
option vLLM accepts. vLLM builds this vision tower unquantized no matter what the checkpoint
says, because of a divisibility gate in vllm/model_executor/models/siglip.py:
python
quantizable = (config.hidden_size % 64 == 0 and config.intermediate_size % 64 == 0)
PaddleOCR-VL's vision tower has intermediate_size = 4304, and 4304 / 64 = 67.25. A checkpoint
with a quantized vision MLP therefore fails to load outright:
markdown
ValueError: There is no module or parameter named'vision_model.encoder.layers.0.mlp.fc1.weight_packed' in SiglipVisionModel
The same 4304 also blocks a group_size = 128 scheme for mlp.fc2 on the transformers side.
Quantizing only the vision attention (1152, divisible) does load and gets weights down to
1.27 GiB, but measured worse on OCR — see the table below.
vLLM 0.29.0, one RTX 3090, --max-model-len 16384 --gpu-memory-utilization 0.40:
| BF16 | this checkpoint | |
|---|---|---|
| weights | 1.82 GiB | 1.48 GiB |
Same flags but deliberately small (
markdown
--max-model-len 8192 --max-num-seqs 4 --gpu-memory-utilization 0.11 --enforce-eager
| BF16 | this checkpoint | |
|---|---|---|
| weights | 1.82 GiB | 1.48 GiB |
| KV cache | 0.43 GiB | 0.77 GiB |
| total process VRAM | 2726 MiB | 2706 MiB |
The 0.34 GiB saved on weights went straight into the KV cache; the total footprint is the same.
Quantization here buys more KV cache at equal footprint, not a smaller footprint. What shrinks
the footprint is --gpu-memory-utilization; the weights only set the floor, and this checkpoint's
floor is 0.34 GiB lower. That matters if you must fit under ~2.4 GB, and not otherwise.
This is the expected outcome for a 0.9B model: only 255M of its 906M parameters are quantizable, so INT4 gives ~1.2x on weights, not ~4x.
Evaluated through the real PaddleOCRVL pipeline (paddleocr 3.7.0, PP-DocLayoutV2 layout
detection on CPU, this checkpoint served by vLLM), not by prompting the model with whole pages —
in production the pipeline sends one region crop per request with a task prompt. 20 held-out pages
of getomni-ai/ocr-benchmark,
scored against the reference page markdown after stripping markup from both sides (the pipeline
emits HTML tables, the reference uses markdown pipe tables, so raw text similarity would measure
the table dialect).
tokF1 is F1 over the token multiset. broken counts pages with a >=40-character run of one
repeated character, or an output under half the BF16 length.
| Checkpoint | Weights | tokF1 | broken pages |
|---|---|---|---|
| BF16 original | 1.82 GiB | 0.803 | 0 / 20 |
| this checkpoint (LLM, asym) | 1.48 GiB | 0.732 | 2 / 20 |
| LLM, crop-calibrated, symmetric | 1.48 GiB | 0.719 | 5 / 20 |
| LLM, crop-calibrated, asym | 1.48 GiB | 0.692 |
On the pages that come out intact, every quantized variant matches BF16 (this checkpoint:
0.807 vs 0.813 on its 18 good pages, −0.7%). The whole average gap is a handful of pages that
fail outright: on a dense credit-card statement the model runs away into 000000... for thousands
of characters, and on one IRS 1040 form it emits 336 characters where BF16 emits 12081, dropping
the table body.
Six configurations were tried against this — page-level and crop-level calibration, symmetric and asymmetric, and holding out the layers usually blamed for 4-bit degeneracy. None removed it, and the two hold-out variants cost more memory than they saved. The failure looks intrinsic to running a 0.3B language model at 4 bits over long table generations; there is little redundancy to spare.
Recommendation: if you can afford BF16, use BF16 and control memory with
--gpu-memory-utilization / --max-model-len, which is where the memory actually is. Reach for
this checkpoint when you must fit under roughly 2.4 GB, and validate on your own documents first —
especially dense financial tables. Consider adding a repetition penalty and a max_tokens cap.
python
from llmcompressor import oneshotfrom llmcompressor.modifiers.awq import AWQMapping, AWQModifierIGNORE = ["re:.*lm_head", r"re:model\.projector\..*", r"re:.*visual\..*"]# PaddleOCRVLForConditionalGeneration is not in llm-compressor's AWQ mapping# registry; the default mappings are written for a plain decoder stack.MAPPINGS = [AWQMapping("re:.*language_model.*input_layernorm$",["re:.*language_model.*self_attn.q_proj$","re:.*language_model.*self_attn.k_proj$","re:.*language_model.*self_attn.v_proj$"]),AWQMapping("re:.*language_model.*self_attn.v_proj$",["re:.*language_model.*self_attn.o_proj$"]),AWQMapping("re:.*language_model.*post_attention_layernorm$",["re:.*language_model.*mlp.gate_proj$","re:.*language_model.*mlp.up_proj$"]),AWQMapping("re:.*language_model.*mlp.up_proj$",["re:.*language_model.*mlp.down_proj$"]),]oneshot(model=model, # AutoModelForImageTextToText, transformers >= 5.2processor=processor,dataset=ds,recipe=AWQModifier(targets=["Linear"], scheme="W4A16_ASYM", ignore=IGNORE, mappings=MAPPINGS,# AWQ caches activations for every calibration sample at once, and the# vision tower sees patches before the 2x2 merge -- 24 GB is not enough.offload_device=torch.device("cpu"),),data_collator=data_collator,num_calibration_samples=96,)model.save_pretrained(SAVE_DIR, save_compressed=True) # without this the file does not shrink
Calibration: 96 document pages from getomni-ai/ocr-benchmark, each as a user turn (image +
OCR:) with the ground-truth markdown as the assistant turn, capped at 512·28·28 pixels.
recipe.yaml in this repo is what llm-compressor recorded.
llm-compressor writes quantization_config.ignore as absolute module paths of the runtime
transformers tree (model.visual..., model.projector...). The checkpoint and vLLM name those
same modules differently (visual..., mlp_AR..., language_model.lm_head), so the entries match
nothing there, vLLM concludes the vision tower is quantized, builds weight_packed parameters for
it, meets a plain weight in the file and dies with
markdown
'QKVParallelLinear' object has no attribute 'data'
ignore list in this repo was rewritten as suffix regexes, which match in every
namespace:
json
["re:.*lm_head", "re:.*visual\\..*", "re:.*vision_model\\..*","re:.*mlp_AR\\..*", "re:.*projector\\..*"]
bash
vllm serve Ar4ikov/PaddleOCR-VL-1.6 \--served-model-name PaddleOCR-VL-1.6-0.9B \--max-model-len 16384 --gpu-memory-utilization 0.4 \--limit-mm-per-prompt.image 1
python
from paddleocr import PaddleOCRVLpipeline = PaddleOCRVL(vl_rec_backend="vllm-server",vl_rec_server_url="http://127.0.0.1:8000/v1",)for res in pipeline.predict("page.png"):print(res.markdown["markdown_texts"])
The pipeline asks for the model under the name PaddleOCR-VL-1.6-0.9B, hence
--served-model-name. Prompts, if you drive the model directly: OCR:,
Table Recognition:, Formula Recognition:, Chart Recognition:, Spotting:,
Seal Recognition:.
| 4 / 20 |
| LLM minus first/last layer | 1.52 GiB | 0.714 | 2 / 20 |
LLM minus every down_proj | 1.56 GiB | 0.716 | 2 / 20 |
| LLM + vision attention | 1.27 GiB | 0.676 | 3 / 20 |
| LLM + vision MLP | — | — | does not load in vLLM |