Paper
On the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer
PLatour explores Prefill-to-Autoregressive Cross-model Transfer (PACT): a larger sender encodes the prompt and supplies hidden representations, while a smaller receiver generates the response independently. The main method trains lightweight linear bridges between frozen backbones using task rewards, starting from zero-initialized projections and supporting hybrid-attention models.
The paper also introduces PACT-Check, with Dummy Transfer, Irrelevant Transfer, Self-Communication and Trained Receiver controls. These comparisons examine whether improvements depend on useful sender information or can arise through adaptation of the receiver. Better task performance alone does not demonstrate effective cross-model communication.
Standard PEFT adapter for the text decoder of the official Qwen3.5-4B backbone. It does not adapt the vision encoder.
These are historical inference artifacts without benchmark scores. Packaging does not establish a checkpoint-selection protocol or an unbiased evaluation. Independent evaluation and accurate provenance reporting are required before new comparative research claims.
Code · Model collection · Agent/API guide
Project environment
Use Linux with CUDA, Python 3.11 and enough GPU memory for the selected models.
Run the project commands from an editable checkout so the release manifests and
task files are available. See the installation guide.
git clone https://github.com/zhenyi4/platour.gitcd platourpython3.11 -m pip install --user uvpython3.11 scripts/install_environment.pysource .venv/bin/activate
Training, generation and scoring use the active Python. The examples below need
no vllm_python argument. See the environment guide
for installation options, runtime checks and compatibility details.
Use AutoModelForImageTextToText for the official multimodal backbone, then
attach the PEFT adapter. Only the text decoder is adapted. The adapter does
not include backbone weights; all original FP32 adapter values and fused QKV
modules are preserved. This differs from the project's vLLM runtime conversion.
Activate the project environment from the setup section above.
It includes Transformers and PEFT for this standard loading route.
import torchfrom peft import PeftModelfrom transformers import AutoModelForImageTextToText, AutoTokenizer base_id = "Qwen/Qwen3.5-4B"revision = "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a"base = AutoModelForImageTextToText.from_pretrained( base_id, revision=revision, dtype="auto", device_map="auto",)model = PeftModel.from_pretrained(base, "zen-E/PLatour-TR-LoRA-Qwen3.5-4B-DAPO")tokenizer = AutoTokenizer.from_pretrained(base_id, revision=revision)model.eval() messages = [{"role": "user", "content": "What is 2 + 2?"}]inputs = tokenizer.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt",).to(model.get_input_embeddings().weight.device)with torch.inference_mode(): output = model.generate(**inputs, max_new_tokens=256, do_sample=False)new_tokens = output[0, inputs["input_ids"].shape[-1]:]print(tokenizer.decode(new_tokens, skip_special_tokens=True))
This short custom-prompt example does not reproduce the benchmark protocol.
Increase max_new_tokens for longer answers. Generating code requires no coding
judge; running benchmark tests does.
Evaluate the released checkpoint
Run from the project environment above. This uses the published TEST inputs and
the selected experiment's sampling/scoring settings; no training is performed.
platour-exps evaluate --experiment 24 \ --weights zen-E/PLatour-TR-LoRA-Qwen3.5-4B-DAPO \ --receiver Qwen/Qwen3.5-4B \ --output ./outputs/24-release-eval
--sender and --receiver accept HF IDs or local directories. Missing backbones,
released weights and published task inputs download automatically. The release
loader verifies the configuration and weight hashes. Omitting --weights selects
the same experiment's published checkpoint. Use a new output directory.
To download and verify just the released checkpoint without loading models:
platour-exps download --experiment 24
See the task protocols and
checkpoint index.
The default evaluates AIME, HMMT and IMO with the published 32K response cap.
Add --eval-suites aime for AIME-135 and its Clean80 subset only; partial
runs report three_suite_macro: null. Adding --response-cap 16384 shortens
the cap and is recorded as a change from the published test protocol.
GPU placement for project evaluation
Table with columns: Option, Behavior for this checkpoint| Option | Behavior for this checkpoint |
|---|
Omit --n-gpu | One process with the existing automatic placement; no automatic data-parallel replicas. |
--n-gpu 8 | 8 independent replicas, each using 1 GPU, with disjoint question shards. |
--n-gpu N --gpus-per-replica G | N total GPUs split into N / G replicas; N must be divisible by G. |
The default group size for this checkpoint is 1 GPU per replica.
--n-gpu 1 is supported when the complete workload fits on one GPU.
Every replica needs enough memory for its models and decoding workload.
Use CUDA_VISIBLE_DEVICES to select/order GPUs. Do not combine --n-gpu
with torchrun. These flags apply to the project evaluator, not the Python
generate() examples. See evaluation GPU defaults.
Export dependency and scope
The base configuration/tokenizer used for this export is pinned to Qwen/Qwen3.5-4B revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. This pins the present-day export dependency; it is not a claim that the historical training run used that revision.
The original export/loading checks used Torch 2.15.0, Transformers 5.22.1 and PEFT 0.21.2. Loading and synthetic-prompt checks do not reproduce benchmark accuracy or a historical training trajectory. Upstream model terms and notices remain applicable.
Citation
@misc{pact_platour, title={On the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer}, author={Zhenyi Shen and Junru Lu and Wei Liu and Jiangnan Ye and Jiazheng Li and Di Yin and Xing Sun and Tameem Adel and Yali Du and Lin Gui and Yulan He}, year={2026}, eprint={2611.06976}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2611.06976},}