Model details
- Role: Generator
- Dataset: GSM8K
- Base model:
meta-llama/Meta-Llama-3-8B-Instruct
- Adapter type: LoRA
- LoRA rank (
r): 16
- LoRA alpha: 32
- LoRA dropout: 0.1
- Target modules:
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
- Bias:
"none"
Usage
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
adapter_id = "sxiong/SWAP_v2_GSM8K_Gen_Llama3-8B-LoRA"
tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(
base_model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter_id)
For additional information and implementation details, please refer to the SWAP GitHub repository.
Citation
@inproceedings{xiong2025deliberate,
title={Deliberate reasoning in language models as structure-aware planning with an accurate world model},
author={Xiong, Siheng and Payani, Ali and Yang, Yuan and Fekri, Faramarz},
booktitle={Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
pages={31900--31931},
year={2025}
}