Model details
Wayfarer 12B was trained on top of the Nemo base model using a two-stage SFT approach, with the first stage containing 180K chat-formatted instruct data instances and the second stage consisting of a 50/50 mixture of synthetic 8k context text adventures and roleplay experiences.
How It Was Made
Wayfarer’s text adventure data was generated by simulating playthroughs of published character creator scenarios from AI Dungeon. Five distinct user archetypes played through each scenario, whose character starts all varied in faction, location, etc. to generate five unique samples.
One language model played the role of narrator, with the other playing the user. They were blind to each other’s underlying logic, so the user was actually capable of surprising the narrator with their choices. Each simulation was allowed to run for 8k tokens or until the main character died.
Wayfarer’s general emotional sentiment is one of pessimism, where failure is frequent and plot armor does not exist. This serves to counter the positivity bias so inherent in our language models nowadays.
Inference
The Nemo architecture is known for being sensitive to higher temperatures, so the following settings are recommended as a baseline. Nothing stops you from experimenting with these, of course.
"temperature": 0.8,
"repetition_penalty": 1.05,
"min_p": 0.025
Limitations
Wayfarer was trained exclusively on second-person present tense data (using “you”) in a narrative style. Other styles will work as well but may produce suboptimal results.
Additionally, Wayfarer was trained exclusively on single-turn chat data.
ChatML was used for both finetuning stages.
<|im_start|>system
You're a masterful storyteller and gamemaster. Write in second person present tense (You are), crafting vivid, engaging narratives with authority and confidence.<|im_end|>
<|im_start|>user
> You peer into the darkness.<|im_end|>
<|im_start|>assistant
You have been eaten by a grue.
GAME OVER<|im_end|>
Credits
Thanks to Gryphe Padar for collaborating on this finetune with us!