Method
YaRN positions combined with LongLoRA training: shifted sparse attention, LoRA on the attention projections, and trainable embedding and normalization layers. Inference uses normal full attention.
Training data: AmareshHebbar/Stretch32K_Data_TinyLlama_1.1B. Memory tricks: PyTorch FlashAttention kernels, gradient checkpointing, chunked loss over the vocabulary, and packed sequences with no padding.
Why finetune
Retrieval feeds a model a few chunks; it cannot reason across a whole book or contract in one pass. The base model rows below show what happens past 2048 tokens with no training, and with the popular training free scaling tricks applied at load time.
Results
Perplexity is measured on PG19 test books. Passkey hides a five digit key in repeated filler. Needle hides a seven digit number in real book text across a grid of lengths and depths. The RULER subset has multiple keys, multiple values and variable tracking. Long score is the mean of passkey, needle and RULER. LongAlpaca answer NLL is the loss on answers to held out long document questions (lower is better).
Perplexity by context length
Retrieval by context length (this model)



Training hyperparameters
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("AmareshHebbar/Stretch32K_TinyLlama_1.1B_YARN_LONGLORA")model = AutoModelForCausalLM.from_pretrained("AmareshHebbar/Stretch32K_TinyLlama_1.1B_YARN_LONGLORA", torch_dtype="bfloat16", attn_implementation="sdpa").cuda()text = open("long_document.txt").read()ids = tok(text + "\n\nSummary:", return_tensors="pt").input_ids[:, -16128:].cuda()out = model.generate(ids, max_new_tokens=200, do_sample=False)print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
The RoPE scaling settings are stored in config.json, so the model loads with the right positions in transformers and in vLLM. This is a base model, not a chat model.