What it is
- 14.4M parameters: 8
plain causal attention layers, model_dim 384, seq_len 512, nucleotide vocab 9
(A/C/G/T/N + BOS/EOS/PAD/MASK). A standard GPT-style stack, exported as a native Hugging Face
GPT2LMHeadModel.
- Pretrained next-base on GRCh38 chromosomes 1-3 (a continuous nucleotide stream), 20,000 steps, with
reverse-complement augmentation. Validation loss 1.08.
Load and generate
It loads with stock transformers as a GPT2LMHeadModel, no trust_remote_code. Token ids are BOS=1,
EOS=2, PAD=0, and A/C/G/T/N = 4/5/6/7/8.
import torch
from transformers import AutoModelForCausalLM
repo = "mrothroc/dna-enhancers-attention-mixlab"
model = AutoModelForCausalLM.from_pretrained(repo).eval()
ids2base = {4: "A", 5: "C", 6: "G", 7: "T", 8: "N"}
out = model.generate(torch.tensor([[1]]), do_sample=True, max_length=60, temperature=1.0, top_k=0)
print("".join(ids2base.get(t, "") for t in out[0].tolist() if t in ids2base))
For classification, load with AutoModelForSequenceClassification and fine-tune. The full recipe is in the
cookbook below.
Reproduce / learn more
Full recipe (reproduce this baseline, then improve it by swapping the mixer to Mamba, all on a laptop):
mixlab cookbook, genomics-mamba.
Citation
Please cite what this builds on:
- Genomic Benchmarks (the task): Gresova, K. et al. "Genomic benchmarks: a collection of datasets for
genomic sequence classification." BMC Genomic Data 2023, 24, 25. doi:10.1186/s12863-023-01123-8
- GRCh38 human reference genome (Genome Reference Consortium), via the UCSC Genome Browser: Kent, W. J.
et al. "The Human Genome Browser at UCSC." Genome Research 2002, 12 (6), 996-1006. doi:10.1101/gr.229102