Model
- Llama-style decoder (
LlamaForCausalLM): hidden 256, 6 layers,
8 heads, intermediate 1024
- Context: 1024 bases (+
[BOS]/[EOS])
- Objective: next-base prediction (causal language modeling)
- Tokenizer: character level,
[PAD] [UNK] [BOS] [EOS] A C G T N. One base = one token,
so a single-base change alters exactly one token. Lowercase (soft-masked) input is
uppercased and any other letter becomes N.
Training data
- UCSC hg19 chromosomes from
hgdownload.soe.ucsc.edu
- Train: chr22 (both strands), validation: chr21 (+ strand)
- Non-overlapping 1024 bp windows, dropping windows with more than
1% unknown bases (
N)
- 4000 steps, batch size 16, learning rate 0.001 (cosine)
Evaluation
Bits per base on chr21 (35,087,360 bases in windows without N),
each base predicted from the bases before it in its window. Uniform guessing = 2.0;
lower is better.
Table with columns: Method, Bits per base| Method | Bits per base |
|---|
| This model | 1.7439 |
| Markov order 0 | 1.9905 |
| Markov order 1 | 1.9522 |
| Markov order 2 | 1.9383 |
| Markov order 3 | 1.9326 |
| Markov order 4 | 1.9258 |
| Markov order 5 | 1.9178 |
| Markov order 6 | 1.9014 |
| Markov order 8 | 1.8574 |
| Markov order 10 | 1.8308 |
| Uniform | 2.0000 |
Usage
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("rubenich/genetic-dna-tiny")model = AutoModelForCausalLM.from_pretrained("rubenich/genetic-dna-tiny") ids = tok("ACGTTGCA", return_tensors="pt")["input_ids"]with torch.no_grad(): logits = model(input_ids=ids).logits
Zero-shot variant scoring compares log P(alt window) - log P(ref window) on an hg19
window around a single-base change; see genetic_model.score_variant in the source repo.
Limitations
- Trained on 1 chromosome(s) of one reference genome: it has not seen
human variation, other chromosomes, or other species.
- Tiny model and short context: it captures local sequence statistics, not
long-range regulatory structure.
- Coordinates and alleles must be GRCh37/hg19, + strand.