microsoft

Dayhoff-3b-UR90-20350

Model Details

Model Description

Developed by: Kevin K. Yang, Sarah Alamdari, Alex J. Lee, Kaeli Kaymak-Loveless, Samir Char, Garyk Brixi, Carles Domingo-Enrich, Chentong Wang, Suyue Lyu, Nicolo Fusi, Neil Tenenholtz, Ava P. Amini
Model type: Hybrid state-space-model transformer architecture with mixture-of-experts
License: MIT

Model Sources

Repository: https://github.com/microsoft/dayhoff

Uses

Downstream Use

Dayhoff is intended for broad research use on protein language modeling. The model has been used and assessed on the following capabilities:

Unconditional design of protein sequences
Zero-shot mutation effect prediction on ProteinGym
Designing scaffolds for structural motifs in sequence space on RFDiffusion and MotifBench
Homolog conditioning with Dayhoff-3b-GR-HM and Dayhoff-3b-GR-HM-c

Bias, Risks, and Limitations

This model should not be used to generate anything that is not a protein sequence or a set of homologuous protein sequences. It is not meant for natural language or other biological sequences, such as DNA sequences. Not all sequences are guaranteed to be realistic. It remains difficult to generate high-quality sequences with no sequence homology to any natural sequence.

How to Get Started with the Model

The simplest way to use these models and datasets is via the HuggingFace interface. You will need PyTorch, mamba=ssm, causal-conv1d, and flash-attn.

Requirements:

PyTorch: 2.7.1
CUDA 12.8 and above

We recommend using uv and creating a clean environment.

bash
uv venv dayhoff 
source dayhoff/bin/activate

In that new environment, install PyTorch 2.7.1.

bash
uv pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu128

Now, we need to install mamba-ssm, flash-attn, causal-conv1d, and their prerequisites.

bash
uv pip install wheel packaging
uv pip install --no-build-isolation flash-attn causal-conv1d mamba-ssm

To import from HuggingFace, you will need to install these versions:

bash
uv pip install datasets==3.2.0 #for HF datasets
uv pip install transformers==4.51.3
uv pip install huggingface_hub~=0.34.4

Sample protein generation code:

py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed

set_seed(0)
torch.set_default_device("cuda")

model = AutoModelForCausalLM.from_pretrained("microsoft/Dayhoff-3b-UR90-20350").to("cuda")
tokenizer = AutoTokenizer.from_pretrained("microsoft/Dayhoff-3b-UR90-20350", trust_remote_code=True)


inputs = tokenizer(tokenizer.bos_token, return_tensors="pt", return_token_type_ids=False)

outputs = model.generate(inputs['input_ids'],max_length=50,do_sample=True)
sequence = tokenizer.batch_decode(outputs,skip_special_tokens=True)
print(sequence)

For detailed instructions on package usage, please refer to the README in model repo.

Evaluation

Results

See the preprint for the latest benchmark results and evaluations.

Model perplexity on held-out test sequences for Dayhoff models.

Table with columns: Model, UniRef50, GigaRef, Aligned homologs, Unaligned homologs
Model	UniRef50	GigaRef
170m-UR50	11.62	11.88
170m-UR90	11.52	11.85
170m-GR	13.67	9.36
170m-UR50-BRn

Quality of generated sequences as measured by ESMFold pLDDT and scPerplexity. Dataset statistics are for 1024 randomly-sampled sequences. Model statistics are for 1024 generations at T=1 in the N-to-C direction.

Table with columns: Model or dataset, pLDDT (mean ± s.d.), scPerplexity (mean ± s.d.)
Model or dataset	pLDDT (mean ± s.d.)	scPerplexity (mean ± s.d.)
Natural sequences
UniRef50	0.653 ± 0.196	9.45 ± 2.89
GigaRef-clusters	0.619 ± 0.199	9.69 ± 2.83
GigaRef-singletons	0.561 ± 0.201	10.07 ± 2.88
Generated sequences
170m-UR50

ProteinGym zero-shot performance Spearman’s correlation coefficient on ProteinGym substitutions and indels.

Table with columns: Input, Model, Parameters, Substitutions, Indels
Input	Model	Parameters	Substitutions	Indels
Single sequence	170m-UR50	170M	0.353	0.479
	170m-UR90	170M	0.354	0.483
	170m-GR	170M	0.199	0.292

RFDiffusion Benchmark Performance Motif scaffolding performance, problems solved, successes out of 100, and MotifBench score.

Table with columns: Problem, 170m-UR50, 170m-UR90, 170m-GR, 170m-UR50-BRn, 170m-UR50-BRq, 170m-UR50-BRu, 3b-UR90, 3b-GR-HM, 3b-GR-HM-c, EvoDiff-Seq
Problem	170m-UR50	170m-UR90	170m-GR	170m-UR50-BRn	170m-UR50-BRq	170m-UR50-BRu	3b-UR90	3b-GR-HM	3b-GR-HM-c	EvoDiff-Seq
1PRW	62	72	81	95	91	90	94	81

MotifBench Benchmark Performance Motif scaffolding performance, problems solved, successes out of 100, and MotifBench score.

Table with columns: Problem, 170m-UR50, 170m-UR90, 170m-GR, 170m-UR50-BRn, 170m-UR50-BRq, 170m-UR50-BRu, 3b-UR90, 3b-GR-HM, 3b-GR-HM-c, EvoDiff-Seq
Problem	170m-UR50	170m-UR90	170m-GR	170m-UR50-BRn	170m-UR50-BRq	170m-UR50-BRu	3b-UR90	3b-GR-HM	3b-GR-HM-c	EvoDiff-Seq
01_1LDB	1	1	3	0	0	1	20	2

Technical Specifications

Compute Infrastructure

170M-parameter models: trained on 8 NVIDIA A100 or 8 NVIDIA H100 GPUs using Distributed Data Parallel.
3B-parameter models: trained on 176 NVIDIA H100 GPUs using Fully Sharded Data Parallel in hybrid-shard mode.

Responsible AI Considerations

The intended use of this model is to generate high-quality, realistic, protein sequences or sets of homologous protein sequences. Generations can be designed from scratch or conditioned on partial sequences in both N→C and C→N directions.

The code and datasets released in this repository are provided for research and development use only. They are not intended for use in clinical decision-making or for any other clinical use, and the performance of these models for clinical use has not been established. You bear sole responsibility for any use of these models, data and software, including incorporation into any product intended for clinical use.

Citation

If you use the code, data, models, or results. please cite our preprint.

Data Summary

https://huggingface.co/microsoft/Dayhoff-3b-UR90-20350/blob/main/data_summary_card.md

Model provider

microsoft

Model tree

Base

this model

Modalities

Input

Output

Pricing

Dedicated Endpoints

View details

Supported Functionality

Model APIs