QizhiPei

BioMatrix-1.7B-Base

README

License: apache-2.0

Model Overview

BioMatrix maps all biological modalities into a shared discrete token space via a unified tokenization scheme:

Molecular 1D sequences (both SMILES and SELFIES notations)
Molecular 3D structures (via MolStrucTok with branch-decoupled decoder)
Protein 1D sequences (residue-level tokens)
Protein 3D structures (via GCP-VQVAE backbone tokenizer)
Natural language (inherited from Qwen3 tokenizer)

All modalities are consumed and produced uniformly under a single next-token prediction objective—without external encoders, projection adapters, or modality-specific output heads.

Table with columns: Model, Molecule 1D, Molecule 3D, Protein 1D, Protein 3D, Natural Language
Model	Molecule 1D	Molecule 3D	Protein 1D	Protein 3D	Natural Language
ESM3	✗	✗	✓	✓	✓
3D-MoLM	✓	✓	✗	✗	✓
AlphaFold3	✓	✓	✓	✓	✗
BioT5/BioT5+	✓	✗	✓	✗

Model Details

Base Architecture: Qwen3-1.7B-Base
Parameters: 1.7B
Training Stage: Multimodal Continual Pretraining only (not instruction-tuned)
Training Tokens: 304.4B
Context Length: 8,192 tokens
Tokenizer: Extended Qwen3 vocabulary with:
- 11,294 joint molecular 3D tokens (composed from SELFIES atom × MolStrucTok codes)
- 4,096 protein 3D tokens (GCP-VQVAE codebook)
- 26 protein 1D tokens (amino acids + non-standard/unknown)
- SELFIES atom tokens and modality-specific control tokens

Embedding Initialization

New vocabulary entries are initialized via a description-based scheme: each new token is grounded in the pretrained Qwen3 embedding space by averaging the embeddings of the subword tokens of a short natural-language description (e.g., <A_W> → "Tryptophan"), plus a small isotropic Gaussian perturbation to break symmetry. This provides a more stable starting point than random initialization.

Pretraining Corpus (304.4B tokens)

Table with columns: Category, Tokens, Sources
Category	Tokens	Sources
Text (105.3B)	General: 25.6B	FineWeb-Edu
	Scientific: 79.7B	FineFineWeb (biology/chemistry/medical/health), PubMed Full Articles
Molecule (73.7B)	1D: 36.0B	PubChem, MolTextNet
	3D: 17.6B	PubChem, PCQM4Mv2, PubChemQC
	Other: 24.0B	(text descriptions, properties, IUPAC names)
(77.4B)

Training Configuration

Framework: LLaMA-Factory
Hardware: 64 NVIDIA H100 GPUs
Global Batch Size: 1,024
Maximum Sequence Length: 8,192 tokens
Optimizer: AdamW
Peak Learning Rate: 2.0 × 10⁻⁴ (cosine schedule)
Warmup Steps: 2,000
Total Steps: ~36.4K (1 epoch over the full 304.4B-token corpus)

Intended Use

This Base model is not instruction-tuned. It is suitable for:

Further fine-tuning on custom biological tasks
Continued pretraining on domain-specific corpora
Research on representation learning across biomolecular modalities
Embedding extraction for downstream classification/regression tasks

For ready-to-use instruction-following capabilities (e.g., molecule captioning, protein design, property prediction), please use the SFT variant.

The BioMatrix SFT variants cover downstream molecule and protein task families such as generation, prediction, captioning, editing, folding, inverse folding, and structure-aware question answering:

Quick Start

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "QizhiPei/BioMatrix-1.7B-Base"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True
)

# Example: Continue a SMILES sequence
prompt = "<|mol_smi_start|>CC(=O)"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))

Modality Wrapping

When constructing inputs, biomolecular content must be wrapped with the corresponding control tokens:

Table with columns: Modality, Wrapping Example
Modality	Wrapping Example
Molecule SMILES	`<\|mol_smi_start\|>CC#CC#N<\|mol_smi_end\|>`
Molecule SELFIES	`<\|mol_sfi_start\|>[C][#C][C][#N]<\|mol_sfi_end\|>`
Molecule 3D	`<\|mol_3d_start\|>[H 3][C 0][#C 6]...<\|mol_3d_end\|>`
Protein 1D	`<\|prot_aa_start\|><A M><A R><A A>...<\|prot_aa_end\|>`
Protein 3D	`<\|prot_3d_start\|><S 4012><S 153><S 2091>...<\|prot_3d_end\|>`

Natural language text is left unwrapped and serves as the default carrier modality.

Limitations

This model is not instruction-tuned and is unlikely to follow natural-language instructions out-of-the-box. Use the SFT variant for instruction-following.
Molecular and protein 3D structures are tokenized in disjoint geometric reference frames, so the model cannot natively represent biomolecular complexes (e.g., docking poses).
Heavy domain specialization may erode some general-purpose language capabilities of the underlying Qwen3 backbone.
Coverage is limited to small molecules and proteins; nucleic acids, carbohydrates, and lipids are not currently supported.

Citation

If you find BioMatrix useful, please cite:

bibtex
@article{pei2026biomatrix,
  title={BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language},
  author={Pei, Qizhi and Zhou, Zhimeng and Duan, Yi and Zhao, Yiyang and Li, Wei and Guo, Han and He, Liang and Li, Chengping and Hsieh, Chang-Yu and He, Conghui and Yan, Rui and Wu, Lijun},
  journal={arXiv preprint arXiv:2606.22138},
  year={2026}
}

License

This model is released under the Apache 2.0 license. The base model (Qwen3-1.7B-Base) is subject to its own license terms.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

QizhiPei

Model Tree

Base

Qwen/Qwen3-1.7B-Base

Fine-tuned

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Overview

BioMatrix maps all biological modalities into a shared discrete token space via a unified tokenization scheme:

Molecular 1D sequences (both SMILES and SELFIES notations)
Molecular 3D structures (via MolStrucTok with branch-decoupled decoder)
Protein 1D sequences (residue-level tokens)
Protein 3D structures (via GCP-VQVAE backbone tokenizer)
Natural language (inherited from Qwen3 tokenizer)

All modalities are consumed and produced uniformly under a single next-token prediction objective—without external encoders, projection adapters, or modality-specific output heads.

Table with columns: Model, Molecule 1D, Molecule 3D, Protein 1D, Protein 3D, Natural Language
Model	Molecule 1D	Molecule 3D	Protein 1D	Protein 3D	Natural Language
ESM3	✗	✗	✓	✓	✓
3D-MoLM	✓	✓	✗	✗	✓
AlphaFold3	✓	✓	✓	✓	✗
BioT5/BioT5+	✓	✗	✓	✗

Model Details

Base Architecture: Qwen3-1.7B-Base
Parameters: 1.7B
Training Stage: Multimodal Continual Pretraining only (not instruction-tuned)
Training Tokens: 304.4B
Context Length: 8,192 tokens
Tokenizer: Extended Qwen3 vocabulary with:
- 11,294 joint molecular 3D tokens (composed from SELFIES atom × MolStrucTok codes)
- 4,096 protein 3D tokens (GCP-VQVAE codebook)
- 26 protein 1D tokens (amino acids + non-standard/unknown)
- SELFIES atom tokens and modality-specific control tokens

Embedding Initialization

Pretraining Corpus (304.4B tokens)

Table with columns: Category, Tokens, Sources
Category	Tokens	Sources
Text (105.3B)	General: 25.6B	FineWeb-Edu
	Scientific: 79.7B	FineFineWeb (biology/chemistry/medical/health), PubMed Full Articles
Molecule (73.7B)	1D: 36.0B	PubChem, MolTextNet
	3D: 17.6B	PubChem, PCQM4Mv2, PubChemQC
	Other: 24.0B	(text descriptions, properties, IUPAC names)
(77.4B)

Training Configuration

Framework: LLaMA-Factory
Hardware: 64 NVIDIA H100 GPUs
Global Batch Size: 1,024
Maximum Sequence Length: 8,192 tokens
Optimizer: AdamW
Peak Learning Rate: 2.0 × 10⁻⁴ (cosine schedule)
Warmup Steps: 2,000
Total Steps: ~36.4K (1 epoch over the full 304.4B-token corpus)

Intended Use

This Base model is not instruction-tuned. It is suitable for:

Further fine-tuning on custom biological tasks
Continued pretraining on domain-specific corpora
Research on representation learning across biomolecular modalities
Embedding extraction for downstream classification/regression tasks

For ready-to-use instruction-following capabilities (e.g., molecule captioning, protein design, property prediction), please use the SFT variant.

The BioMatrix SFT variants cover downstream molecule and protein task families such as generation, prediction, captioning, editing, folding, inverse folding, and structure-aware question answering:

Quick Start

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "QizhiPei/BioMatrix-1.7B-Base"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True
)

# Example: Continue a SMILES sequence
prompt = "<|mol_smi_start|>CC(=O)"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))

Modality Wrapping

When constructing inputs, biomolecular content must be wrapped with the corresponding control tokens:

Table with columns: Modality, Wrapping Example
Modality	Wrapping Example
Molecule SMILES	`<\|mol_smi_start\|>CC#CC#N<\|mol_smi_end\|>`
Molecule SELFIES	`<\|mol_sfi_start\|>[C][#C][C][#N]<\|mol_sfi_end\|>`
Molecule 3D	`<\|mol_3d_start\|>[H 3][C 0][#C 6]...<\|mol_3d_end\|>`
Protein 1D	`<\|prot_aa_start\|><A M><A R><A A>...<\|prot_aa_end\|>`
Protein 3D	`<\|prot_3d_start\|><S 4012><S 153><S 2091>...<\|prot_3d_end\|>`

Natural language text is left unwrapped and serves as the default carrier modality.

Limitations

This model is not instruction-tuned and is unlikely to follow natural-language instructions out-of-the-box. Use the SFT variant for instruction-following.
Molecular and protein 3D structures are tokenized in disjoint geometric reference frames, so the model cannot natively represent biomolecular complexes (e.g., docking poses).
Heavy domain specialization may erode some general-purpose language capabilities of the underlying Qwen3 backbone.
Coverage is limited to small molecules and proteins; nucleic acids, carbohydrates, and lipids are not currently supported.

Citation

If you find BioMatrix useful, please cite:

bibtex
@article{pei2026biomatrix,
  title={BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language},
  author={Pei, Qizhi and Zhou, Zhimeng and Duan, Yi and Zhao, Yiyang and Li, Wei and Guo, Han and He, Liang and Li, Chengping and Hsieh, Chang-Yu and He, Conghui and Yan, Rui and Wu, Lijun},
  journal={arXiv preprint arXiv:2606.22138},
  year={2026}
}

License

This model is released under the Apache 2.0 license. The base model (Qwen3-1.7B-Base) is subject to its own license terms.