mamba-50m API & Inference Endpoint

Model details

Table

Architecture	Mamba (selective SSM)
Parameters	~50M
Context length	512 tokens
Tokenizer	GPT-NeoX-20B (BPE, ~50k vocab) — the same tokenizer used by the original `state-spaces/mamba-*` models
Language	English
License	Apache-2.0

Limitations

Pretrained only. No fine-tuning, instruction tuning, or alignment. It does not follow instructions and has no safety filtering; it simply continues text.
Small. At ~50M parameters it has limited fluency and reasoning; expect frequent hallucination and repetition.
English only. Trained solely on English Wikipedia; other languages are out of distribution.
Domain-narrow. Only Wikipedia was used as training data.

Training data

Pretrained on the English subset of Wikipedia: over 3 million articles.

Training procedure

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Learning rate	5e-4
Sequence length	512
Batch size	64
Tokenizer	GPT-NeoX-20B (`EleutherAI/gpt-neox-20b`)
Optimizer	`AdamW`
LR schedule / warmup	`constant` / `10000`
Total tokens seen

Evaluation

Evaluated on a held-out set of 10,000 Wikipedia articles that were not seen during training. The training and evaluation loss curves are shown below.

Training and evaluation loss

Citation

If you use this model, please cite the Mamba paper:

bibtex
@article{gu2023mamba,
  title={Mamba: Linear-Time Sequence Modeling with Selective State Spaces},
  author={Gu, Albert and Dao, Tri},
  journal={arXiv preprint arXiv:2312.00752},
  year={2023}
}

Model details

Table

Architecture	Mamba (selective SSM)
Parameters	~50M
Context length	512 tokens
Tokenizer	GPT-NeoX-20B (BPE, ~50k vocab) — the same tokenizer used by the original `state-spaces/mamba-*` models
Language	English
License	Apache-2.0

Limitations

Pretrained only. No fine-tuning, instruction tuning, or alignment. It does not follow instructions and has no safety filtering; it simply continues text.
Small. At ~50M parameters it has limited fluency and reasoning; expect frequent hallucination and repetition.
English only. Trained solely on English Wikipedia; other languages are out of distribution.
Domain-narrow. Only Wikipedia was used as training data.

Training data

Pretrained on the English subset of Wikipedia: over 3 million articles.

Training procedure

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Learning rate	5e-4
Sequence length	512
Batch size	64
Tokenizer	GPT-NeoX-20B (`EleutherAI/gpt-neox-20b`)
Optimizer	`AdamW`
LR schedule / warmup	`constant` / `10000`
Total tokens seen

Evaluation

Evaluated on a held-out set of 10,000 Wikipedia articles that were not seen during training. The training and evaluation loss curves are shown below.

Training and evaluation loss

Citation

If you use this model, please cite the Mamba paper:

bibtex
@article{gu2023mamba,
  title={Mamba: Linear-Time Sequence Modeling with Selective State Spaces},
  author={Gu, Albert and Dao, Tri},
  journal={arXiv preprint arXiv:2312.00752},
  year={2023}
}

mamba-50m

README

Model details

Limitations

Training data

Training procedure

Evaluation

Citation

Explore FriendliAI today

README

Model details

Limitations

Training data

Training procedure

Evaluation

Citation