Save on Training Costs of Generative AI with PeriFlow

Generative AI is already widely used for chatbots, translation, code generation, summarization, image generation, and much more. Thanks to recent advances in generative AI, it can now generate high-quality texts and images. A report from Sequoia Capital says “Just as mobile unleashed new types of applications …, we expect these large models to motivate a new wave of generative AI applications.”¹ A notable example of generative AI is GPT-3², a pre-trained language model for diverse text generation tasks.

We recently had a chance to compare PeriFlow with Microsoft DeepSpeed on training a 16B GPT-3 model.

To train the model, we used 16 VMs, each of which hosts 8 NVIDIA A100 40GB GPUs. In total, we used 128 A100 GPUs and ran 150K steps to train the model.

PeriFlow speeds up training by 3.5x compared to Microsoft DeepSpeed on one of the top 3 public clouds thanks to its engine-cloud co-optimization. PeriFlow chooses the best time-cost tradeoff based on the chosen model and cloud, supporting AWS, Azure, and GCP. Save time and reduce costs significantly with PeriFlow. Try out PeriFlow now to train your own generative AI models!

[1] https://www.sequoiacap.com/article/generative-ai-a-creative-new-world/

[2] https://arxiv.org/abs/2005.14165