FlexFlow

Distributed Deep Neural Network Training Framework

Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training

FlexFlow is a Deep Neural Network (DNN) framework that automatically discovers fast parallelization strategies for distributed DNN training. It goes beyond traditional methods like data and model parallelism by exploring various parallelization opportunities across Samples, Operators, Attributes, and Parameters. FlexFlow features a unique execution simulator to quantify the runtime performance of different strategies and employs an automated search algorithm to identify highly optimized approaches that typically exceed the performance of manually designed strategies. This framework allows for flexible DNN training through combinations of dimension-based parallelization.

pwa

https://flexflow.ai