USENIX ATC '23 - Accelerating Distributed MoE Training and Inference with Lina

USENIX ATC '23 - Accelerating Distributed MoE Training and Inference with Lina

Jiamin Li, City University of Hong Kong, Yimin Jiang, ByteDance Inc., Yibo Zhu, Unaffiliated, Cong Wang, City University of Hong Kong, Hong Xu, The Chinese University of Hong Kong

Scaling model parameters improves model quality at the price of high computation overhead. Sparsely activated models, usually in the form of Mixture of Experts (MoE) architecture, have sub-linear scaling of computation cost with model size, thus providing opportunities to …

bytedance chinese city distributed hong kong kong quality scaling training university usenix wang