Differentially Private Training of Mixture of Experts Models

This position paper investigates the integration of Differential Privacy (DP) in the training of Mixture of Experts (MoE) models within the field of natural language processing. As Large Language Models (LLMs) scale to billions of parameters, leveraging expansive datasets, they exhibit enhanced linguistic capabilities and emergent abilities. However, this growth raises significant computational and privacy concerns. Our study addresses these issues by exploring the potential of MoE models, known for their computational efficiency, and the application of DP, a standard …

capabilities cs.cr cs.lg datasets differential privacy experts growth integration language language models large linguistic llms natural natural language natural language processing privacy private scale training