Selective Pre-training for Private Fine-tuning

Suppose we want to train text prediction models in email clients or word processors. These models, which serve billions of predictions per hour, must preserve the privacy of user data and adhere to specific model size constraints to meet memory, inference time requirements, and to reduce inference cost. Building small, fast, and private domain-specific language models is a thriving area of research. In this work, we show that a careful pre-training on a {\em subset} of the public dataset that …

building clients constraints cost cs.cr cs.lg data domain email fast fine-tuning language memory prediction predictions privacy private processors requirements size text train training user data word