DPZero: Private Fine-Tuning of Language Models without Backpropagation

arXiv:2310.09639v2 Announce Type: replace-cross
Abstract: The widespread practice of fine-tuning large language models (LLMs) on domain-specific data faces two major challenges in memory and privacy. First, as the size of LLMs continues to grow, the memory demands of gradient-based training methods via backpropagation become prohibitively high. Second, given the tendency of LLMs to memorize training data, it is important to protect potentially sensitive information in the fine-tuning data from being regurgitated. Zeroth-order methods, which rely solely on forward passes, substantially …

arxiv challenges cs.cr cs.lg data demands domain fine-tuning high language language models large llms major math.oc memory practice privacy private size stat.ml training