Privacy-Aware Semantic Cache for Large Language Models

arXiv:2403.02694v1 Announce Type: cross
Abstract: Large Language Models (LLMs) like ChatGPT, Google Bard, Claude, and Llama 2 have revolutionized natural language processing and search engine dynamics. However, these models incur exceptionally high computational costs. For instance, GPT-3 consists of 175 billion parameters and inference on these models also demands billions of floating-point operations. Caching is a natural solution to reduce LLM inference costs on repeated queries. However, existing caching methods are incapable of finding semantic similarities among LLM queries, leading …

arxiv aware bard cache chatgpt claude computational computational costs cs.ai cs.cl cs.cr cs.dc cs.lg demands engine google google bard gpt gpt-3 high instance language language models large llama llama 2 llms natural natural language natural language processing operations point privacy search search engine semantic