Baselines for Identifying Watermarked Large Language Models. (arXiv:2305.18456v1 [cs.LG])

We consider the emerging problem of identifying the presence and use of
watermarking schemes in widely used, publicly hosted, closed source large
language models (LLMs). We introduce a suite of baseline algorithms for
identifying watermarks in LLMs that rely on analyzing distributions of output
tokens and logits generated by watermarked and unmarked LLMs. Notably,
watermarked LLMs tend to produce distributions that diverge qualitatively and
identifiably from standard models. Furthermore, we investigate the
identifiability of watermarks at varying strengths and consider …

algorithms baselines distributions emerging generated language language models large llms problem tokens watermarking