Concerned with Data Contamination? Assessing Countermeasures in Code Language Model

arXiv:2403.16898v1 Announce Type: new
Abstract: Various techniques have been proposed to leverage the capabilities of code language models (CLMs) for SE tasks. While these techniques typically evaluate their effectiveness using publicly available datasets, the evaluation can be subject to data contamination threats where the evaluation datasets have already been used to train the concerned CLMs. This can significantly affect the reliability of the evaluation. Different countermeasures have been suggested to mitigate the data contamination threat. Countermeasures include using more recent …

arxiv can capabilities code countermeasures cs.cr cs.se data datasets evaluation language language models techniques threats