Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain

arXiv:2403.10576v1 Announce Type: new
Abstract: Cybersecurity information is often technically complex and relayed through unstructured text, making automation of cyber threat intelligence highly challenging. For such text domains that involve high levels of expertise, pretraining on in-domain corpora has been a popular method for language models to obtain domain expertise. However, cybersecurity texts often contain non-linguistic elements (such as URLs and hash values) that could be unsuitable with the established pretraining methodologies. Previous work in other domains have removed or …

arxiv automation cs.cl cs.cr cs.lg cyber cybersecurity cyber threat cyber threat intelligence domain domains don expertise high information intelligence language linguistic making non popular text threat threat intelligence