A debugging story: corrupt packets in AF_XDP; a kernel bug or user error?

A race condition in the virtual ethernet driver of the Linux kernel led to occasional packet content corruptions, which resulted in unwanted packet drops by one of our DDoS mitigation systems. This blogpost describes the thought process and technique we used to debug this complex issue.

bug corrupt ddos ddos mitigation debug debugging deep dive driver error ethernet issue kernel kernel bug led linux linux kernel mitigation networking packet packets process race condition story systems thought user error virtual