Reliably Processing Trillions of Kafka Messages Per Day

Authors: Vilas Athavale, Ravinder Matte, Sid Anand, Shrity Verma, Naresh Gopalani, Bhaven Avalani

Summary

Walmart deploys Apache Kafka, with 25K+ Kafka consumers, across multiple clouds (public and private). Supporting business critical use cases including data movement, event driven microservices and streaming analytics. These use-cases demand 4 nines (i.e., 99.99) of availability and require us to quickly drain any backlogs arising from sudden traffic spikes. At Walmart scale, we have a diverse set of Kafka consumer applications written in multiple languages. …

analytics apache apache kafka authors availability business cases clouds consumers critical data demand event kafka messages microservices private public sid streaming use cases