SREcon23 Americas - Avoiding Cachepocalypse in the Land of the Monolith

SREcon23 Americas - Avoiding Cachepocalypse in the Land of the Monolith

David Amin, Duolingo

Like many companies who switched to a microservice architecture, Duolingo runs a legacy monolith that is both business critical and not entirely owned by any one team. This monolith hit a scaling limit by exhausting memcached connections, but strangely only on a single node of the cluster. What followed was a debugging journey that spanned Python and C, took down production for an hour, and required …

americas architecture binary business cluster companies connections critical debugging down duolingo journey legacy memcached microservice node python scaling single team