SREcon23 Americas - We're Still Down: A Metastable Failure Tale

We're Still Down: A Metastable Failure Tale

Kyle Lexmond

""The status? The system has been down for hours, and we haven't been able to get it back up yet""—words on an incident conference call that you probably don't want to hear.

This talk explores how a globally distributed CDN experienced a metastable failure, design changes that make future failures less likely, and the unorthodox fix that made a recovery possible (and can hopefully apply to future metastable failures—maybe even yours). …

americas back back up call cdn conference design distributed don down fix future incident recovery system