At around 13:00 PST - we noticed an increase in message send latency, at this time we narrowed down the issue to be affecting a cassandra node that was maxing out CPU load within our messages cluster - affecting messages sent in channels that were on that given token range. We believe this bug to have been related an issue between cassandra 2.2.3 causing it to spin CPU - and be starved for nice time when running normal compactions. Doing a rolling upgrade on the cluster to 2.2.8 remidied this issue. We had been investigating this issue for the past few days - and in addition to upgrading the cluster, we plan to increase capacity by bootstrapping new nodes and further tune the configuration of the cluster.

At around 16:00 pm, we noticed an increase in latency again, this time due to another cassandra cluster failing. We identified the root cause of this issue to be a bias towards a specific node due to incorrect configuration within the cluster, causing a hotspot and degradation of service due to one of the nodes being maxed out in CPU. In order to fix this issue, we needed to expand the cluster to add more compute resources to get rid of the hotspot on the specific node. While bootstrapping the new node, a configuration issue caused a cascading failure across the rest of nodes - causing the cluster to be unavailable for a few minutes while we quickly reverted the configuration changes. After fixing the configuration issue, we began bootstrapping the new node again. Between 17:30 and 18:00 while the new node was bootstrapping, due to the existing hotspot, one of the nodes in the cluster struggled to serve requests (due to the initially mentioned misconfiguration) causing a period of increased latency. At around 18:00 the new node had bootstrapped and was ready to serve traffic - causing the latency to normalize again as we were able to get rid of the hotspot.

Although this wasn't a major outage, service was still degraded during this time, causing message send delays & failures, and excessive load times in Discord. Many lessons have been learned, and we have a list of things we will be doing over the next few days to improve our infrastructure to hopefully minimize the chances of service degradation like this happening again.