Messages not loading/sending

Incident Report for Discord

Resolved

The database repair operation attempted earlier has completed without issue - and we're noticing no further issues at this time.

Posted Aug 10, 2016 - 02:51 PDT

Monitoring

A wild database repair caused Cassandra to start thrashing it's GC, causing STW GC pauses for upwards of 7+ seconds, increasing message send and load latency while the cluster was paused for GC. We've fixed the issue by doing a rolling-restart of the cluster - allowing the heap to start over new. Things seem to be stable now. We're monitoring.

tldr: we turned it off and back on again until it worked.

Posted Aug 10, 2016 - 01:53 PDT

Investigating

We're looking into issues with messages failing to send or load.

Posted Aug 10, 2016 - 01:20 PDT