Things look calm, all alarms have resolved on our end. We are going to close this out and continue our march towards Python 3 tomorrow... thanks for your patience, and our apologies for the interruptions!
Posted 6 years ago. Jul 10, 2019 - 22:19 PDT
Monitoring
We've deployed fixes to the relevant systems and are monitoring. At this point, we believe everything is resolved and will not recur.
Posted 6 years ago. Jul 10, 2019 - 22:03 PDT
Update
We have rolled back a problematic change to our workers (machines that run periodic/on-demand tasks) that were getting duplicated and causing stress to our underlying databases. We are still working through some downstream problems this has caused in our search system.
Posted 6 years ago. Jul 10, 2019 - 21:46 PDT
Identified
We are having some intermittent issues with sending messages and logging in. We know what the cause is and we're working to remediate the issue now.