High API Latency

Incident Report for Discord

Resolved

The incident has been resolved.
Posted Sep 29, 2017 - 08:30 PDT

Update

Latencies and resize times look to have returned back to normal now.
Posted Sep 29, 2017 - 08:01 PDT

Monitoring

We have been able to provision more capacity and service should be recovering for API latency. Images backlog is still being processed, and we expect another 10-15 minutes until everything is all caught up.
Posted Sep 29, 2017 - 07:57 PDT

Update

We have isolated the issue to our autoscalers for our API and Media Proxy services being unable to provision more instances to handle the normal load increase that happens during the day. We are working with Google Compute to resolve the issue.
Posted Sep 29, 2017 - 06:54 PDT

Update

We are also aware of images failing to load and be resized with our image proxy, causing things like uploads to appear to fail.
Posted Sep 29, 2017 - 06:37 PDT

Investigating

We are aware of elevated API latency and are working on resolving the issue.
Posted Sep 29, 2017 - 06:28 PDT