We now have confidence that the event is resolved.
Posted 3 months ago. Jan 08, 2018 - 14:15 PST
The database resource update took much less time than anticipated, and response-times are returning to normal. We are continuing to monitor the infrastructure to ensure backpressure clears appropriately.
Thank you for your patience.
Posted 3 months ago. Jan 08, 2018 - 13:04 PST
We have high confidence that this issue is related to recently deployed patches for Meltdown and Spectre applied on the part of our database vendor. We were aware that we would accept some performance hit from this, but had no way to anticipate the exact percentage. Monday has turned, and Monday is proving to be a worst-case scenairo. We now know what that percentage is, and it is too much.
We will be taking steps to upgrade our capacity in our database tier. This will not be fast work, so this period of decreased performance will likely extend at least another hour, possibly as long as three. We will continue to provide updates.
HelloWorks is impacted by this event, but not to the same extent.
Posted 3 months ago. Jan 08, 2018 - 12:22 PST
We have identified an extensive set of delays in loading our Web and API endpoints. Due to the severity of the delays, API calls may time out before returning.