Trading Engine Interruption
Incident Report for BitMEX
Postmortem

At 05:48:10 UTC and 06:10:10 UTC, BitMEX experienced minor outages for approximately 30 seconds whereby requests to the trading engine were load-shed as the engine was busy. During these times, clients would have observed a lack of updates over the WebSocket API for the same reason. The outages were due to data replay complications during a regularly scheduled market data distribution component restart.

There was no data loss during these events and an additional safety mechanism to prevent a similar situation from impacting the trading engine has already been deployed. The root causes have also been identified and we are currently working on permanent fixes to prevent a recurrence. Updates will follow in the future.

On 15 Jan 2019, we successfully released an enhancement to our internal market data distribution component's re-subscription logic. This addresses the root cause of the previous week's issue and along with the additional safety mechanism to prevent impact to the trading engine deployed at the time, we don't anticipate a reoccurrence of last week's issue.

Posted Jan 15, 2020 - 00:53 UTC

Resolved
The issue has been resolved. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.
Posted Jan 09, 2019 - 06:10 UTC
Investigating
We are observing a series of short trading engine interruptions. We apologise for any inconvenience this may be causing. We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.
Posted Jan 09, 2019 - 05:48 UTC
This incident affected: BitMEX.com (Trading Engine).