Interruption of the Trading Engine
Incident Report for BitMEX
Postmortem

Between 04:52:11 UTC and 04:53:47 UTC on 20 December 2019, the trading engine stopped all operations as a result of an infrastructure related issue. No data was lost during this time, and upon resolution 95 seconds later the trading engine continued normal operations.

As there was no activity in the trading engine during this time, there were no data updates published via the Websocket API. During this 95 second period, requests made via the REST API (e.g. new/amend/cancel order requests and queries for public or private data) returned an HTTP status 500 response (server side error). Similarly, new subscription requests via the Websocket API during this time returned an error.

The infrastructure issue was identified shortly before the incident via our internal alerting and resolved by 04:53:47 UTC. Root cause of the issue has been identified and steps implemented to prevent a similar issue from recurring. We have improved our infrastructure monitoring to detect and alert about potential issues much sooner.

Posted Jan 14, 2020 - 19:38 UTC

Resolved
The issue has been resolved. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.
Posted Dec 20, 2019 - 04:53 UTC
Investigating
We are observing the interruption of the trading engine. We apologise for any inconvenience this may be causing. We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.
Posted Dec 20, 2019 - 04:52 UTC
This incident affected: BitMEX.com (Trading Engine).