ETHUSD Orderbook Issue

Incident Report for BitMEX

Postmortem

Between 09:25:54 UTC and 09:44:30 UTC 24 June 2019 the orderBookL2, orderBookL2_25, orderBook10, and quote realtime websocket feeds for ETHUSD were in a degraded state. During this period, the state of the ETHUSD orderbook on these feeds was incorrect.

We were able to identify and resolve the root cause of the issue within a minute of detection. The issue was caused by a rare sequence of order events that triggered a bug in an optimisation of the orderBookL2 calculation which had been deployed to the production environment several hours earlier. This change has since been reverted.

There was no impact to orders in the trading engine itself – just the presentation of the calculated orderbook for ETHUSD downstream of the trading engine.

We have deployed additional automated feed validators to detect potential similar issues in the future and to alert us earlier.

Posted Jan 14, 2020 - 20:37 UTC

Resolved

The issue has been resolved. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

Posted Jun 24, 2019 - 19:36 UTC

Investigating

We have identified that the orderbook for the ETHUSD product is an incorrect state. We apologise for any inconvenience this may be causing. We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

Posted Jun 24, 2019 - 19:35 UTC

This incident affected: BitMEX.com (REST API).

Site Announcement