Websocket API Latency
Incident Report for BitMEX
Postmortem

Between 21:30 UTC 25 November 2019 and 07:05 UTC 26 November 2019 the Websocket API was running with degraded capacity. This resulted in slightly more latent feeds during this time and some isolated occurrences of substantial lag during traffic spikes correlated with large market moves.

The issue was identified when latency thresholds were breached in our automated monitoring systems around 06:55 UTC. From 07:05 the issue was resolved and full capacity was restored. We continue to monitor the impacted services closely.

The issue was caused by incorrect CPU pinning following a release of a market data distribution component at 21:30 UTC 25 November 2019. The impact of this was only observed during large traffic spikes which occurred several hours later and so was not identified during the post release checks at the time. The configuration of this service has been corrected and the deployment tested to prevent this from happening again.

Posted Jan 14, 2020 - 20:07 UTC

Resolved
The issue has been resolved. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.
Posted Nov 26, 2019 - 07:05 UTC
Investigating
We are observing a slight increase in latency in the feeds data transiting through the websocket API. We apologise for any inconvenience this may be causing. We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.
Posted Nov 25, 2019 - 21:30 UTC
This incident affected: BitMEX.com (REST API).