Wednesday's Bitfinex Data Issue Recap
On Wednesday, April 20 at 3:45 PM CST, we were alerted by a user that our system had errantly triggered several alerts on the Bitfinex BTC/USD pair. After quick analysis, it was discovered that our primary market history table showed several trades as executed at a price many deviations lower than the average trading range. Our backend systems performed as normal and immediately fired alerts based on the errant data, causing users to falsely believe the price had sunk to new lows and thus triggering a maelstrom of reports of errant alerts on twitter and in Coinigy chat.
After doing forensic analysis on our websocket daemon, polling daemon, and internal databases, we strongly suspect that we received some errant data via Bitfinex’s websocket feed. We maintain a live connection to Bitfinex via a custom nodejs websocket daemon. This daemon in particular has been running without issue for months, save for the occasional server refresh. In addition, we run a backup polling method that captures data via Bitfinex’s REST API. We are able to rule out the data coming in via REST as there’s a special flag in our database to denote which client the data came from- All data points to these trades coming in via WS.
At a granular level, it appears as though somewhere close to 300 trades came across websockets via incorrect channels. Our websocket daemon is essentially just a passthrough for the data sent by the Bitfinex API. We parse the trade id, market, price, quantity and timestamp directly from the stream and insert it into our database (as well as push the data out via our own pub/sub websockets API). We attempted to cross-reference against other sites, and we were evidently the only ones showing this errant data. After double-and-triple-checking the source code, we have come to the conclusion that the only way for our system to have written these trades to the wrong market, is if we had been sent this data directly, but under the incorrect channel ID.
We are continuing to explore edge cases to identify any possibility of a patch or exception handling to prevent this in the future. We’re suspecting that during Bitfinex’s update we potentially lost connection for a moment, and the channel IDs changed without triggering a resubscribe event. In addition, we were not yet subscribed to any “ETH” channels via websockets, yet we somehow received ETH data through these errant bursts. The odd thing is that the issue appeared to “fix itself”, so immediately after the bursts trades were received through the incorrect channel, the correct trades then continued coming through the same channels. Unfortunately we do not have full, direct logs of what was sent to us at the JSON level, but we do have all parsed and posted data.
We proceeded to scrub our tables of these errant trades, identified by a foreign tradeid format and price, and restored our aggregate OHLCV charts. A few screengrabs were taken of the errant trades prior to scrubbing, pictured below is our LTC/USD market which evidently was passed trades from ETH/USD, with two separate “tradeid” formats. Other markets displayed similar information (BTC/USD had data from LTC/BTC, ETH/BTC, etc).
It’s also (perhaps coincidental) that Bitfinex deployed an update right around the same time we experienced these issues: https://www.reddit.com/r/BitcoinMarkets/comments/4flknr/daily_discussion_wednesday_april_20_2016/d2au56n
As a result of this mixup, several users’ Price Alerts have been fired errantly. We pride ourselves on data integrity here at Coinigy and we sincerely apologize if these alerts crossed your path and caused any discomfort (and thank you to those of you who alerted us immediately). If anyone who interfaces with the Bitfinex websocket API experienced a similar issue, we’d love to hear your workaround as well!
We will continue to research this situation and amend this blog post as we learn more, as well as work on failsafes to prevent this from potentially happening again in the future.