Incident Post Mortem: November 19, 2021

Abstract

On November 19, 2021, Coinbase learned that it had erroneously charged some customers transacting in GYEN and POWR either 100x or 1/100 of the amount they had purchased. Coinbase immediately disabled trading in POWR and GYEN, worked around the clock to resolve the underlying technical issue, and then made adjustments to customer accounts to reflect how much GYEN and POWR customers had actually purchased. This incident affected approximately 0.0072% of all approved Coinbase users.

What happened?

On November 19 at approximately 4:00 PM EST, Coinbase updated an internal data source regarding POWR and GYEN accuracy. The update has been tested with our standard automated testing procedures and deployment monitoring. However, testing did not find that the update would spread at different speeds through a number of internal systems and result in customers receiving 100x or 1/100 of the amount of GYEN or POWR they purchased.

The data propagation error was identified by our Site Risk Monitoring systems shortly after the EST 19 4:00 PM update. At 5:35 PM EDT, we disabled transactions in GYEN and POWR pending resolution of the underlying issue. At 7:26 p.m. EDT, we identified accounts that were handled in GYEN or POWR during the release of data, and we temporarily restricted those accounts pending further investigation. By November 21, 98.8% of these accounts had their restrictions removed, and by December 13, Coinbase had fully restored GYEN and POWR trading.

What did Coinbase do to correct the problem?

Coinbase immediately dedicated significant engineering resources to quickly fix the issue, ensuring that our customers receive the correct amount of GYEN and POWR they purchased. For clients who accidentally overpaid 100 times the GYEN and POWR they purchased, we made sure they were getting the correct amount of assets they paid for. For those who still had GYEN and POWR in their accounts, this was relatively straightforward – we reported the error to customers and simply debited those customers’ accounts, removing the additional GYEN or POWR that had been restricted in error.

Some clients have already converted GYEN and POWR into other digital assets, such as Bitcoin. Other clients sent GYEN and POWR to wallets outside the Coinbase platform, but held other digital assets on the Coinbase platform. For such customers, we have notified them of the error, and in accordance with the Coinbase User Agreement, we have withdrawn other assets from these customers’ Coinbase accounts equal to the amount of GYEN or POWR that they were over-credited.

In determining the amount debited from the accounts of these clients, we have used the exchange rate that is most favorable to our clients. Specifically, we have calculated the USD value of GYEN or POWR owed to Coinbase using minimum The exchange rate on the Coinbase exchange from the time this incident began until trading stopped ($0.00825/Jin, $0.4742/POWR). This reduced the amount owed to Coinbase by these customers. We then deducted funds from users’ accounts up to this USD value, starting with their fiat balances, then USDC and other stablecoin balances, followed by their other digital asset balances sorted by descending market capitalization. The value of these digital assets was calculated using the market price at the time of debiting the users’ accounts.

A small group of customers who were wrongly certified by GYEN or POWR sent these digital assets off the platform and left no other assets on Coinbase. Coinbase contacts these clients individually and appreciates our clients’ cooperation in returning erroneously registered GYEN and POWR. Excess deposited funds are required to be repaid under the Coinbase User Agreement.

For customers who had reduced reliance on GYEN or POWR and received less than they purchased, Coinbase first determined the amount of GYEN or POWR owed to those customers. Coinbase then calculated the USD value of GYEN or POWR owed to customers using the above The exchange rate from the start of the accident until the repair is completed ($0.009799/Gen, $0.9617/POWR), which is the most favorable exchange rate for our customers. This means that no matter what price clients bought, we assumed that clients might sell these assets at the highest price while trading was disrupted. After calculating this USD value, we deposit an equivalent amount of Bitcoin to clients. We credit these clients with Bitcoin because GYEN and POWR trading is still pending, and Bitcoin is used in every country where clients are affected.

To further benefit our customers, we used an exchange rate of $55,000/Bitcoin, which was minimum from the market price of BTC at the time of making such BTC payments. Within this exchange rate our customers get more Bitcoin than they would have if we used the actual BTC-USD exchange rate at that time.

In addition, for all customers whose accounts have been restricted, Coinbase has offered a customer experience credit of up to $100 in BTC.

what happened after that?

Many customers still have questions about how their accounts were credited or debited due to this incident. For questions about your account, feel free to contact Coinbase support.

Coinbase also reviews the information in our customers’ account statements and tax forms to correctly reflect our customers’ GYEN and POWR transactions. If your tax details or forms appear incorrect, please contact Coinbase support, but know that we are working to correct this information as well.

GYEN values ​​before the data subtraction error.

In the days prior to the data release error, between November 16 and November 19, Coinbase Exchange noticed a breakout parity of GYEN-USD in comparison to JPY/USD. We’ve seen clients speculate on social media that this incident is somehow connected to this break in parity. We’ve also seen customers speculate that this break in parity was somehow caused by Coinbase. These claims are false and reflect a misunderstanding about what GYEN is and how Coinbase operates.

A parity fracture occurred prior to the accident and had nothing to do with it. This break in parity occurred days before the accident. At the height of this parity breakout, on November 17, 1 GYEN traded for approximately 7.48.

GYEN (blue) price in yen (red) during parity break, with a high of 7.48 yen (red line indicates when the data was subtracted error)

By the time the data release went wrong, on November 19, GYEN price stability had recovered and GYEN was trading around 0.96-0.98. When unrestricted Coinbase affected customer accounts, GYEN was trading at approximately 0.98 yen. In other words, the parity break occurred before the data-publishing error and the two issues had nothing to do with each other.

How did the equivalence break happen? When Coinbase listed GYEN, there was a huge demand for GYEN that the supply could not match. The surge in buyer demand for GYEN, along with the insufficient supply of GYEN in all markets (not just Coinbase), eventually led to the collapse of parity. From November 17 through November 19, Coinbase implemented an alert informing its customers who were buying, selling and trading GYEN of “Extraordinary Market Activity – Due to unusual market activity for GYEN, you may have trouble trading GYEN on Coinbase.com. We apologize for any inconvenience this caused. “. The parity break occurred due to these market conditions for the GYEN digital asset unrelated to Coinbase operations.


Incident Post Mortem: November 19, 2021 was originally posted on the Coinbase Blog on Medium, where people continue the conversation by highlighting and responding to this story.

Leave a Comment