Bots shut down the Solana network for seven hours. The chaos is over, an investigation has been conducted, the causes have been discovered, and actions to stabilize the network are being taken.
On the fateful Saturday night of April 30 — May 1, something strange and frightening happened. Thousands of thousands of bots flooded the proof-of-stake Solana network with huge garbage packets of data. Validators were knocked out of the network, the production of new blocks stopped and the chaos continued for seven hours.
Earlier on Saturday, bots flooded the NFT Minting Tool “Candy Machine” and flooded the network with an unprecedented amount of inbound traffic. Four million transaction requests and 100 gigabits of data every second, according to data from the Jump Crypto Engineering tweet:
Horde of bots flooded the network and pushed validators out of consensus. Subsequent block production became impossible and the network completely shut down at 4:32 p.m. ET. Thanks to the coordinated actions of the validators, the cluster restarted at 11 p.m. ET at slot 131973970. Solana wrote about the successful restart on Twitter:
Also, one of the validators created a Google Doc, which described the actions to restart the network. You can read it here.
The chaos has calmed down. Solana slowly began to recover. Metaplex, the key steward of Solana NFT infrastructure and one closely intertwined with Candy Machine, tweeted that they would introduce 0.01 SOL “botting penalty” to help NFT projects handle colossal traffic. Penalty “will be collected when a wallet attempts to complete an invalid transaction, which is typically done by bots that are blindly trying to mint.”
According to CoinGecko, the Solana token significantly sank against the background of Saturday’s accident, reaching a daily minimum of $ 85.02, but later, the price rose.
But, by the next weekend, the price fell much more significantly (below $ 80), which can also be attributed to the seven-hour blackout.
On May third, Solana developers published a detailed analysis of what happened, which describes the possible causes of the failure, “and the steps core developers are taking to increase network resilience and stability.” This was announced on Twitter by Solana Status:
The reasons for the network shutdown were explained by the developers as follows: “An enormous amount of inbound transactions (6 million per second) flooded the network, surpassing 100 Gbps of traffic at individual nodes. There is no evidence of a denial of service attack, but instead evidence indicates bots tried a programmed win of new NFT being minted using the popular Candy Machine program. Since the mint price had a fixed floor and not a dynamic Dutch auction, the first user to call the mint received the NFT, which created an economic incentive to send a huge number of transactions in hopes of winning the mint”.
Also, the developers said that the specific reason of stopping the consensus was “due to validators running out of memory and crashing. The root cause of the high memory usage was insufficient votes landing to finalize earlier blocks, preventing abandoned fork clean up. The number of forks validators had to evaluate exceeded their capacity to do so, even after a reboot, necessitating manual intervention.” Where the bots came from, the report does not specify
It is worth noting that this is not the first outage in the Solana network this year. According to the Solana status tracker, nine crashes of varying seriousness occurred in January only. Apparently, something has been going wrong with Solana since the beginning of the year. Well, I can only wish the developers success and continued vigilance.
