- An infinite loop bug in Solana’s JIT cache caused transactions to stall, triggering a 5-hour network outage on February 6.
- Developers had previously identified this bug and patched one trigger but missed a second trigger that ended up causing the outage.
- Solana has a history of outages, with 9 major incidents totaling over 150 hours of downtime since September 2021, impacting network reliability.
Solana Labs recently published a detailed report explaining the cause of the major Solana outage on February 6th, which halted blockchain transactions for five hours. The outage was triggered by a known bug that developers had only partially fixed prior to the incident.
The Technical Cause
The report reveals that a deploy-evict-request cycle of a legacy loader program triggered an infinite recompile loop in Solana’s Just-In-Time (JIT) cache. In simpler terms, the transaction process entered an infinite loop due to the bug. The issue affected version 1.17 of Solana’s validator client, which accounted for 95% of the network’s stake at the time, causing most validators to stall.
A Recurring Bug
Interestingly, developers had identified the same bug as the cause of an outage on Solana’s devnet one week prior. They had patched one possible trigger but missed the second trigger that ended up causing the mainnet outage. Solana Labs says they plan to release a more comprehensive fix in the future.
A History of Outages
The incident continues Solana’s notorious trend of frequent outages, which occurred regularly in its early days. According to CryptoManiaks, Solana has had 9 major outages since September 2021 totaling over 150 hours of downtime, excluding this latest 5-hour incident. Prior to this, Solana’s previous outage occurred in February 2022 and lasted 18 hours.
Market Impact and Response
SOL prices dropped to a six-day low of $9.375 during the outage but have since recovered. The token was trading at $10.546 at press time, up 24% in the past 24 hours. The Solana community appears to have moved on from the temporary network disruption.
Conclusion
While frustrating for users, Solana’s latest outage stems from an ongoing challenge of balancing innovation with network stability. The investigation provides helpful insight into the difficulty of eliminating bugs on rapidly evolving blockchain platforms. As Solana continues to refine its design and quality assurance processes, the recurrence of such incidents is likely to diminish over time.
