TL;DR: cosmic ray bit flips
- Cosmic rays cause single-event upsets (SEUs) that flip bits in memory at a rate of roughly one per gigabyte per month at sea level. At planet scale, this is not a rare curiosity.
- Famous bit-flip incidents include a Belgian election voter miscounted by 4,096 votes (2003), and the Super Mario 64 speedrun 'Upwarp' skip caused by a single flipped bit. Both are consistent with cosmic-ray SEU probability at exposure duration.
- At web-application scale, cosmic-ray bit flips are one of the small residual causes of unreproducible bugs. ECC memory, checksummed storage, and idempotent request handling are the practical mitigations.
Short on time? See cside's in-browser Magecart and skimmer blocking. It covers everything below in one deployment.
In 2013, competitive Super Mario 64 speedrunner DOTA_Teabag was running through Tick Tock Clock, a level infamous for punishing even the best of players due to a challenging sequence of jumps. While making their way through, Mario suddenly shot directly upwards through the floor, bypassing a full section of the level and saving DOTA_Teabag precious seconds.
Super Mario 64 Bitflip Incident video
TTC Upwarp: Ceiling Warp vs Byte Change
The speedrunning community, which had been playing and dissecting Super Mario 64 for decades, was stunned. A bounty was offered to reproduce the glitch; some speedrunners claimed that DOTA_Teabag had "rocked the cartridge" (a way to sometimes predictably glitch a Nintendo 64 game by slightly modifying the angle of the game cartridge). The mystery came to a close when YouTuber and esteemed video game glitch hunter pannenkoek12 (translated 'Pancake12') discovered that a single bit in the Nintendo 64's memory was the cause. The bit flip happened at 0xC5837800, the address responsible for Mario's vertical position. The change in binary was from 1100 0101 to 1100 0100, and by extraordinary luck, the exact change needed to warp Mario onto a higher platform.
So what caused this in the first place? All signs point to a cosmic ray bit flip, a random error in computer chips caused by a high-energy particle from space that happens to affect a system right at a particular time.
What is a cosmic ray bit flip?
A cosmic ray bit flip (also called a "single-event upset", or SEU) occurs when a stray ionizing particle from a cosmic ray in space strikes a single memory cell or transistor, flipping a binary bit from 0 to 1, or 1 to 0. These bit flips can cause a soft error, which leaves no physical damage to the hardware, but they can alter data stored in memory or logic adjusting application behaviours or even disabling security features.
The Super Mario 64 speedrun isn't the only affected documented case of a cosmic event having on-Earth effects. During the 2003 Belgian election, an unknown candidate received exactly 4,096 extra votes seemingly out of nowhere on the country's electronic voting machines. The error was only caught because she received more votes than the number of voters, making it mathematically impossible.
Upon investigation, researchers concluded that a cosmic ray had flipped single bit at position 13 in the voting machine's memory, giving the candidate 4,096 extra votes [1]. In binary, each bit doubles the value, so the bit at that position was computed to be 2^12 - or exactly 4,096.
Rare doesn't mean impossible, especially at scale
These cosmic ray bit flips are incredibly rare on a per-bit basis, but they're not impossible. In 1996, researchers at IBM estimated that a typical desktop with 256 megabytes of RAM could experience one cosmic-ray-induced bit flip per month. Fast forward to today, where modern systems commonly run with 16 gigabytes of RAM (62.5x more than the study!), and the risk scales accordingly. Combine that with millions of cosmic rays hitting the Earth every second, and it's only a matter of time before one hits the wrong bit at the wrong time. Researchers at Vanderbilt University confirmed this: their research found that an internet service provider's router farm with 25 gigabytes of memory could experience a bit flip up to every 17 hours [2].
A "one in a million" event will happen daily if you do a million-plus things a day. Here at cside, we scan roughly over 10 million scripts every 24 hours for malicious behavior. Even if a bit flip error has a one-in-100-million chance, scanning at that volume means these impossible odds are, in fact, quite possible.
While enterprise grade memory may not be subjected to cosmic ray bit flips anymore through adoption of error-code correcting memory, which can often detect and correct single-bit errors, consumer hardware isn't usually protected. In a world where millions of actions happen, 'rare' just means it's a matter of time.
Cosmic hardening and embracing resilience
Given that cosmic bit flips are a game of statistics, how can systems be built that expect the unexpected? The answer lies in engineering for fault tolerance. For example, NASA spacecraft run critical calculations in triplicate - multiple processors run the same operation, and if one disagrees due to a stray bit flip, the other two out-vote the process. Back home on Earth, ECC memory can play a role in safeguarding routers, servers, and cloud infrastructure against single-bit errors (and even sometimes detect two-bit errors!). But like everything, ECC memory has its own limitations. It doesn't typically protect the CPU register, GPU memory, or network buffers. Even in enterprise environments equipped with ECC memory, multi-bit flips or logic-level faults can go undetected. That's why a layered resilience strategy matters. At cside, we apply the same philosophy to client-side security. Even though a malformed script or edge-case exploit may be rare individually, scanning millions of JavaScript files daily means they're not rare in aggregate.
At scale, cosmic ray bit flips are a matter of probability, not certainty. A one-in-a-million event becomes inevitable once a system runs billions of operations, whether that system is a database, an election machine, or a client-side browser. On the client side, unpredictability scales with volume, so security systems need to assume something will eventually fail, and be built to catch it when it does.
References:
