A database permission change cascaded into a six-hour global outage affecting millions of websites. There were multiple failure points that aligned like pins in a lock tumbler: a lack of graceful failover and a simple static check that would have caught a configuration file exceeding 200 features before it ever hit production.
I recently read Matthew Prince's public postmortem on Cloudflare's major outage on November 18, 2025. While it's tough to see a foundational piece of the internet struggle, the transparency is invaluable—and the failure chain offers critical lessons for every engineering team building high-availability systems.
The Incident Timeline
The core issue began at 11:20 UTC. Main impact was mitigated around 14:30 UTC, with all systems fully restored at 17:06 UTC. Nearly six hours of total incident duration.
The ripple effects were severe: pervasive HTTP 5xx errors for customers, along with failures in key services like Workers KV and Cloudflare Access. For the US market, this outage occurred from 6am to 12pm Eastern—a critical morning shopping window during the holiday sale period. For online retailers, this timing could have impacted their annual goals.
Here's the number that puts it in perspective: 346 minutes of downtime. That single incident drops Cloudflare (and any company entirely dependent on Cloudflare) below four nines (99.9%) availability for the entire year.
The 5 Whys
Understanding how this happened requires tracing the cascade from symptom to root causes. The "5 Whys" framework is ideal for this kind of analysis.
Why did customers see 5xx errors?
The core proxy Bot Management system crashed when it attempted to load its "feature file". The new proxy system (FL2) had no graceful fallback, so it returned 5xx errors.
Why did the Bot Management system crash?
The bot management feature file exceeded a fixed memory preallocation limit of 200 features. A query that normally returned ~60 features suddenly returned ~200, pushing it past the hardcoded limit.
Why did the feature file grow in size suddenly?
A change to ClickHouse database permissions explicitly made underlying tables (r0) visible. The query lacked a filter to deduplicate results, so when the underlying tables became visible, matching rows from the underlying r0 table were returned in addition to rows in primary tables.
This database change was a breaking change that impacted a core system. It should have been rolled out defensively, placed behind an opt-in flag for queries or accounts. That isolation boundary would have contained the blast radius and allowed teams to discover the duplicate-row issue before it propagated network-wide.
Why wasn't the oversized file caught before hitting production?
This is, in my view, the core operational failure: the absence of mandatory static checks on this critical configuration file.
There should have been an automated check on the file generation logic to ensure the file met static constraints—maximum size, maximum feature count—before propagation to the network. Had an automated check flagged the file for exceeding 200 features, deployment would have been stopped cold.
This is such a fundamental safeguard. The file had a known constraint. That constraint should have been enforced at the point of generation, not discovered at the point of consumption.
Why did FL2 crash while the legacy system (FL) degraded gracefully?
Cloudflare was mid-transition between two proxy engines. Customers on FL2 experienced 5xx errors. Customers still on the legacy platform (FL) were not impacted the same way—the legacy system handled the Bot Management failure more gracefully by allowing all traffic through with a bot score of zero.
That graceful degradation behavior was likely a functional requirement of the battle-hardened legacy system that didn't make it to FL2. This underscores a critical practice: comprehensive functional requirement documentation (AKA specs) for existing systems must be propagated to next-gen replacements. The absence of this documentation turned what could have been a degradation into a catastrophe.
Transparency as a Gift
I deeply appreciate Cloudflare's willingness to publish such a detailed postmortem. This level of transparency—detailing the specific database change, the resulting file duplication, and the memory preallocation limit being hit—serves as a learning opportunity for every engineering and operations team. Postmortems like this make the entire industry better.
The Diversification Imperative
The fact that it took over three hours to mitigate and nearly six hours to fully resolve this incident sends a clear signal to every organization: if your business cannot tolerate a six-hour outage, you must treat major CDN and DNS providers as infrastructure that can and will fail.
This isn't about abandoning Cloudflare or any other provider. It's about building the capability to route around failure. Companies that had the ability to quickly failover from one CDN to another—or even route around the failed service—would have been able to keep serving traffic and minimize financial damage during this critical retail season.
At minimum, this means having a manual failover capability: the ability to update DNS records to point to an alternative origin within minutes. The next level of maturity is automated failover based on health checks. The highest level is active-active multi-CDN deployment where traffic is already distributed.
Where Spec-Driven Development Fits In
This incident highlights exactly why critical resilience requirements need to be systematized, not left to individual project decisions.
If a company decides it cannot tolerate an outage of this magnitude, that decision should be encoded as a requirement in their "spec constitution"—a shared set of best practices that apply across all projects when generating and grading requirements or tasks.
Example Constitution Requirements
- Multi-origin deployment: All customer-facing services must be deployable to at least two independent CDN/origin providers.
- Graceful degradation: All services must define degraded operation modes and handle upstream failures without returning 5xx errors to end users.
- Static constraint validation: All configuration files with known size or count limits must be validated at generation time, not consumption time.
When these requirements live in the constitution, they're automatically considered for every new feature, every new service, every new deployment. Teams don't have to remember to think about resilience—it's baked into the process.
The graceful degradation requirement is particularly relevant here. Had Cloudflare's FL2 development been governed by a spec that required documenting and preserving the degradation behaviors of FL, the outcome of this incident would have been significantly different.
Conclusion
This incident is a powerful reminder that complex systems rely on layers of defense. A database permission change cascaded into a global network panic due to a missing filter, a missing static validation check, and a missing graceful degradation requirement.
The defenses that would have prevented or contained this failure are not exotic. Static validation. Functional requirement documentation. Isolation boundaries for breaking changes. These are practices any team can implement.
The question is whether those practices are systematized—encoded into your development process in a way that makes them automatic—or whether they're left to tribal knowledge and individual vigilance.
Kudos to Cloudflare for the transparency. This postmortem is required reading.
Want to get weekly insights like this in your inbox? Subscribe to the SPEQ mailing list on our homepage.
Visit getspeq.com