Cloudflare has pinpointed the cause of Tuesday’s major outage, revealing it began with an internal database permission change that went awry. This change forced a critical system file to swell to twice its intended size, momentarily crippling the company’s network and knocking countless websites offline.

Cloudflare’s brief outage caused a widespread internet outage on Tuesday, 18 November, taking down a swathe of major online platforms. The incident, which impacted services from ChatGPT and Claude to X (formerly Twitter), originated from a critical error within the infrastructure of Cloudflare, a company that provides security and performance services for a vast portion of the web.

In a detailed post-incident report, Cloudflare co-founder and CEO Matthew Prince confirmed the outage was not the result of a cyberattack, but rather an internal misstep. The chain of failure began with a routine change to a database system’s permissions. This change caused the database to erroneously output duplicate entries into a crucial feature file used by Cloudflare’s Bot Management system.

Part of the reason Cloudflare took so long to resolve yesterday’s outage was that it had initially misdiagnosed the problem as a cyber attack.

“The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind,” co-founder and CEO Matthew Prince said in a post on Cloudflare’s website.

“Instead, it was triggered by a change to one of our database systems’ permissions, which caused the database to output multiple entries into a ‘feature file’ used by our Bot Management system,” Prince explained.

“That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.”

This feature file, which is essentially a rulebook that helps distinguish human users from malicious automated bots, subsequently doubled in size. As per its design, this now-oversized file was rapidly propagated across Cloudflare’s entire global network. However, the software on their servers had a built-in size limit for this specific file, a limit that was now exceeded. This caused the critical software responsible for routing traffic to fail, leading to the global disruption.

The outage highlighted the fragility of the modern internet’s interconnected infrastructure. While sites like X loaded, user feeds remained empty and posting was impossible. AI platforms like OpenAI’s ChatGPT became unresponsive, and even outage-tracking service Downdetector was knocked offline, ironically unable to report on its own downtime.

The resolution involved halting the propagation of the bloated file and rolling back to a previous, stable version. Core traffic was largely restored by 16:30 SAST, with all systems reported as fully normal by 19:06 SAST.