After partial outages in the past couple of weeks, a CenturyLink IP misconfiguration wreaked havoc across the global network and knocked out multiple providers, including Cloudflare, Amazon, Reddit, Hulu and others.
Internet service provider CenturyLink Opens a new window suffered a major outage on Sunday in the U.S. that affected widely used online services. The outage, which struck down Cloudflare, the PlayStation Network, EA, Xbox Live, Reddit, Blizzard, Steam, Amazon, Hulu, Duo Security, and others, was caused by a problem of a third-party transit provider — CenturyLink. The problem caused HTTP 5xx class errors like HTTP 522, 502, 503, according to the status update pageOpens a new window of Cloudflare, which took down Discord, Feedly, League of Legends among others.
The problem was traced back to a CenturyLink data center in Ontario, Canada, that was misconfigured with an incorrect Flowspec update. CenturyLink provides video, voice and core IP services for several carriers across the globe. The misconfigured data center radiated outwards, causing a chain reaction that hamstrung services for millions of users globally.
See Also: It’s The Week of Outages: Users Face Issues As Slack Goes Down
Cloudflare explainedOpens a new window , “Flowspec is an extension to BGP, which allows firewall rules to be easily distributed across a network, or even between networks, using BGP. Flowspec is a powerful tool. It allows you to efficiently push rules across an entire network almost instantly. It is great when you are trying to quickly respond to something like an attack, but it can be dangerous if you make a mistake.â€
John Graham-Cumming, Cloudflare CTO told CNNOpens a new window , “Today we saw a widespread Internet outage online that impacted multiple providers. This was not a Cloudflare-specific outage. Level 3/CenturyLink was responsible for an outage that affected many internet services, including Cloudflare.â€
An analysis of the incident by Cloudflare deemed it to be one of the biggest internet outages in recent times. Cloudflare observed a 3.5% drop in global internet traffic during the CenturyLink outage, which is one of the six major tier-1 networks. Cloudflare said CenturyLink is one of the network providers they connect to apart from Cogent, NTT, GTT, Telia, and Tata.Â
A comparison between the six before, during, and after the outage shows the drop the CenturyLink network suffered:
Source: CloudflareOpens a new window
See Also: Google Issues a Fix for Gmail Outage, but Offers No Explanation
Restoration of services could be achieved only after seven hours of effort, and the telco apprised other tier 1 providers of the situation, to de-peer, and to re-route incoming traffic from CenturyLink. Successful service restorationOpens a new window was achieved by 2:58 PM ET on Sunday.
Let us know if you liked this news on LinkedInOpens a new window , TwitterOpens a new window , or FacebookOpens a new window . We’d love to hear from you!