Outages that made Microsoft Azure and a number of Microsoft cloud companies broadly unavailable for 90 minutes on Jan. 25 could be traced to the cascading results of repeated, fast readvertising of BGP router prefixes, based on a ThousandEyes evaluation of the incident.
The Cisco-owned community intelligence firm traced the Microsoft outage to an exterior BGP change by Microsoft that affected service suppliers. (Learn extra about community and infrastructure outages in our high 10 outages of 2022 recap.)
A number of Microsoft BGP prefixes had been withdrawn fully after which virtually instantly readvertised, ThousandEyes mentioned. Border gateway protocol (BGP) tells Web site visitors what path to take, and the BGP best-path choice algorithm determines the optimum routes to make use of for site visitors forwarding.
The withdrawal of BGP routes previous to the outage appeared largely to influence direct friends, ThousandEyes mentioned. With a direct path unavailable through the withdrawal durations, the subsequent finest out there path would have been by means of a transit supplier. As soon as direct paths had been readvertised, the BGP best-path choice algorithm would have chosen the shortest path, leading to a reversion to the unique route.
These re-advertisements repeated a number of instances, inflicting vital route-table instability. “This was quickly altering, inflicting quite a lot of churn within the world web routing tables,” mentioned Kemal Sanjta, principal web analyst at ThousandEyes, in a webcast evaluation of the Microsoft outage. “Consequently, we will see that quite a lot of routers had been executing finest path choice algorithm, which isn’t actually an inexpensive operation from a power-consumption perspective.”
Extra importantly, the routing adjustments brought on vital packet loss, leaving prospects unable to succeed in Microsoft Groups, Outlook, SharePoint, and different purposes. “Microsoft was volatilely switching between transit suppliers earlier than putting in finest path, after which it was repeating the identical factor once more, and that is by no means good for the shopper expertise,” Sanjta mentioned.
Along with the fast adjustments in site visitors paths, there was a large-scale shift of site visitors by means of transit supplier networks that was tough for the service suppliers to soak up and explains the degrees of packet loss that ThousandEyes documented.
“Given the recognition of Microsoft companies comparable to SharePoint, Groups and different companies that had been affected as a part of this occasion, they had been most probably receiving fairly giant quantities of site visitors when the site visitors was diverted to them,” Sanjta mentioned. Relying on the routing know-how these ISPs had been utilizing – for instance, software-defined networking or MPLS site visitors engineering enabled by the network-control protocol RSVP – “all of those options required a while to react to an inflow of a considerable amount of site visitors. And if they do not have sufficient time to react to the inflow of huge quantities of site visitors, clearly, what you are going to see is overutilization of sure interfaces, in the end leading to drops.”
The ensuing heavy packet loss “is one thing that will undoubtedly be noticed by the shoppers, and it will replicate itself in a extremely poor expertise.”
As for the reason for the connectivity disruptions, ThousandEyes mentioned the scope and rapidity of adjustments point out an administrative change, probably involving automation know-how, that brought on a destabilization of world routes to Microsoft’s prefixes.
“Given the rapidity of those adjustments within the routing desk, we expect that a few of this was attributable to automated motion on the Microsoft facet,” Sanjta mentioned. “Basically, we expect that there was sure automation that kicked in, that did one thing that was surprising from a traffic-engineering perspective, and it repeated itself a number of instances.”
The majority of the service disruptions lasted roughly 90 minutes, though ThousandEyes mentioned it noticed residual connectivity points the next day.
What Microsoft has mentioned concerning the outage
Microsoft mentioned it’ll publish a ultimate post-incident assessment of the incident with extra particulars, probably inside the subsequent two weeks, after it finishes its inside assessment.
Primarily based on what Microsoft has mentioned to date, a community configuration change brought on the outage, which it first acknowledged in a tweet at 7:31 AM UTC on the Microsoft 365 Standing Twitter account: “We’re investigating points impacting a number of Microsoft 365 companies.”
Roughly 90 minutes later, the Twitter account posted: “We’ve remoted the issue to networking configuration points, and we’re analyzing the most effective mitigation technique to deal with these with out inflicting further influence.” And at 9:26 UTC: “We have rolled again a community change that we consider is inflicting influence. We’re monitoring the service because the rollback takes impact.”
Microsoft shared extra particulars in a preliminary post-incident assessment printed through its Azure standing web page.
“Between 07:05 UTC and 12:43 UTC on 25 January 2023, prospects skilled points with networking connectivity, manifesting as lengthy community latency and/or timeouts when trying to connect with assets hosted in Azure areas, in addition to different Microsoft companies together with Microsoft 365 and Energy Platform. Whereas most areas and companies had recovered by 09:00 UTC, intermittent packet loss points had been totally mitigated by 12:43 UTC. This incident additionally impacted Azure Authorities cloud companies that had been depending on Azure public cloud.”
A change made to the Microsoft WAN impacted connectivity, Microsoft decided:
“As a part of a deliberate change to replace the IP deal with on a WAN router, a command given to the router brought on it to ship messages to all different routers within the WAN, which resulted in all of them recomputing their adjacency and forwarding tables. Throughout this re-computation course of, the routers had been unable to appropriately ahead packets traversing them. The command that brought on the difficulty has completely different behaviors on completely different community gadgets, and the command had not been vetted utilizing our full qualification course of on the router on which it was executed.”
Copyright © 2023 IDG Communications, Inc.
