A software defect deployed in July proved to be the root cause of last week's widespread Microsoft 365 outage, Microsoft disclosed in a preliminary incident report. The bug triggered a cascading infrastructure failure that rendered more than 80% of the company's networking gateway servers unresponsive during the nearly 24-hour disruption on November 12.

The outage, which began at 11:20 AM ET on November 12 and persisted until 11:10 AM ET on November 13, disrupted access to Outlook, Exchange Online, Teams, and Copilot globally. Users faced email delivery delays and were unable to reach their mailboxes throughout the incident.

image

Microsoft identified three contributing factors in its preliminary report: the undetected software bug, a surge in traffic following a Microsoft Teams failover event, and a concurrent reduction in available server capacity. The combination of these conditions created the perfect environment for the defect to manifest at scale.

How the Bug Remained Hidden

The problematic update had been running in production for more than three months without triggering failures. However, the underlying defect could cause servers to become unresponsive when subjected to specific traffic patterns and load conditions.

Microsoft's testing procedures failed to catch the issue because the lab environment did not replicate production conditions. According to the company's report, "Our normal load testing did not detect the latent issue because the traffic mix and SKUs used in our labs" differed significantly from production systems.

The company further explained that "The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on weaker hardware with fewer cores." This hardware disparity between testing and production environments allowed the defect to escape detection.

The Trigger Event

The bug was activated when Microsoft Teams automatically failed over to the networking infrastructure supporting Exchange Online approximately 15 hours before the outage began, significantly increasing traffic on those servers. A separate Teams outage occurred earlier that day, though the relationship between the two incidents remains unclear.

Adding to the strain, roughly 14% of the gateway server capacity was already offline due to an unrelated maintenance issue. While the remaining infrastructure theoretically possessed sufficient capacity to absorb this loss, the combination of elevated traffic and reduced capacity proved sufficient to trigger the latent bug.

Cascading Failure

As servers began failing, Microsoft's traffic management systems automatically redirected user connections to remaining available servers. These backup systems quickly became saturated by the redirected load and also failed, creating a cascading effect.

"In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive," Microsoft stated in its report.

Regional safeguards intended to prevent outages from spreading between geographic areas failed to contain the problem, allowing the disruption to propagate worldwide. While Teams, OneDrive, and SharePoint could switch to alternative infrastructure and recover sooner, Exchange Online lacked this flexibility, leaving users locked out of their mailboxes.

Recovery and Remediation

Recovery proved challenging because restarting affected servers caused them to immediately become overwhelmed again by accumulated traffic waiting to be processed. Once engineers identified the defect, they developed and deployed a fix across the affected fleet, which allowed services to stabilize throughout Friday.

Microsoft has since implemented additional capacity, enhanced monitoring systems, and deployed new safeguards designed to prevent traffic redirection from triggering similar cascading failures in the future.

Charles Lamanna, Microsoft's executive vice president of Copilot, Agents, and Platform, issued an apology on X: "We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations."

Microsoft indicated that its investigation continues, with a comprehensive post-incident report expected to be published later this week.