GitHub's August 2026: Five incidents expose capacity challenges amid Azure migration

During August, GitHub encountered five separate incidents that degraded service performance across its platform. Despite ongoing progress in infrastructure improvements and a migration to Azure designed to expand capacity, the month presented significant availability challenges. Details on each incident appear in earlier blog coverage. The company is making substantial investments in architectural enhancements and the Azure transition to handle platform growth, yet acknowledges that risk elimination remains impossible given the scale of operations.

Each incident provided valuable learning opportunities that have driven priority adjustments. Improvements implemented as a result include enhanced capacity monitoring and management systems, refined retry policies to prevent amplification effects, and strengthened resilience across core infrastructure. Multiple ongoing work streams have also advanced significantly.

On August 11, GitHub successfully ran a production MySQL primary from Azure for the first time, with minimal client-observed write impact and zero customer disruption during the transition. The pattern repeated with two additional primaries on August 27, with further migrations scheduled in the coming weeks at increasing complexity levels as teams learn from each failover.

Read traffic reached record levels during the month. Migrated services peaked at 60.4% read traffic from Azure, while GitHub's monolith reached 64.3% in Azure. Git reads achieved 54%.

The authentication-core cohort comprising 24 tables migrated away from GitHub's oldest shared database, mysql1, eliminating approximately one million queries per second from its replicas. Additional query-hygiene optimizations removed another 120,000 queries per second and eliminated roughly 59,000 seconds of wasted database work hourly.

GitHub Actions received additional capacity as longer-term isolation efforts continue. Job-routing modifications shifted 33% of jobs from a constrained production cluster to available capacity, reducing peak cache CPU utilization from 98% to 80% and providing an estimated three months of additional headroom. This represents a near-term solution rather than a permanent fix; the August outage underscored the necessity for more robust capacity and isolation measures.

Pull request isolation efforts progressed, with authenticated reads for the first production cohort now reaching 100% in addition to unauthenticated traffic already being served.

Git overload protection investments handled 6.4% more traffic while improving 95th-percentile duration by 24% and maximum delay by 78%. Edge-level load-shedding protections also advanced, providing mechanisms to safeguard GitHub during unexpected load spikes—protections that proved instrumental in mitigating August incidents.

Monitoring and telemetry enhancements were implemented across multiple systems. Pull request monitoring now independently measures merge, review, and comment failures, preventing high read volume from masking write path issues. Beginning August 21, automated high-impact incident detection began combining customer-support signals with service telemetry. API monitoring underwent recalibration and 30-day validation, reducing noise and improving signal quality. These changes strengthen detection and response capabilities.

Upcoming work includes migrating additional database primaries, continuing service and traffic migration to Azure, addressing database health particularly on shared databases, expanding automation for capacity management and auto-scaling, and extending dependency-failure handling across more pull request functionality.

"This principle continues to guide us: availability, then capacity, then features."

August 6, 15:22 UTC (10 hours 42 minutes)

What happened?

A routine deployment to an internal GitHub Actions service that processes incoming events and converts them into action jobs initiated the incident. The deployment code itself was not responsible—rollback confirmed this—but replacing pods during the rollout temporarily reduced capacity in one site, pushing remaining sites beyond their limits as traffic redistributed. The heaviest impact occurred during the incident's middle hours, when a substantial portion of Actions workflow runs failed to start or complete.

What went wrong and why?

The affected Actions services operated near their capacity and concurrency limits. A routine deployment that briefly reduced running pod counts proved sufficient to exhaust available headroom. This triggered CPU throttling and out-of-memory restarts in service mesh sidecars, which cascaded into cache, DNS, and API errors spanning multiple clusters. The ingress service mesh for these services lacked sufficient headroom to absorb the temporary capacity loss during deployment.

As core services recovered, a latent bug in the job-assignment logic slowed recovery: runners received jobs that had already been revoked, then became stuck retrying them instead of claiming valid work, creating a self-amplifying backlog.

How did we respond?

1. A routine deployment to an internal Actions service briefly reduced running capacity in one data center, and within minutes the service mesh and remaining pods reached saturation.
2. Failures cascaded across clusters as cache, DNS, and API errors spread and Actions infrastructure failures increased. A public incident was declared, the triggering deployment identified, and rolled back to confirm its contents were not the cause.
3. Over roughly the next two hours, capacity for the saturated services was expanded and incoming webhook-triggered work was throttled to allow system stabilization.
4. With core services recovering, a large backlog of queued jobs remained. A latent bug caused runners to be assigned jobs no longer valid, then get stuck retrying them, blocking real work.
5. Fixes were deployed so runners stopped attempting to acquire invalid jobs, accumulated queues were drained, and internal rate limits that were slowing recovery were raised. Workflow success rates climbed toward normal.
6. System-wide queues drained and Actions returned to normal operation. A smaller set of self-hosted runners remained stuck and were recovered manually, and some events from the incident could not be replayed automatically and required re-triggering.

How are we making incidents like this less likely or less impactful?

• Add headroom and enable autoscaling for the service mesh ingress and the affected Actions services so a routine deployment cannot tip them into saturation.
• Make deployments safer for these services by avoiding capacity reductions during rollout.
• Strengthen monitoring for the saturation and database-proxy conditions that preceded the incident so they are caught earlier.
• Improve how the system sheds load and drains backlogged work during large Actions incidents, and prevent runners from getting stuck retrying invalid jobs.
• Ship automatic recovery for self-hosted Actions Runner Controller runners affected by this failure mode in upcoming runner and ARC releases.

August 17, 13:40 UTC (7 hours 35 minutes)

What happened?

A new traffic peak pushed one datacenter's load balancers past their limits. A service-mesh sidecar reached its concurrency limit and did not scale up.

What went wrong and why?

As requests accumulated, several of the datacenter's load-balancer nodes exhausted their network flow limits, which degraded the shared gateway authentication path and produced widespread authentication latency and failures across the many services routing through that datacenter.

A latent client retry bug substantially amplified traffic to one internal authentication endpoint, which slowed recovery for the Copilot Token Service. The core weakness involved the service-mesh sidecar failing to scale up combined with retry behavior, and clients were not sufficiently constrained to prevent a partial degradation from amplifying into broader overload.

How did we respond?

1. A new traffic peak pushes a datacenter's load balancers toward their limits; a service-mesh sidecar hits its concurrency limit and fails to scale up.
2. The overload cascades as several load-balancer nodes exhaust their network flow limits and the shared authentication path degrades; issues, pull requests, the APIs, Actions, Copilot, and other services begin returning errors and slow responses.
3. Automated monitoring detects the elevated errors and an incident is opened; the affected products are marked degraded on the public status page as engineers from across the affected services converge.
4. Engineers trace the failure to network saturation on the load balancers in a single datacenter and begin shifting some traffic to another datacenter and reducing gateway retries to relieve the pressure.
5. The team stops the load-balancer processes on the saturated nodes and blocks the retry-triggering requests to the most-affected internal endpoint, which produces broad and immediate recovery.
6. Remaining authentication errors driven by client retry amplification are stabilized by ramping traffic back up gradually, and after a sustained period of healthy telemetry the incident is resolved.

How are we making incidents like this less likely or less impactful?

• Correct autoscaling policies so they account for service-mesh sidecar concurrency and capacity, not just the host service.
• Audit request, concurrency, and scaling limits for the service mesh across the affected services.
• Review retry and backoff limits across gateways and clients so a partial degradation cannot be amplified into a broader overload.
• Fix the client retry behavior that amplified authentication traffic during the incident.
• Improve load-balancer capacity monitoring and strengthen regional failover safeguards.

August 20, 14:43 UTC (9 hours 54 minutes)

What happened?

During the incident window, the Copilot cloud agent task experienced impact. The tasks themselves still ran to completion, so no work was lost. Once processing caught up, the correct status and results appeared. Waiting a short time or checking back later would have shown the up-to-date state.

Across the incident, at least 54 organizations experienced Copilot Cloud Agent task status and results lagging well above their normal level. Per-minute customer-facing impact peaked at 37.5% of measured task-status activity.

What went wrong and why?

Copilot cloud agent stores the status and results of each agent task in a managed cloud database. One region of that database suffered a provider-side outage, and the calls that read and write task status in the affected region began failing and running slow.

The processors that stream task-status updates into that database then fell behind as database latency climbed. Their throughput is bounded by a fixed number of processing partitions sized for normal latency plus some headroom. The latency in this incident went well past that headroom, causing the backlog of task-status updates to grow instead of clearing. A storage configuration on the database also made the affected region slow to fail over, so the first failover attempts did not take effect, and recovery took longer than expected.

How did we respond?

1. A region of the managed cloud database that stores Copilot cloud agent task status began failing and running slow after a provider-side regional outage.
2. On-call engineers were paged, opened an incident, and traced the errors to the affected database region.
3. Engineers began a regional failover, but it did not take effect—a storage configuration on the database made the region slow to move—so task-status updates kept backing up.
4. The team forced the affected region offline and shifted task-status processing to a healthy region; write latency remained elevated, and the backlog kept the status view delayed.
5. Additional streaming capacity was added and the provider's region gradually recovered, letting the processors work through the backlog so task status and results caught up.
6. Latency returned to normal, the backlog cleared, and the incident was mitigated and resolved.

How are we making incidents like this less likely or less impactful?

• Remove the database storage configuration that made the affected region slow to fail over, so a single region's problems can be exited quickly.
• Improve runbooks for regional failover of this database, including a vetted, ordered list of fallback regions that keep the service healthy.
• Review failover priority so the next region chosen during a failover is the next-best healthy option.
• Make task-status streaming more resilient to elevated database latency, so a latency spike does not immediately throttle throughput and build a backlog.
• Improve monitoring and escalation on the managed databases.

August 26, 15:11 UTC (2 hours 50 minutes)

What happened?

Dependent services like Copilot code review and some GitHub Pages deployments that run on top of Actions were impacted during the incident window.

In most cases, delayed runs started once the backlog drained, and runs that failed to start succeeded when re-run after the incident. A small set of runs created during the earliest part of the incident could not be recovered by re-running and had to be started fresh.

What went wrong and why?

The quick summary is that shared infrastructure services have not kept up with month-over-month Actions growth and peak load.

A burst of incoming events arrived on top of already-high load and pushed the database past its tipping point. Query times climbed, and the database primary saturated.

With the database overloaded, the internal service that turns incoming events into runner assignments could not keep up, so Actions runs failed to start and began queuing well past their normal start time.

Failing over the database primary helped only partially. The throttles used to relieve inbound load were initially set slightly too high to fully protect the database, so recovery had to be ramped up slowly and manually.

There was no automatic circuit breaker to throttle inbound Actions load when the database showed early signs of stress, so the protective throttling had to be applied and tuned manually during the incident. This is one of the learnings from this incident.

How did we respond?

1. During a daily traffic peak, a burst of incoming events landed while a shared database that Actions depends on was already running near its limit. Write and query pressure on the database primary rose sharply and began to saturate.
2. The internal service that turns incoming events into runner assignments could no longer keep up. Actions runs began failing and incident investigation began.
3. The database primary was failed over to a replica. This improved things briefly but did not fully mitigate, so runs continued to fail or start late.
4. Inbound event processing was throttled to relieve pressure on the database and let it recover. Core service health returned once the throttling and service restarts took effect, though inbound work was now intentionally slowed.
5. The throttles were raised gradually, watching telemetry at each step so the database was not re-overwhelmed, until full event processing was restored and the backlog of delayed work drained. The incident was marked mitigated.
6. A subset of jobs on larger and self-hosted runners remained stuck waiting for a runner. A change was deployed to release them, and continued follow-up work cleared runs that had been left showing as queued.

How are we making incidents like this less likely or less impactful?

• Improve query efficiency of database usage by optimizing specific code paths in the client code.
• Add an automatic circuit breaker that throttles inbound Actions load when the database shows signs of stress, instead of relying on manual throttling during an incident.
• Add protections around how often the service falls back to the database primary when a replica is lagging, so a fallback cannot compound database pressure.
• Improve the ability to quickly cancel or clear runs left stuck in a queued or waiting-for-runner state after an incident, so affected jobs recover sooner.
• Continue the scaling and resiliency work already in flight for this part of Actions, including changes that were completing and rolling out around the time of the incident.

August 27, 10:04 UTC (2 hours 8 minutes)

What happened?

Customers who had configuration to use the Kimi K3 model were impacted by this incident. Customers who were using other models or switched to using other models were not impacted.

What went wrong and why?

Copilot offers a choice of AI models. One of them, Kimi K3, is served by an upstream model provider.

That provider had a serving degradation that caused a large share of Kimi K3 requests to fail with errors. Because the problem was with the upstream provider, requests that used other models—and requests made with the Auto setting, which routed to a different model—were not affected.

A steady share of Kimi K3 requests kept failing until the provider's mitigation took hold. At peak, more than half of the requests using Kimi K3 were failing.

How did we respond?

1. The upstream provider for Kimi K3 experienced a degradation, resulting in elevated failure rates for Copilot requests routed to that model.
2. Within a few minutes, monitoring flagged the elevated errors and investigation began.
3. An incident was declared, the failures were traced to a degradation at the upstream provider affecting Kimi K3 specifically, and a public status update was posted pointing to the provider.
4. Requests that used other models, or the Auto setting, kept working throughout, so retrying or switching models would have succeeded.
5. A ticket was opened with the provider and recovery was monitored as success rates climbed back toward normal on dashboards.
6. The incident remained open until the provider confirmed Kimi K3 was fully restored, then was resolved.

How are we making incidents like this less likely or less impactful?

• Work with the upstream provider to improve the reliability of the Kimi K3 model and reduce the errors seen during this incident.
• Investigate backup serving capacity for Kimi K3 so a single provider degradation has a fallback.

Follow the status page for real-time updates on status changes and post-incident recaps. To learn more about ongoing work, check out the engineering section on the GitHub Blog.

Source: GitHub Blog