starslingdev - Notice history

This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.

We have seen recovery in GitHub Actions performance following our earlier mitigation. Workflow runs are processing normally, though jobs queued before 12:40 UTC may still experience failures and will need to be retried.

The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.

We are experiencing issues with GitHub Actions that are causing workflow run failures and delays for some users. Our engineering team is actively investigating the infrastructure issue and working to restore full functionality.

This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.

We identified an issue causing delays in GitHub Actions run starts. Some users may have experienced longer than expected wait times when triggering workflow runs. We have applied mitigations and have recovered. Our team continues to monitor and investigate the root cause.

The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.

This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.

The degradation affecting Pull Requests has been mitigated. We are monitoring to ensure stability.

We have applied a mitigation and are monitoring for recovery

Pull Requests is experiencing degraded availability. We are continuing to investigate.

We are investigating errors creating pull requests

This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.

We are seeing recovery across all services

The degradation affecting API Requests, Actions, Copilot, Issues, Pages and Pull Requests has been mitigated. We are monitoring to ensure stability.

Actions is experiencing degraded performance. We are continuing to investigate.

Actions is experiencing degraded availability. We are continuing to investigate.

Pages is experiencing degraded performance. We are continuing to investigate.

Copilot is experiencing degraded performance. We are continuing to investigate.

We are investigating timeouts to some GitHub services

Pull Requests is experiencing degraded performance. We are continuing to investigate.

This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.

The degradation has been mitigated. We are monitoring to ensure stability.

Webhooks is operating normally.

The degradation affecting Pull Requests has been mitigated. We are monitoring to ensure stability.

We identified the source of latency affecting multiple services and applied a fix. Issues and Actions are recovering, and remaining affected services are seeing improvement as processing backlogs clear. We are actively monitoring recovery across all services.

The degradation affecting Issues has been mitigated. We are monitoring to ensure stability.

This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.

We identified an issue that caused degradation across multiple services including Webhooks, Pull Requests, Actions, and Issues. Customers may have experienced delays or failures with these services. We have applied mitigations and affected services have recovered.

The degradation affecting Actions, Issues, Pull Requests and Webhooks has been mitigated. We are monitoring to ensure stability.

Issues is experiencing degraded performance. We are continuing to investigate.

Actions is experiencing degraded availability. We are continuing to investigate.

This incident has been resolved. Job queue times are nominal. We pinpointed the secondary factor as an issue with a specific cloud provider and shifted load.

We have observed a reoccurrence and are currently investigating this incident.

This incident has been resolved. Zero recurrence in the last 2 hours.

This incident has been resolved. We haven't seen Github API errors for 2 hours and have added a durable fallback if they occur.

We have reached out to Github eng support and implemented a fallback on our side. Currently monitoring the result of our mitigation.

GitHub has been sending invalid job queued webhook events which lead to jobs not being picked up. We are continuing to work on a fix for this incident.

On June 15, 2026, between 15:27 UTC and 16:23 UTC, GitHub webhook deliveries were delayed. During this window, webhook events were delivered later than normal, with average end-to-end delivery latency peaking at approximately 8.8 minutes. No webhook deliveries were lost — delayed events were queued and delivered once processing recovered.

This was caused by a temporary throughput constraint in an internal event-processing system that moves webhook events through GitHub's delivery pipeline. The rate at which events were processed for delivery dropped below the incoming volume, creating a backlog. We restarted the affected pipeline service, after which throughput recovered and the backlog fully drained by approximately 16:29 UTC. Webhook delivery latency returned to normal, the incident was mitigated at 16:39 UTC, and fully resolved at 17:37 UTC.

To reduce the likelihood and impact of similar incidents, we are working on improving the accuracy of the utilization metrics used to scale our delivery worker pools, reviewing connection and capacity headroom in the delivery pipeline.

The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability.

On June 11, 2026, between 19:28 UTC and 21:06 UTC, GitHub webhook deliveries were delayed. Average delivery latency peaked at approximately 3.4 minutes, with some deliveries delayed by as much as 62 minutes at the 99th percentile. No events were lost — delayed events were queued and delivered once processing caught up.

This was due to a change in how webhook traffic was distributed across regions: to relieve load on one region, a portion of processing was shifted to another, where higher latency prevented our delivery workers from keeping pace with incoming volume, creating a backlog. We mitigated the incident by rebalancing webhook traffic distribution; as load returned to normal levels, processing caught up and the delivery backlog fully drained.

We are working on improving the accuracy of the utilization metrics used to scale our delivery worker pools, and reassess how we distribute webhook traffic across regions, to reduce our time to detection and mitigation of issues like this one in the future.

The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability.

We have applied a mitigation and are monitoring for recovery.

We are currently experiencing delays in Web Hook delivery and are actively investigating the root cause.

On May 28, 2026, between 00:54 UTC and 01:19 UTC, some users experienced errors when interacting with the Webhooks API, including webhook delivery history and configuration endpoints. On average, the error rate was 0.28% and peaked at 0.45%. This was due to a bug that caused a single Kubernetes pod to enter a CrashLoopBackOff after receiving a 500 with an empty response body from Cosmos DB.

We mitigated the incident by restarting the service. To prevent future incidents, we are pushing a change to handle this response scenario from Cosmos DB appropriately.

The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability.

On May 27, 2026, between 12:07 UTC and 13:16 UTC, users experienced degraded performance for Git operations, Pull Requests, Issues, GraphQL API, and related services on github.com. During this time, operations that depended on Git file servers experienced elevated error rates (3.5% of pushes via HTTPS and 0.2% of pushes via SSH failed; no fetches/clones failed). An internal analytics component generated unexpectedly high load, which caused CPU saturation on the underlying infrastructure. This led to cascading slowdowns and errors across services that depend on Git operations. The issue was mitigated by stopping the offending component. Services began recovering shortly after mitigation and were fully restored by 13:16 UTC. We are taking steps to add resource limits and kill switches for internal analytics components to prevent similar issues in the future.

We're continuing to investigate degraded performance of Git operations, Issues and Pull requests.

On May 26, 2026, between 10:40 UTC and 12:56 UTC, GitHub Actions jobs were degraded. From 10:40 to 12:16 UTC, all newly queued Actions runs failed to start. From 12:16 to 12:56 UTC, Actions runs that required downloading actions for their workflows continued to fail. GitHub Pages, Copilot Code Review, Copilot coding agent, Octoshift, and GitHub Enterprise Importer were also impacted due to their dependency on Actions.

This was caused by our automated account review system incorrectly suspending the service account used by GitHub Actions to authenticate workflow runs and download actions.

We mitigated by restoring the account at 12:16 UTC, marking it exempt from further automated review at 12:20 UTC, and redeploying a related service at 12:48 UTC to flush cached account state. Full recovery was confirmed at 12:56 UTC.

During this incident, a small number of Issues, PRs, Comments, and Discussions were marked as hidden when the service account was disabled. No data was lost. All content hidden because of this incident has been restored and full search index restoration is in progress.

To prevent a recurrence, we have added an allowlist of all service accounts that cannot be suspended by automated systems, and ensuring these protections are enforced consistently across all account management tooling. We are also improving diagnostic tooling for accounts and reducing cache propagation delays to shorten time to mitigate similar incidents in the future.

The degradation has been mitigated. We are monitoring to ensure stability.

The degradation affecting Actions and Pages has been mitigated. We are monitoring to ensure stability.

We have identified the cause of the authentication issues affecting GitHub Actions and are actively working on mitigation

Actions is experiencing degraded performance. We are continuing to investigate.

We are investigating authentication issues leading to failure in starting Actions runs and downloading actions. At this time the majority of Actions runs is impacted.

Actions is experiencing degraded availability. We are continuing to investigate.

On May 20, 2026, between 16:00 UTC and 17:45 UTC, GitHub Actions customers experienced run start delays exceeding 5 minutes. Approximately 4.5% of all runs were delayed during the impact window, with scale set jobs disproportionately affected. 30% of scale set jobs were delayed and 4% failed to start entirely.

The incident was caused by a misconfigured health check on an internal service that assigns jobs to runners. A brief latency spike in an upstream dependency triggered health check failures across several pods, removing them from service and concentrating load on the remaining capacity. The added load drove memory pressure that escalated into a cascading failure in one regional cluster, leaving it unable to self-recover.

Responders mitigated the incident by scaling capacity in the healthy regional clusters and draining traffic away from the impaired one, after which run start latency recovered. To prevent recurrence, we are strengthening our health check configuration to avoid cascading failure scenarios and evaluating automated mitigations to rebalance traffic when a region is degraded.

Customer impact has fully subsided. We are maintaining yellow status while we deploy a permanent fix to prevent recurrence.

We've applied a mitigation to fix the issues with queuing and running Actions jobs. We are seeing improvements in telemetry and are monitoring for full recovery.

The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.

A subset of runners are taking longer than expected to connect, which may delay some jobs from beginning execution. We are actively working to mitigate the issue.

We are continuing to work on a fix for this incident. Infrastructure and job execution outcomes are nominal. The source of degraded performance is Github API rate limiting.