starslingdev - Notice history

On June 25, 2026, between 17:33 UTC and 17:55 UTC, our background job service experienced degradation which increased delays to pull requests, repository pushes, Actions workflows, and Webhooks, with delays peaking at 7m. The issue was caused by underlying hypervisor issues and an incoming traffic spike, causing service timeouts which led to a connection storm and continual rebalances.

The issue was mitigated by replacing the problem node at 17:49, after which all services saw recovery by 18:07.

We identified an issue that caused degradation across multiple services including Webhooks, Pull Requests, Actions, and Issues. Customers may have experienced delays or failures with these services. We have applied mitigations and affected services have recovered.

The degradation affecting Actions, Issues, Pull Requests and Webhooks has been mitigated. We are monitoring to ensure stability.

Issues is experiencing degraded performance. We are continuing to investigate.

Actions is experiencing degraded availability. We are continuing to investigate.

This incident has been resolved. Job queue times are nominal. We pinpointed the secondary factor as an issue with a specific cloud provider and shifted load.

We have observed a reoccurrence and are currently investigating this incident.

This incident has been resolved. Zero recurrence in the last 2 hours.

This incident has been resolved. We haven't seen Github API errors for 2 hours and have added a durable fallback if they occur.

We have reached out to Github eng support and implemented a fallback on our side. Currently monitoring the result of our mitigation.

GitHub has been sending invalid job queued webhook events which lead to jobs not being picked up. We are continuing to work on a fix for this incident.

We are continuing to work on a fix for this incident. Infrastructure and job execution outcomes are nominal. The source of degraded performance is Github API rate limiting.

Impact is focused on a single customer with high concurrency. We are continuing to work on a fix for this incident.

We identified limited availability on one of the public cloud providers that we use for the 8 and 64 vCPU runners. 4 vCPU runners are not effected.