starslingdev - Job start failures due to GitHub API errors – Incident details

All systems operational

Job start failures due to GitHub API errors

Resolved
Major outage
Started about 1 month agoLasted about 4 hours

Affected

GitHub Runners

Major outage from 7:48 PM to 11:18 PM

Updates
  • Resolved
    UTC
    Resolved

    This incident has been resolved. We haven't seen Github API errors for 2 hours and have added a durable fallback if they occur.

  • Monitoring
    UTC
    Monitoring

    We have reached out to Github eng support and implemented a fallback on our side. Currently monitoring the result of our mitigation.

  • Identified
    UTC
    Identified

    GitHub has been sending invalid job queued webhook events which lead to jobs not being picked up. We are continuing to work on a fix for this incident.

  • Investigating
    UTC
    Investigating

    Up to 10% StarSling Runners aren't starting jobs due to Github API errors. We are continuing to work on a fix for this incident.