Spectra - Service Outage – Incident details

All systems operational

Service Outage

Resolved
Major outage
Started 4 months agoLasted 13 days

Affected

Hosting Infrastructure

Major outage from 4:00 AM to 4:47 AM, Partial outage from 4:00 AM to 4:47 AM, Degraded performance from 4:00 AM to 5:17 AM, Operational from 4:47 AM to 5:17 AM, Degraded performance from 5:17 AM to 8:08 PM, Operational from 5:17 AM to 8:08 PM

Web Server

Partial outage from 4:00 AM to 4:47 AM, Degraded performance from 4:47 AM to 8:08 PM

Object Storage

Degraded performance from 4:00 AM to 4:47 AM, Operational from 4:47 AM to 8:08 PM

Database

Major outage from 4:00 AM to 4:47 AM, Operational from 4:47 AM to 8:08 PM

Updates
  • Resolved
    UTC
    Resolved

    This incident has been resolved. All systems are green!

  • Monitoring
    UTC
    Monitoring

    Server is operational, but we're stuck waiting out the draining of the task queue, which is very slow. All infrastructure is operational, but we're advising users to hold off on posting videos if possible, until we can get the queue emptied out significantly.

    Current amount of remaining tasks: 8,961.

    We will continue to monitor the situation.

  • Identified
    UTC
    Identified

    Due to changes in a PeerTube data migration script, an exponentially high amount of jobs to transcode videos and move them into Object Storage has caused a high amount of tasks (roughly 10,000) in our job queue.

  • Investigating
    UTC
    Investigating

    We've been getting reports that our PeerTube server is down. Upon examination, the local server disk appears to have filled up due to failed or pending transcoding jobs / move-to-object-storage tasks.