Spectra - Service Outage – Incident details

Web Server experiencing degraded performance

Service Outage

Monitoring
Major outage
Started 7 days ago

Affected

Hosting Infrastructure

Major outage from 4:00 AM to 4:47 AM, Partial outage from 4:00 AM to 4:47 AM, Degraded performance from 4:00 AM to 5:17 AM, Operational from 4:47 AM to 5:17 AM, Degraded performance from 5:17 AM to 12:00 AM, Operational from 5:17 AM to 12:00 AM

Web Server

Partial outage from 4:00 AM to 4:47 AM, Degraded performance from 4:47 AM to 12:00 AM

Object Storage

Degraded performance from 4:00 AM to 4:47 AM, Operational from 4:47 AM to 12:00 AM

Database

Major outage from 4:00 AM to 4:47 AM, Operational from 4:47 AM to 12:00 AM

Updates
  • Monitoring
    Monitoring

    Server is operational, but we're stuck waiting out the draining of the task queue, which is very slow. All infrastructure is operational, but we're advising users to hold off on posting videos if possible, until we can get the queue emptied out significantly.

    Current amount of remaining tasks: 8,961.

    We will continue to monitor the situation.

  • Identified
    Identified

    Due to changes in a PeerTube data migration script, an exponentially high amount of jobs to transcode videos and move them into Object Storage has caused a high amount of tasks (roughly 10,000) in our job queue.

  • Investigating
    Investigating

    We've been getting reports that our PeerTube server is down. Upon examination, the local server disk appears to have filled up due to failed or pending transcoding jobs / move-to-object-storage tasks.