Spectra - Degraded Performance - Transcoding – Incident details

Experiencing partial outage

Degraded Performance - Transcoding

Monitoring
Major outage
Started 2 days ago

Affected

Hosting Infrastructure

Video Processing

Operational from 7:21 PM to 9:38 PM, Major outage from 9:38 PM to 10:10 PM, Degraded performance from 10:10 PM to 12:00 AM

Remote Runners

Operational from 7:21 PM to 9:38 PM, Major outage from 9:38 PM to 10:10 PM, Degraded performance from 10:10 PM to 12:00 AM

Updates
  • Monitoring
    Monitoring

    Video transcoding is operational, it's simply being handled locally for the instance at this very moment. We will update this ticket when we know more.

  • Update
    Update

    We've temporarily disabled the remote runners to process videos locally instead. This will be more of a strain on the local server in the short term; we ask users to please be patient as our backend works through the queue of unprocessed videos.

  • Update
    Update

    We're still awaiting feedback from the maintainer of the remote runner. There are only a handful of videos in a stuck state, and we will evaluate the best way to handle them in the short term.

  • Update
    Update

    Our server upgrade went off without a hitch, and we've managed to free up some reserve space as we continue to investigate. We're still uncertain about what's happening with the remote runner, but we'll keep an eye on it.

  • Update
    Update

    We are temporarily taking the server down to run some resource-intensive maintenance scripts. Stay tuned!

  • Update
    Update

    We've reached out to our contact at Framasoft regarding the runner. In the meantime, we'll run some non-destructive cleanup scripts in an effort to reclaim some storage space on our server.

  • Investigating
    Investigating

    We've seen a few complaints regarding video transcoding, and it largely seems to be caused by remote processing jobs performed on the server that Framasoft provides to PeerTube admins. We're still in the investigation phase, and are also using this time to clean up the storage of our own server and run upgrades.