Spectra - Degraded Performance - Transcoding – Incident details

Degraded Performance - Transcoding

Resolved2 hours 49 minutes

Affected

Hosting Infrastructure

Video Processing

Operational from 7:21 PM to 9:38 PM, Major outage from 9:38 PM to 10:10 PM

Remote Runners

Operational from 7:21 PM to 9:38 PM, Major outage from 9:38 PM to 10:10 PM

  • ResolvedUTC

    Video transcoding is operational, it's simply being handled locally for the instance at this very moment. We will update this ticket when we know more.

  • UpdateUTC

    We've temporarily disabled the remote runners to process videos locally instead. This will be more of a strain on the local server in the short term; we ask users to please be patient as our backend works through the queue of unprocessed videos.

  • UpdateUTC

    We're still awaiting feedback from the maintainer of the remote runner. There are only a handful of videos in a stuck state, and we will evaluate the best way to handle them in the short term.

  • UpdateUTC

    Our server upgrade went off without a hitch, and we've managed to free up some reserve space as we continue to investigate. We're still uncertain about what's happening with the remote runner, but we'll keep an eye on it.

  • UpdateUTC

    We are temporarily taking the server down to run some resource-intensive maintenance scripts. Stay tuned!

  • UpdateUTC

    We've reached out to our contact at Framasoft regarding the runner. In the meantime, we'll run some non-destructive cleanup scripts in an effort to reclaim some storage space on our server.

  • InvestigatingUTC

    We've seen a few complaints regarding video transcoding, and it largely seems to be caused by remote processing jobs performed on the server that Framasoft provides to PeerTube admins. We're still in the investigation phase, and are also using this time to clean up the storage of our own server and run upgrades.