Affected Components: Database, Object Storage, Web Server, Remote Runners

Nov 26, 01:52:36 GMT+0
Resolved - After a LOT of investigation, we finally figured out the root cause. It turns out, the server was silently failing on transferring files of a certain size over to our Object Storage. We couldn't easily debug it, because it gave very little context. However, we managed to fix a long-standing issue, and the clog of old videos that got stuck are all being published with their original datestamp values. We apologize for any inconvenience, but we should be way more stable moving forward!.

Duration: 7 days, 1 hour and 8 minutes

Affected Components: , Remote Runners, Hosting Infrastructure →

Nov 11, 19:21:10 GMT+0
Investigating - We've seen a few complaints regarding video transcoding, and it largely seems to be caused by remote processing jobs performed on the server that Framasoft provides to PeerTube admins. We're still in the investigation phase, and are also using this time to clean up the storage of our own server and run upgrades..

Nov 11, 19:34:26 GMT+0
Investigating - We've reached out to our contact at Framasoft regarding the runner. In the meantime, we'll run some non-destructive cleanup scripts in an effort to reclaim some storage space on our server..

Nov 11, 19:49:57 GMT+0
Investigating - We are temporarily taking the server down to run some resource-intensive maintenance scripts. Stay tuned!.

Nov 11, 19:58:06 GMT+0
Investigating - Our server upgrade went off without a hitch, and we've managed to free up some reserve space as we continue to investigate. We're still uncertain about what's happening with the remote runner, but we'll keep an eye on it..

Nov 11, 20:26:52 GMT+0
Investigating - We're still awaiting feedback from the maintainer of the remote runner. There are only a handful of videos in a stuck state, and we will evaluate the best way to handle them in the short term..

Nov 11, 22:10:18 GMT+0
Resolved - Video transcoding is operational, it's simply being handled locally for the instance at this very moment. We will update this ticket when we know more..

Nov 11, 21:38:12 GMT+0
Investigating - We've temporarily disabled the remote runners to process videos locally instead. This will be more of a strain on the local server in the short term; we ask users to please be patient as our backend works through the queue of unprocessed videos..

Duration: 2 hours and 15 minutes

Affected Components: Web Server, Object Storage, Database, , Hosting Infrastructure →

Jul 9, 23:59:51 GMT+0
Investigating - Some of our videos don't seem to be getting successfully uploaded to our Object Storage provider. We're currently investigating why this is happening, and trying to fix up a couple of stuck jobs. It's possible that something changed in the integration when we moved over that needs to be accounted for..

Jul 10, 01:02:36 GMT+0
Investigating - We are running some tests to see whether the culprit has been isolated or not. Stay tuned!.

Jul 10, 01:12:30 GMT+0
Monitoring - Our upload / transcode / store test passed, now we're just monitoring an existing stuck job to see if we can get it to follow suit..

Jul 10, 02:14:39 GMT+0
Resolved - It looks like most videos should work just fine with uploading and storage now, but we do still have a few videos stuck in limbo. We'll be reaching out after a few hours to check in..

Duration: 6 hours and 33 minutes

Affected Components: Web Server, Object Storage, Database, , Hosting Infrastructure →

Jul 8, 20:59:27 GMT+0
Identified - Our server's operating system needs to be upgraded to better support the instance. Sit tight!.

Jul 9, 03:32:30 GMT+0
Completed - We've completed our migration! The new server is up and running, and we were able to save about 98% of the existing data in the move. .

Jul 8, 20:59:28 GMT+0
Identified - Maintenance is now in progress.

Jul 9, 01:14:11 GMT+0
Identified - We've successfully recovered most avatars and thumbnails, we're now migrating over additional backed up data to fill in some gaps. Really close to being done!.

Jul 8, 23:12:40 GMT+0
Identified - Data has been successfully migrated to a new server. We're in the process of debugging a few issues with the new setup, which prevent PeerTube from successfully starting. Sit tight while we poke through this!.

Jul 8, 23:39:00 GMT+0
Identified - PeerTube is operational, however, there are some last-minute data migrations for user avatars and unprocessed videos that needs to be attended to. We're currently investigating a way forward..