Spectra Status - Incident history https://spectra.instatus.com Spectra Tue, 26 Nov 2024 01:52:36 +0000 Transcoding / Storage / Object Storage Issue Type: Incident Affected Components: Database, Object Storage, Web Server, Remote Runners Nov 26, 01:52:36 GMT+0 - Resolved - After a LOT of investigation, we finally figured out the root cause. It turns out, the server was silently failing on transferring files of a certain size over to our Object Storage. We couldn't easily debug it, because it gave very little context. However, we managed to fix a long-standing issue, and the clog of old videos that got stuck are all being published with their original datestamp values. We apologize for any inconvenience, but we should be way more stable moving forward! Type: Incident

Affected Components: , , ,

<p><small>Nov <var data-var='date'> 26</var>, <var data-var='time'>01:52:36</var> GMT+0</small><br><strong>Resolved</strong> - After a LOT of investigation, we finally figured out the root cause. It turns out, the server was silently failing on transferring files of a certain size over to our Object Storage. We couldn't easily debug it, because it gave very little context. However, we managed to fix a long-standing issue, and the clog of old videos that got stuck are all being published with their original datestamp values. We apologize for any inconvenience, but we should be way more stable moving forward!.</p> ]]> Tue, 26 Nov 2024 01:52:36 +0000 https://spectra.instatus.com/incident/cm3xsy0p80008ntffv466hwxy https://spectra.instatus.com/incident/cm3xsy0p80008ntffv466hwxy Degraded Performance - Transcoding Type: Incident Duration: 7 days, 1 hour and 8 minutes Affected Components: , Remote Runners, Hosting Infrastructure → Nov 11, 19:21:10 GMT+0 - Investigating - We've seen a few complaints regarding video transcoding, and it largely seems to be caused by remote processing jobs performed on the server that Framasoft provides to PeerTube admins. We're still in the investigation phase, and are also using this time to clean up the storage of our own server and run upgrades. Nov 11, 19:34:26 GMT+0 - Investigating - We've reached out to our contact at Framasoft regarding the runner. In the meantime, we'll run some non-destructive cleanup scripts in an effort to reclaim some storage space on our server. Nov 11, 19:49:57 GMT+0 - Investigating - We are temporarily taking the server down to run some resource-intensive maintenance scripts. Stay tuned! Nov 11, 19:58:06 GMT+0 - Investigating - Our server upgrade went off without a hitch, and we've managed to free up some reserve space as we continue to investigate. We're still uncertain about what's happening with the remote runner, but we'll keep an eye on it. Nov 11, 20:26:52 GMT+0 - Investigating - We're still awaiting feedback from the maintainer of the remote runner. There are only a handful of videos in a stuck state, and we will evaluate the best way to handle them in the short term. Nov 11, 22:10:18 GMT+0 - Resolved - Video transcoding is operational, it's simply being handled locally for the instance at this very moment. We will update this ticket when we know more. Nov 11, 21:38:12 GMT+0 - Investigating - We've temporarily disabled the remote runners to process videos locally instead. This will be more of a strain on the local server in the short term; we ask users to please be patient as our backend works through the queue of unprocessed videos. Type: Incident

Duration: 7 days, 1 hour and 8 minutes

Affected Components: ,

<p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>19:21:10</var> GMT+0</small><br><strong>Investigating</strong> - We've seen a few complaints regarding video transcoding, and it largely seems to be caused by remote processing jobs performed on the server that Framasoft provides to PeerTube admins. We're still in the investigation phase, and are also using this time to clean up the storage of our own server and run upgrades..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>19:34:26</var> GMT+0</small><br><strong>Investigating</strong> - We've reached out to our contact at Framasoft regarding the runner. In the meantime, we'll run some non-destructive cleanup scripts in an effort to reclaim some storage space on our server..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>19:49:57</var> GMT+0</small><br><strong>Investigating</strong> - We are temporarily taking the server down to run some resource-intensive maintenance scripts. Stay tuned!.</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>19:58:06</var> GMT+0</small><br><strong>Investigating</strong> - Our server upgrade went off without a hitch, and we've managed to free up some reserve space as we continue to investigate. We're still uncertain about what's happening with the remote runner, but we'll keep an eye on it..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>20:26:52</var> GMT+0</small><br><strong>Investigating</strong> - We're still awaiting feedback from the maintainer of the remote runner. There are only a handful of videos in a stuck state, and we will evaluate the best way to handle them in the short term..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>22:10:18</var> GMT+0</small><br><strong>Resolved</strong> - Video transcoding is operational, it's simply being handled locally for the instance at this very moment. We will update this ticket when we know more..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>21:38:12</var> GMT+0</small><br><strong>Investigating</strong> - We've temporarily disabled the remote runners to process videos locally instead. This will be more of a strain on the local server in the short term; we ask users to please be patient as our backend works through the queue of unprocessed videos..</p> ]]> Mon, 11 Nov 2024 19:21:10 +0000 https://spectra.instatus.com/incident/cm3desov3000si6zouxjvbh9m https://spectra.instatus.com/incident/cm3desov3000si6zouxjvbh9m Degraded Performance - Object Storage Type: Incident Duration: 2 hours and 15 minutes Affected Components: Web Server, Object Storage, Database, , Hosting Infrastructure → Jul 9, 23:59:51 GMT+0 - Investigating - Some of our videos don't seem to be getting successfully uploaded to our Object Storage provider. We're currently investigating why this is happening, and trying to fix up a couple of stuck jobs. It's possible that something changed in the integration when we moved over that needs to be accounted for. Jul 10, 01:02:36 GMT+0 - Investigating - We are running some tests to see whether the culprit has been isolated or not. Stay tuned! Jul 10, 01:12:30 GMT+0 - Monitoring - Our upload / transcode / store test passed, now we're just monitoring an existing stuck job to see if we can get it to follow suit. Jul 10, 02:14:39 GMT+0 - Resolved - It looks like most videos should work just fine with uploading and storage now, but we do still have a few videos stuck in limbo. We'll be reaching out after a few hours to check in. Type: Incident

Duration: 2 hours and 15 minutes

Affected Components: , , ,

<p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>23:59:51</var> GMT+0</small><br><strong>Investigating</strong> - Some of our videos don't seem to be getting successfully uploaded to our Object Storage provider. We're currently investigating why this is happening, and trying to fix up a couple of stuck jobs. It's possible that something changed in the integration when we moved over that needs to be accounted for..</p> <p><small>Jul <var data-var='date'> 10</var>, <var data-var='time'>01:02:36</var> GMT+0</small><br><strong>Investigating</strong> - We are running some tests to see whether the culprit has been isolated or not. Stay tuned!.</p> <p><small>Jul <var data-var='date'> 10</var>, <var data-var='time'>01:12:30</var> GMT+0</small><br><strong>Monitoring</strong> - Our upload / transcode / store test passed, now we're just monitoring an existing stuck job to see if we can get it to follow suit..</p> <p><small>Jul <var data-var='date'> 10</var>, <var data-var='time'>02:14:39</var> GMT+0</small><br><strong>Resolved</strong> - It looks like most videos should work just fine with uploading and storage now, but we do still have a few videos stuck in limbo. We'll be reaching out after a few hours to check in..</p> ]]> Tue, 9 Jul 2024 23:59:51 +0000 https://spectra.instatus.com/incident/clyf2plof194097hsn1bnvnb3iq https://spectra.instatus.com/incident/clyf2plof194097hsn1bnvnb3iq OS Upgrade Type: Maintenance Duration: 6 hours and 33 minutes Affected Components: Web Server, Object Storage, Database, , Hosting Infrastructure → Jul 8, 20:59:27 GMT+0 - Identified - Our server's operating system needs to be upgraded to better support the instance. Sit tight! Jul 9, 03:32:30 GMT+0 - Completed - We've completed our migration! The new server is up and running, and we were able to save about 98% of the existing data in the move. Jul 8, 20:59:28 GMT+0 - Identified - Maintenance is now in progress Jul 9, 01:14:11 GMT+0 - Identified - We've successfully recovered most avatars and thumbnails, we're now migrating over additional backed up data to fill in some gaps. Really close to being done! Jul 8, 23:12:40 GMT+0 - Identified - Data has been successfully migrated to a new server. We're in the process of debugging a few issues with the new setup, which prevent PeerTube from successfully starting. Sit tight while we poke through this! Jul 8, 23:39:00 GMT+0 - Identified - PeerTube is operational, however, there are some last-minute data migrations for user avatars and unprocessed videos that needs to be attended to. We're currently investigating a way forward. Type: Maintenance

Duration: 6 hours and 33 minutes

Affected Components: , , ,

<p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>20:59:27</var> GMT+0</small><br><strong>Identified</strong> - Our server's operating system needs to be upgraded to better support the instance. Sit tight!.</p> <p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>03:32:30</var> GMT+0</small><br><strong>Completed</strong> - We've completed our migration! The new server is up and running, and we were able to save about 98% of the existing data in the move. .</p> <p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>20:59:28</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>01:14:11</var> GMT+0</small><br><strong>Identified</strong> - We've successfully recovered most avatars and thumbnails, we're now migrating over additional backed up data to fill in some gaps. Really close to being done!.</p> <p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>23:12:40</var> GMT+0</small><br><strong>Identified</strong> - Data has been successfully migrated to a new server. We're in the process of debugging a few issues with the new setup, which prevent PeerTube from successfully starting. Sit tight while we poke through this!.</p> <p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>23:39:00</var> GMT+0</small><br><strong>Identified</strong> - PeerTube is operational, however, there are some last-minute data migrations for user avatars and unprocessed videos that needs to be attended to. We're currently investigating a way forward..</p> ]]> Mon, 8 Jul 2024 20:59:27 +0000 https://spectra.instatus.com/maintenance/clydgu8bj1945218boig1t8c7t2 https://spectra.instatus.com/maintenance/clydgu8bj1945218boig1t8c7t2