Spectra - Degraded Performance - Object Storage – Incident details

Experiencing partial outage

Degraded Performance - Object Storage

Resolved
Partial outage
Started 5 months agoLasted about 2 hours

Affected

Hosting Infrastructure

Partial outage from 11:59 PM to 1:02 AM, Degraded performance from 11:59 PM to 1:02 AM, Operational from 11:59 PM to 2:14 AM, Partial outage from 1:02 AM to 1:12 AM, Degraded performance from 1:02 AM to 1:12 AM, Partial outage from 1:12 AM to 2:14 AM, Degraded performance from 1:12 AM to 2:14 AM

Web Server

Partial outage from 11:59 PM to 2:14 AM

Object Storage

Degraded performance from 11:59 PM to 2:14 AM

Database

Operational from 11:59 PM to 2:14 AM

Updates
  • Resolved
    Resolved

    It looks like most videos should work just fine with uploading and storage now, but we do still have a few videos stuck in limbo. We'll be reaching out after a few hours to check in.

  • Monitoring
    Monitoring

    Our upload / transcode / store test passed, now we're just monitoring an existing stuck job to see if we can get it to follow suit.

  • Update
    Update

    We are running some tests to see whether the culprit has been isolated or not. Stay tuned!

  • Investigating
    Investigating

    Some of our videos don't seem to be getting successfully uploaded to our Object Storage provider. We're currently investigating why this is happening, and trying to fix up a couple of stuck jobs. It's possible that something changed in the integration when we moved over that needs to be accounted for.