Page MenuHomePhabricator

Score extension outputting "Unable to copy the generated files to their final location: An unknown error occurred in storage backend "global-ceph"." occasionally
Closed, ResolvedPublic

Description

Relevant thread: https://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_(technical)&oldid=557624423#Score_block

Example input:


<score>
\new Score {

\new Staff {
  \relative c' {
    \time 3/4
    \clef treble
    \key g \major
    
    \partial 4 d4 |
    b'4( c) b | d4( b) a | g( a) e | g fis e |
    d2 d4 | g4( a) b | a2 g4 | fis2 \bar "||"
  }
}

}</score>


Current output on the English Wikipedia: "Unable to copy the generated files to their final location: An unknown error occurred in storage backend "global-ceph".".

Example here on the test Wikipedia: https://test.wikipedia.org/wiki/Music_notation.

I could swear this exact output was working earlier today (the relevant thread explains what happened). Something seems to have broken in Wikimedia's configuration within the past few hours.


Version: wmf-deployment
Severity: normal

Details

Reference
bz49004

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 1:44 AM
bzimport set Reference to bz49004.
bzimport added a subscriber: Unknown Object (MLST).

It seems like this is related to Ceph (https://wikitech.wikimedia.org/wiki/Ceph) more than Score specifically. Wikimedia image scalers are having a rough time right now.

Thanks for the heads up, MZ.

It looks like Tim and Aaron were able to resolve the Ceph issue last night (Pacific timezone) and things now work (I just had to edit the Score code on that test.wiki page to regenerate the image).

There is still some debugging needed to figure out what happened last night, but we at least were able to successfully fail over to Swift (from Ceph). Closing this bug, though.