Just got another from https://www.mediawiki.org/wiki/Special:NewMessages?lqt_method=mark_as_unread&lqt_operand=35761

[37086746] 2014-02-20 17:51:16: Fatal exception of type MWException

First works for me (Special:NewMessage) but lqt_method link gives me:

[af6d304a] 2014-02-20 18:06:05: Fatal exception of type MWException

It seems to be sporadic and not consistently happening. But it is happening quite a bit (as in more than once an hour).

https://www.mediawiki.org/wiki/Project:Calendar
[3c3c6555] 2014-02-20 18:47:00: Fatal exception of type MWException

deleted cookies and hit reload a few times and got it again.
[e74e1887] 2014-02-20 18:48:41: Fatal exception of type MWException

(but cookies had repopulated by then)

scap?

The updated version of scap is having some problems right now during the weekly major upgrade (ie: we cut the new wmf15 branch this morning, so new i10n messaging and all that).

The MediaWiki Core team is on it. You can see progress in #wikimedia-operations on Freenode.

Sorry about this, will update when there's more.

2014-02-20 17:47:15 mw1073 mediawikiwiki: [7e522347] /wiki/User_talk:Tpt Exception from line 468 of /usr/local/apache/common-local/php-1.23wmf15/includes/cache/LocalisationCache.php: No localisation cache found for English. Please run maintenance/rebuildLocalisationCache.php.
#0 /usr/local/apache/common-local/php-1.23wmf15/includes/cache/LocalisationCache.php(326): LocalisationCache->initLanguage('en')
#1 /usr/local/apache/common-local/php-1.23wmf15/includes/cache/LocalisationCache.php(260): LocalisationCache->loadItem('en', 'namespaceGender...')
#2 /usr/local/apache/common-local/php-1.23wmf15/languages/Language.php(641): LocalisationCache->getItem('en', 'namespaceGender...')
#3 /usr/local/apache/common-local/php-1.23wmf15/includes/Title.php(847): Language->needsGenderDistinction()
#4 /usr/local/apache/common-local/php-1.23wmf15/includes/Title.php(1258): Title->getNsText()
#5 /usr/local/apache/common-local/php-1.23wmf15/includes/Title.php(1270): Title->prefix('Tpt')
#6 /usr/local/apache/common-local/php-1.23wmf15/includes/WikiPage.php(829): Title->getPrefixedDBkey()
#7 /usr/local/apache/common-local/php-1.23wmf15/includes/WikiPage.php(377): WikiPage->getCachedLastEditTime()
#8 /usr/local/apache/common-local/php-1.23wmf15/includes/WikiPage.php(458): WikiPage->loadPageData()
#9 /usr/local/apache/common-local/php-1.23wmf15/includes/WikiPage.php(511): WikiPage->exists()
#10 /usr/local/apache/common-local/php-1.23wmf15/includes/WikiPage.php(222): WikiPage->getContentModel()
#11 /usr/local/apache/common-local/php-1.23wmf15/includes/WikiPage.php(208): WikiPage->getContentHandler()
#12 /usr/local/apache/common-local/php-1.23wmf15/includes/Action.php(92): WikiPage->getActionOverrides()
#13 /usr/local/apache/common-local/php-1.23wmf15/includes/Action.php(145): Action::factory('view', Object(WikiPage), Object(RequestContext))
#14 /usr/local/apache/common-local/php-1.23wmf15/includes/Wiki.php(151): Action::getActionName(Object(RequestContext))
#15 /usr/local/apache/common-local/php-1.23wmf15/includes/Wiki.php(514): MediaWiki->getAction()
#16 /usr/local/apache/common-local/php-1.23wmf15/includes/Wiki.php(447): MediaWiki->main()
#17 /usr/local/apache/common-local/php-1.23wmf15/index.php(49): MediaWiki->run()
#18 /usr/local/apache/common-local/w/index.php(3): require('/usr/local/apac...')
#19 {main}

Temporary update: Things should be back to normal for now. We will attempt another deployment when we are comfortable we won't hit this problem again.

The scap problem was that all hosts in the cluster (across both datacenters) were calling tin's rsync server. This bug was fixed in https://gerrit.wikimedia.org/r/#/c/114514/

21:20 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf15
21:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf14

we should be good now, let me know if not.