See https://integration.wikimedia.org/ci/job/mediawiki-core-qunit/2638/consoleFull
Version: wmf-deployment
Severity: critical
Description
Description
Details
Details
- Reference
- bz47639
Related Objects
Related Objects
Event Timeline
Comment Actions
Raising priority. Happening on more branches.
Something is causing it to be read-only. Needs to be fixed soon. Blocking smooth development.
Comment Actions
I've ruled out relation to QUnit, PhantomJS or Grunt as those are the same across all branches.
The error from PhantomJS (about .start not being called) is bogus. The problem is that the page doesn't contain QUnit at all (or MediaWiki for that matter) since it is an HTTP 500 response:
23:16:50 HTTP/1.1 500 MediaWiki exception
23:16:50 Date: Mon, 29 Apr 2013 23:16:51 GMT
23:16:50 Server: Apache/2.2.22 (Ubuntu)
23:16:50 X-Powered-By: PHP/5.3.10-1ubuntu3.5+wmf1
23:16:50 X-Content-Type-Options: nosniff
23:16:50 Status: 500 MediaWiki exception
23:16:50 Vary: Accept-Encoding
23:16:50 Content-Length: 392
23:16:50 Connection: close
23:16:50 Content-Type: text/html; charset=utf-8
23:16:50
23:16:50 <!doctype html>
23:16:50 <html><head><title>Database error</title></head><body>
23:16:50 A database error has occurred. Did you forget to run
maintenance/update.php after upgrading? See:
https://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script<br />
23:16:50 Query: DELETE FROM msg_resource<br />
23:16:50 Function: MessageBlobStore::clear<br />
23:16:50 Error: 8 attempt to write a readonly database<br />
23:16:50 </body></html>
Comment Actions
So I guess that is because several queries hits MessageBlobStore::clear() and possibly over writing queries. SQLite only accepts one connection so multiples HTTP queries doing writes to the database would cause the 'readonly issue'.
I am not sure whether we can solve that.
Comment Actions
The multiple connections seemed plausible at first, though not on second thought.
Lots of people use sqlite for their MediaWiki installs. MediaWiki and/or php-sqlite can handle that.
Also, it is not happening on master builds, only on older branches.
Comment Actions
Following a discussion with Timo:
- this happen constently with REL1_20
- no issue with REL1_21
Maybe it can be bisected to find out the commit that fixed the issue.
Comment Actions
(In reply to comment #2 by Krinkle)
Happening on more branches.
(In reply to comment #6 by Hashar)
- this happen constently with REL1_20
- no issue with REL1_21
Asking for clarification: So this only happens with REL1_20, not with git master or REL1_19 (if there are builds)?
Comment Actions
(In reply to comment #7)
(In reply to comment #2 by Krinkle)
Happening on more branches.
(In reply to comment #6 by Hashar)
- this happen constently with REL1_20
- no issue with REL1_21
Asking for clarification: So this only happens with REL1_20, not with git
master or REL1_19 (if there are builds)?
I'm not sure about REL1_19, but assuming this is related to a (yet to be identified) problem with the Database interface in MediaWiki that was fixed in 1.21, I'd guess that this will fail for REL1_19 as well just like it fails for REL1_20.
REL1_21 and master work fine.
Comment Actions
If that is indeed problem, we need to identify the fix (we already fixed it, apparently) and backport to REL1_20/REL1_19.
Comment Actions
Assigning to myself.
Using reverse git-bisect (good=bad, bad=good) has brought me to 1da33a49a0a95ad7066926ca1a1a771411074946 as the point where the problem was fixed.
Comment Actions
Related URL: https://gerrit.wikimedia.org/r/62519 (Gerrit Change I9441d897d45bca189c17b8bcca4ea7b3469af8b0)
Comment Actions
Looking at this change tells me something else however.
Though it is (beyond a doubt) the commit that makes the readonly warning disappear, it does just that.
The readonly problem is real, and not caused by something in MediaWiki 1.20, instead it is happening on 1.21 and master as well, but MediaWiki is no longer throwing on it.
Let's backport the change and then fix the chmod of the sqlite file as well.
Comment Actions
Related URL: https://gerrit.wikimedia.org/r/62639 (Gerrit Change Id3862690f035d026b579403fb2b4d2c2dfb62617)
Comment Actions
Fixed.
- Backported Idc83a0fe042806263f to fix the mediawiki fatal caused by ResourceLoader previously not accounting for read-only mode.
- Submitted Id3862690f035d026b to fix the fact the db is not writable by Apache.