During E263, Jaime (@jcrespo) put the following choice to us. Should he:

a. Apply this change to all wikis
b. Apply this change to a subset of wikis, on request
c. Not apply this change

We seemed to get consensus around "a" (applying this change to all wikis), so that's the decision we're putting in "final comment". TechCom plans to move this task to "Approved" during E265 (barring any unaddressed objections in the comments on this task).

21:33:56 <Dereckson> jynus: I'm not sure why people seemed to want this optional, perhaps it was only a suggestion to ease your dba life not to apply it where not needed

I believe this is indeed the case. Our goal is to get it to the wikis who have requested it, not to make it optional.

I hope with this, (a ratification of Mediawiki's Architecture comittee), which is perfectly documented and announced, things are clear.

Legoktm made this edit yesterday: https://wikitech.wikimedia.org/w/index.php?title=Schema_changes&diff=818276&oldid=542880.

• If a table is deployed to multiple wikis/databases, any schema changes must be applied to all wikis where that table is present for consistency.

Hmmmm, as a code-illiterate I just read the entire discussion (having been referred here by Nikerabbit from T145039). Could some kind soul please sum it up for me so that I knew whether there is a way to set the page language on Outreach wiki, and if not - what do I do to set it up correctly?

This was approved by TechCom following an IRC meeting we held a couple of weeks ago (E263) and following the subsequent "final comment" period for people who weren't able to attend the IRC conversation. (my apologies for not setting making the status clear here nearer the conclusion of the final comment period, which ended 2016-08-31)

In response to @Halibutt's request to have this summed up (T69223#2622513), TechCom believes the schema change has MediaWiki developer consensus. This still needs to go through the Wikimedia database schema changes process, which Legoktm recently clarified during the IRC conversation (logs available: E263). In short, my understanding of the test to see if this is deployed to Outreach Wiki, you should be able to visit https://outreach.wikimedia.org/wiki/Special:PageLanguage and get a similar page to what you see at https://test.wikipedia.org/wiki/Special:PageLanguage . From what I'm seeing, the schema change hasn't been made for Outreach wiki yet. It is one of the many tasks in the backlog column of the Schema-change-in-production board.

A longer chronology for those that are interested, but don't want to do the research that @Halibutt did:

1. This code was committed to Extension:Translate in 2014
2. This feature was deployed to translatewiki.net in 2014, but deployment to the Wikimedia production cluster required a schema change
3. In 2015, Sean Pringle (@Springle) commented that he started on the schema change for some wikis. Sean was the sole member of our DBA team at the time, and was presumably juggling many tasks. It is possible to misinterpret his update as as "this change is will inevitably conclude soon" (as opposed to "this change is starting, but completion is uncertain")
4. Later in 2015, Jaime Crespo (@jcrespo) joined the DBA team at WMF, and soon after that, Sean Pringle departed.
5. In late 2015 and early 2016, Jaime expressed confusion about the priority and workflow of this issue. After much discussion here and elsewhere, Mark Bergsma (@mark) clarified that this is a potentially a very disruptive change and that WMF has a lot of DBA work to do in addition to this change (see comment T69223#2157651)
6. During 2016, Jaime has been asking for consensus among many developers before making the change.
7. In T69223#2539532, Sébastien Santoro (@Dereckson) marked this as an TechCom-RFC as a tool for gauging MediaWiki developer consensus
8. In August 2016, TechCom discussed this in the public IRC office hour (E263). We seemed to have consensus around Jaime's suggestion that if we apply this change in production, we apply this change to all wikis that Extension:Translate is deployed to.
9. On 2016-08-31, the ArchCom-RFC "final comment" period ended as noted on the ArchCom Status page
10. The next step: this task needs to be picked up on the Schema-change-in-production board. Wikimedia Technical Operations' schema change policy is described at wikitech:Schema changes.

That's my understanding of the situation. I hope I've got this right, and I hope this helps!

Thanks for the very good summary. Some clarifications for the chronology.

1. This code was committed to Extension:Translate in 2014

The page content language functionality is in MediaWiki core, not in Translate.

8. In August 2016, TechCom discussed this in the public IRC office hour (E263). We seemed to have consensus around Jaime's suggestion that if we apply this change in production, we apply this change to all wikis that Extension:Translate is deployed to.

Per my understanding, the consensus is to apply this to all wikis because schema inconsistencies are already causing a lot of trouble and no more of that is wanted.

The page content language functionality is in MediaWiki core, not in Translate.

This is exactly at the core of the issue. If this was an extension and was only deployed on a few wikis (note that an "extension" doesn't mean that is on a lower standards, it only means that it is technically optional; wikibase is an extension, and probably wikipedia would not survive without it, nor the many translation-related extensions) most of the issues would disappear as it would be deployed on a separate, optional table. Maybe this should have been an extension given those issues to facilitate the deployment- but everybody agreed that it is too late now for "what if...?".

What was being asked here initially was (implicitly) for WMF to fork Mediawiki code, and that was for me, a possible, but horrible solution. I think neither me at the start, nor many of the developers or the original deployers had realized this fact. We all made a mistake of thinking that this was really easy; in reality it was very complex, and on that meeting we all promised to do in the future, do the process better for now on (I think the schema-changes process is a step towards that).

Summary (ETA): This can now go ahead, and I believe I will be able to start working on this on 2 weeks' time, once the current ongoing schema change (T139090) finishes. How much this will take will depend on issues found on large wikis such as enwiki and commons; the current change was in theory easy and is probably going to take one month of non-stop (24/7) table rewriting.

Per my understanding, the consensus is to apply this to all wikis because schema inconsistencies are already causing a lot of trouble and no more of that is wanted.

Correct, as per above. I will be updating any news when I start working here. Ideally, I will also be working on finding and eliminating all inconsistencies in the following months T104459.

Thanks everyone, holding my breath then :)

Because T139090 needs more work and other more urgent changes (plus a week of deployment freeze) I have to delay this change.

However, once the ongoing changes finishes and the emergency ones are solved, this will be, as promised, the next to be scheduled.

Mentioned in SAL (#wikimedia-operations) [2016-11-10T14:53:51Z] <jynus> applying schema change on s3 (page) T69223

Note that with the above change, 95% of wikis will have the change applied, but not on the difficult ones (wikidata, commons, enwiki, ...). Wait for my ok before any deployment- there may be some servers that failed or were down while it was running.

Mentioned in SAL (#wikimedia-operations) [2016-11-15T18:23:30Z] <jynus> applying schema change on s6 (page) T69223

Mentioned in SAL (#wikimedia-operations) [2016-11-16T10:16:34Z] <jynus> applying schema change on s2 (page) T69223

Mentioned in SAL (#wikimedia-operations) [2016-11-16T16:08:01Z] <jynus> applying schema change on s7 (page) T69223

Mentioned in SAL (#wikimedia-operations) [2016-11-17T10:17:17Z] <jynus> applying schema change on s5 (page) T69223

Mentioned in SAL (#wikimedia-operations) [2016-11-17T15:24:47Z] <jynus> applying schema change on s4 (page) T69223

Mentioned in SAL (#wikimedia-operations) [2016-11-17T20:26:22Z] <jynus> applying schema change on s1 (page) T69223

This is the list of changes that failed to apply and failure was preferred over causing production issues (plus the ones affected by T151029):

$ while read host port; do while read db; do mysql -A -h $host --port $port $db -e "SHOW CREATE TABLE page\G" | grep --silent page_la
ng || echo "$host:$db"; done < s1.dblist; done < s1.hosts
labsdb1003.eqiad.wmnet:enwiki
labsdb1001.eqiad.wmnet:enwiki
db1083.eqiad.wmnet:enwiki
db1089.eqiad.wmnet:enwiki
db1051.eqiad.wmnet:enwiki
db1057.eqiad.wmnet:enwiki

$ while read host port; do while read db; do mysql -A -h $host --port $port $db -e "SHOW CREATE TABLE page\G" | grep --silent page_la
ng || echo "$host:$db"; done < s2.dblist; done < s2.hosts
labsdb1001.eqiad.wmnet:itwiki
db1076.eqiad.wmnet:enwiktionary

$ while read host port; do while read db; do mysql -A -h $host --port $port $db -e "SHOW CREATE TABLE page\G" | grep --silent page_lang || echo "$host:$db"; done < s3.dblist; done < s3.hosts
** All wikis on dbstore2001 (and 2002?) ****
labsdb1003.eqiad.wmnet:cebwiki
labsdb1003.eqiad.wmnet:cywiki
labsdb1003.eqiad.wmnet:dawiki
labsdb1003.eqiad.wmnet:etwiki
labsdb1003.eqiad.wmnet:iswiki
labsdb1003.eqiad.wmnet:ltwiki
labsdb1003.eqiad.wmnet:lvwiki
labsdb1003.eqiad.wmnet:nnwiki
labsdb1003.eqiad.wmnet:ocwiki
labsdb1003.eqiad.wmnet:urwiki
db1078.eqiad.wmnet:cebwiki
db1075.eqiad.wmnet:cebwiki


$ while read host port; do while read db; do mysql -A -h $host --port $port $db -e "SHOW CREATE TABLE page\G" | grep --silent page_lang || echo "$host:$db"; done < s4.dblist; done < s4.hosts
db2065.codfw.wmnet:commonswiki
db2058.codfw.wmnet:commonswiki
labsdb1001.eqiad.wmnet:commonswiki
db1056.eqiad.wmnet:commonswiki
db1059.eqiad.wmnet:commonswiki
db1084.eqiad.wmnet:commonswiki
db1091.eqiad.wmnet:commonswiki
db1040.eqiad.wmnet:commonswiki

$ while read host port; do while read db; do mysql -A -h $host --port $port $db -e "SHOW CREATE TABLE page\G" | grep --silent page_lang || echo "$host:$db"; done < s5.dblist; done < s5.hosts
labsdb1003.eqiad.wmnet:dewiki
db1082.eqiad.wmnet:wikidatawiki
db1087.eqiad.wmnet:wikidatawiki
db1092.eqiad.wmnet:wikidatawiki

$ while read host port; do while read db; do mysql -A -h $host --port $port $db -e "SHOW CREATE TABLE page\G" | grep --silent page_lang || echo "$host:$db"; done < s6.dblist; done < s6.hosts
labsdb1003.eqiad.wmnet:frwiki

root@neodymium:~$ while read host port; do while read db; do mysql -A -h $host --port $port $db -e "SHOW CREATE TABLE page\G" | grep --silent page_lang || echo "$host:$db"; done < s7.dblist; done < s7.hosts
db1094.eqiad.wmnet:metawiki
db1094.eqiad.wmnet:rowiki

Except Meta-Wiki (and secondarily Wikidata and Commons), those wikis don't need $wgPageLanguageUseDB = true. Can the wmgUseTranslate wikis other than those three proceed with the configuration ($wgPageLanguageUseDB = true and pagelang permission for sysop and translationadmin groups)?

those wikis don't need $wgPageLanguageUseDB = true. Can the wmgUseTranslate wikis other than those three proceed with the configuration ($wgPageLanguageUseDB = true and pagelang permission for sysop and translationadmin groups)?

Which wikis are you talking about? I mention every single wiki on my last post, unless you want to break production replication, I said:

Wait for my ok before any deployment

Mentioned in SAL (#wikimedia-operations) [2016-11-22T17:31:12Z] <jynus> performing schema change on db1078 (page) T69223

Mentioned in SAL (#wikimedia-operations) [2016-11-22T17:35:50Z] <jynus> performing schema change on db1075 (page) T69223

Mentioned in SAL (#wikimedia-operations) [2016-11-22T17:39:26Z] <jynus> performing schema change on db1076 (enwiktionary.page) T69223

Mentioned in SAL (#wikimedia-operations) [2016-11-22T17:47:25Z] <jynus> performing schema change on db1094 (metawiki.page) T69223

Mentioned in SAL (#wikimedia-operations) [2016-11-22T18:50:43Z] <jynus> trying schema change on db1082 (wikidatawiki.page) T69223

Mentioned in SAL (#wikimedia-operations) [2016-11-22T18:53:32Z] <jynus> trying schema change on db1057 (enwiki.page) T69223

I am unable to perform the schema change on some of the servers due to the high load on the page table (metadata locking):

labsdb1001.eqiad.wmnet
db1083.eqiad.wmnet
db1089.eqiad.wmnet
db1051.eqiad.wmnet
db1057.eqiad.wmnet
db1082.eqiad.wmnet
db1087.eqiad.wmnet
db1092.eqiad.wmnet

maybe others.

I will need to depool them, one by one, or in certain cases, like enwiki, setup the shard as read only/perform a master failover in order to apply it to the master, as it is marked on T138810 being a dependency.

Change 326493 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1051 for schema change

https://gerrit.wikimedia.org/r/326493

Change 326493 merged by jenkins-bot:
mariadb: Depool db1051 for schema change

https://gerrit.wikimedia.org/r/326493

Mentioned in SAL (#wikimedia-operations) [2016-12-13T08:30:05Z] <jynus> alter table on db1051, db1057 T69223

Change 326912 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1083 for maintenance and reboot

https://gerrit.wikimedia.org/r/326912

Change 326912 merged by jenkins-bot:
mariadb: Depool db1083 for maintenance and reboot

https://gerrit.wikimedia.org/r/326912

Mentioned in SAL (#wikimedia-operations) [2016-12-13T11:02:51Z] <jynus> alter table on db1083 T69223

Mentioned in SAL (#wikimedia-operations) [2016-12-13T12:14:37Z] <jynus> restart and upgrade of db1083 T69223

Mentioned in SAL (#wikimedia-operations) [2016-12-13T12:41:50Z] <jynus> alter table on db1087 T69223

Mentioned in SAL (#wikimedia-operations) [2016-12-13T12:56:00Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1087 - T148967 and T69223 (duration: 00m 47s)

Change 326931 had a related patch set uploaded (by Marostegui):
db-eqiad.php: Depool db1082

https://gerrit.wikimedia.org/r/326931

Change 326931 merged by jenkins-bot:
db-eqiad.php: Depool db1082

https://gerrit.wikimedia.org/r/326931

Mentioned in SAL (#wikimedia-operations) [2016-12-13T13:02:30Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1082 - T148967 and T69223 (duration: 00m 45s)

Mentioned in SAL (#wikimedia-operations) [2016-12-13T13:14:55Z] <jynus> alter table on db1082 - wikidatawiki T69223

Mentioned in SAL (#wikimedia-operations) [2016-12-13T13:29:28Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1082 - T148967 and T69223 (duration: 00m 45s)

Change 326980 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1089 for schema change

https://gerrit.wikimedia.org/r/326980

Only 3 servers left:

labsdb1001.eqiad.wmnet:enwiki
db1089.eqiad.wmnet:enwiki
db1092.eqiad.wmnet:wikidatawiki

I will probably be able to finish them tomorrow- labsdb1001, however, is suffering both metadata locking at times, and duplicate key problems at others:

ERROR 1062 (23000) at line 1: Duplicate entry '0-Feng_Shen_Bang_3D' for key 'name_title'
WARNING enwiki : page encountered problems

labsdb1001 is important because if it cannot be done, it will create infinite lag on labs.

After that, a last general check for potentially missing hosts will be needed.

Change 326980 merged by jenkins-bot:
mariadb: Depool db1089 for schema change

https://gerrit.wikimedia.org/r/326980

Mentioned in SAL (#wikimedia-operations) [2016-12-14T10:58:53Z] <jynus> alter table on db1089 - enwiki T69223

Change 327172 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1092 for schema change

https://gerrit.wikimedia.org/r/327172

Change 327172 merged by jenkins-bot:
mariadb: Depool db1092 for schema change

https://gerrit.wikimedia.org/r/327172

Mentioned in SAL (#wikimedia-operations) [2016-12-14T11:22:12Z] <jynus> alter table on db1092 - wikidatawiki T69223