Page MenuHomePhabricator

update.php stuck on update revision on large db
Closed, InvalidPublic

Description

Hi, I am working on deployment.wmflabs.org together with Mark Herhberger

I imported full db of simple wiki there to mediawiki 1.18wmf1 using mwimport, the wiki db was ok, however when I attempted to upgrade to latest head of svn, the update.php stuck on updating rev_sha1 on revision table

the process is now running, but seems to be idle,

desc revision;
+----------------+---------------------+------+-----+----------------+----------------+

FieldTypeNullKeyDefaultExtra

+----------------+---------------------+------+-----+----------------+----------------+

rev_idint(10) unsignedNOPRINULLauto_increment
rev_pageint(10) unsignedNOMULNULL
rev_text_idint(10) unsignedNONULL
rev_commenttinyblobNONULL
rev_userint(10) unsignedNOMUL0
rev_user_textvarbinary(255)NOMUL
rev_timestampbinary(14)NOMUL
rev_minor_edittinyint(3) unsignedNO0
rev_deletedtinyint(3) unsignedNO0
rev_lenint(10) unsignedYESNULL
rev_parent_idint(10) unsignedYESNULL

+----------------+---------------------+------+-----+----------------+----------------+

it seems that update script didn't even start with generation of sha_1 keys for some reason, when I tried this on smaller db I've seen the output after few seconds, however on this db it stuck for some reason. Did anyone experienced anything similar? is it a bug of update.php or there is a better way to convert table


Version: 1.20.x
Severity: normal
OS: Linux

Details

Reference
bz33558

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 12:00 AM
bzimport set Reference to bz33558.
bzimport added a subscriber: Unknown Object (MLST).

petrb@deployment-dbdump:/tmp$ tail nohup.out
...user_last_timestamp is already nullable.
...user_email key already set on user table.
...up_property in table user_properties already modified by patch patch-up_property.sql.
...uploadstash table already exists.
...user_former_groups table already exists.
...config table already exists.
...type_action key already set on logging table.
...batch conversion of user_options: nothing to migrate. done.
...user table does not contain user_options field.
Adding rev_sha1 field to table revision...

this is there in a log for several hours

You could run populateRevisionSha1.php.

sounds like this should block deployment or at least tarball.

(In reply to comment #3)

sounds like this should block deployment or at least tarball.

Definitely doesn't block deployment, we don't use update.php.

Does that mean that the Adding rev_sha1 field to table revision... does not output anything? I guess it should either:

  • output something every X sha1 are computed
  • skip that step and raise a nice warning explicitly asking user to run populateRevisionSha1.php

(In reply to comment #5)

Does that mean that the Adding rev_sha1 field to table revision... does not
output anything? I guess it should either:

  • output something every X sha1 are computed
  • skip that step and raise a nice warning explicitly asking user to run

populateRevisionSha1.php

Indeed, with it potentially being really slow, there's little point actually just automagically running it IMHO

Oh wait. It's not an issue populating the field, it's the issue of it taking a long time (Asher suggests about 30 hours for enwiki size) to add the field to the revision table

no actually it didn't work at all, after 10 hours of run it didn't even update structure of db, no update of table was done either

I will check the details soon, the db itself (all tables) have around 38gb, it
was a complete dump of simple db

this query was run after the start of update.php

desc revision;
+----------------+---------------------+------+-----+----------------+----------------+

FieldTypeNullKeyDefaultExtra

+----------------+---------------------+------+-----+----------------+----------------+

rev_idint(10) unsignedNOPRINULL

auto_increment |

rev_pageint(10) unsignedNOMULNULL
rev_text_idint(10) unsignedNONULL
rev_commenttinyblobNONULL
rev_userint(10) unsignedNOMUL0
rev_user_textvarbinary(255)NOMUL
rev_timestampbinary(14)NOMUL
rev_minor_edittinyint(3) unsignedNO0
rev_deletedtinyint(3) unsignedNO0
rev_lenint(10) unsignedYESNULL
rev_parent_idint(10) unsignedYESNULL

+----------------+---------------------+------+-----+----------------+----------------+

it's clear that it didn't even update structure so that I doubt it would actually work. But it happened to me only on larger db's

I've no idea how table alters work if you cancel them, rather than letting them execute through

I didn't cancel anything that is how table was looking during the update, I mean that it was running for hours and meanwhile there was no activity in the sql server

(In reply to comment #13)

I didn't cancel anything that is how table was looking during the update, I
mean that it was running for hours and meanwhile there was no activity in the
sql server

What did SHOW PROCESSLIST output during that time of seemingly no activity?

removing from tarball blocker:

<hashar> I guess we want to try to reproduce 33558 with Peter using the labs
<hashar> then we will be able to investigate the issue
<hashar> not much we can do right now

Umherirrender edited subscribers, added: Umherirrender; removed: wikibugs-l-list.

There is

...doing $idCol from $blockStart to $blockEnd

on the populateRevisionSha1.php script.

But this is about the ALTER to add the new field. For big database that could be a long running operation.
I see nothing to fix here after 8 years.