Page MenuHomePhabricator

Improve performance of round-trip test setup
Closed, ResolvedPublic

Description

Our current round-trip test server is bogging down with the large DB it has accumulated by now. This makes it the bottleneck in rt testing, leaving the clients mostly idle.

Much of the db size is XML-encoded old test results, which we don't really need any more. Moving those large results to a separate db might make it easier to truncate old results by simply re-creating the result xml db.

Apart from the DB size, the node.js sqlite bindings we use don't seem to help performance either. IIRC they don't support transactions and other performance-improving features / pragmas. We mainly used sqlite because it was easy to get started, but it might make sense to re-evaluate that choice. A separate DB server would at least make it possible to use two cores instead of just one.


Version: unspecified
Severity: normal

Details

Reference
bz44652

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 1:33 AM
bzimport added a project: Parsoid-Tests.
bzimport set Reference to bz44652.

Another issue with the current code is that marking repeated crashing titles as failed does not work reliably. There is a counter in the claims table that should mark the title as an error on reaching some number of retries.

(In reply to comment #2)

Another issue with the current code is that marking repeated crashing titles
as
failed does not work reliably. There is a counter in the claims table that
should mark the title as an error on reaching some number of retries.

I've proposed a patch that should fix this:
https://gerrit.wikimedia.org/r/#/c/73985/

Change 75895 had a related patch set uploaded by Marcoil:
Refactor the database schema for performance. To use only the pages table to determine the next title to be processed, store the latest claim (hash, timestamp and number of tries) and latest score for each page. Then, to get the next title query for the o

https://gerrit.wikimedia.org/r/75895

Change 75895 merged by jenkins-bot:
Refactor the database schema for performance.

https://gerrit.wikimedia.org/r/75895

Fix is in rt_testing branch, reopen if more work is necessary before merging into master.