Timo,

Let's use this bug for what we discussed. As I suggested, we should probably run on:

• enwiki featured articles (~ 4k), fixed revision (so if we regress we notice)
• enwiki ~ 5k most recently-changed articles (Special:RecentChanges)
• {en,fr,de,it,es,nl,he,ru,ar,ja,ko,vi}wiki ~ 5k random articles (Special:Random)

Thoughts?

(In reply to comment #1)
...

• {en,fr,de,it,es,nl,he,ru,ar,ja,ko,vi}wiki ~ 5k random articles

Could you add also 'pt' to this list?

(In reply to comment #2)

(In reply to comment #1)
...

• {en,fr,de,it,es,nl,he,ru,ar,ja,ko,vi}wiki ~ 5k random articles

Could you add also 'pt' to this list?

Sure. It was just writing a quick list rather than setting it in stone.

When we expand to cover language variants we'll want to expand the list further - for example, zh. :-)

So based on discussions with Gabriel:

• Parsoid has a better organised infrastructure for this than we do, so let's use that as a base. Right now they periodically run their sets of roundtrip tests on a certain set of articles.

1. Change that set of articles to include and/or match James' specification.
2. Improve ve-dirtydiffbot to not just do parsoid-ve-ve-parsoid roundtrip but also parsoid-ve-ve rountrip (e.g. parsoid dom > ve linmod > ve dom; "sanity check")

2. Extend the test runner to include 2 pieces of information for each article in addition to the data parsoid gathers:
   • result of parsoid-dom > ve linmod > ve dom ("sanity check")
   • diff of parsoid-dom > ve linmod > ve dom > parsoid dom ("full wikitext roundtrip")[1]

[1] this is the one that ve-dirtydiffbot is currently doing.

(In reply to comment #4)

So based on discussions with Gabriel:

• Parsoid has a better organised infrastructure for this than we do, so let's

use that as a base. Right now they periodically run their sets of roundtrip
tests on a certain set of articles.

1. Change that set of articles to include and/or match James' specification.

Include, not switch, please; the stuff that Parsoid is doing for RT tests should also be expanded, IMO.

• Bug 56330 has been marked as a duplicate of this bug. ***

We currently have https://github.com/wikimedia/ve-dirtydiffbot, which:

• For 720 random en.wikipedia.org pages:
• Load VE
• Trigger save dialog
• Assert diff is empty

There's a potential race condition of an edit having occurred and there actually being a diff or edit conflict. Though we've never hit this so far.

The current dirtydiff bot has not successfully run since April 2014.

As of February 2015, we no longer have the "Sanity check" method.

Before reviving this test (the code is a bit outdated, no longer works) and expanding its test input, I'd like to revisit what it is doing.

With Parsoid's selective serialisation mechanism (selser) this may not be a very useful test. I feel like I'm missing something. Isn't this nothing but a test of the browser's ability to parse and serialise HTML? What part of VE, related to different pages as input, would this test? Other than on-load exceptions for exotic content and such. Which is not the primary purpose of this test. Should we involve linmod somehow? Or is it already? What steps do we want the test to do instead?