It would probably be worth fixing bug 50982 first, while you can easily see the empty selser changes in the output file. Subbu thinks these might be the tests "without wt2wt parsoid option ... would be good to atleast verify/confirm that hypothesis."

Re: comment 1. Unfortunately, that doesn't appear to be the case. A counter example is "Parsoid only: Quote balancing context should be ..." which has the options "parsoid=wt2html,wt2wt".

Change 76870 had a related patch set uploaded by Arlolra:
Generate selser change assignments dynamically.

https://gerrit.wikimedia.org/r/76870

Change 76870 merged by jenkins-bot:
Generate selser change assignments dynamically.

https://gerrit.wikimedia.org/r/76870

3rd bullet point in bug description is actually bug 51718 -- need to figure out best approach for this (work through what is best -- technique as outlined in #3 here or something else).

With generate & test our assignments are not guaranteed to be exhaustive, which might be relevant for bug 52139. It might be worth moving to direct permutation generation instead, as that should also make change generation faster.

gwicke: In what way are these permutations? From the blacklist,

add("selser", "Non-word characters don't terminate tag names (bug 17663, 40670, 52022) [[3],3,[3],3,4,3,4,4,4,2,3]");

this just looks like combinations with replacements. Given that there are 11 numbers between 2 and 4 inclusive, you'd have to generate 3^11 changes, rather than 20 random ones. That doesn't seem faster.

Deterministic generation will be faster than random generate & test, as the latter will often result in duplicates which are then filtered out. Keep in mind that we try to generate a random assignment up to 1000 times, even if there are only a handful possible permutations in a small test. The extra attempts to generate permutations will just generate duplicates once the few possible permutations have been found.

I agree that we'll need to limit the number of permutations we generate for large test cases. That means that generating all permutations with the current assignments won't be possible. On the bright side, there is a chance that we can get away with less permutations without really losing test coverage. As an example, case 2 (node insertion before current node) and case 4 (child node insertion) can result in the same actual change, so should probably be collapsed when that happens. Similarly, new node insertion is very similar to attribute changes for selser processing: the full 'outerwikitext' needs to be serialized in both cases. Lets discuss the possible cases and think about which cases need to be handled.

Change 85952 had a related patch set uploaded by Arlolra:
WIP: Remember the output of failing (blacklisted) tests

https://gerrit.wikimedia.org/r/85952

• Bug 49222 has been marked as a duplicate of this bug. ***

Change 85952 merged by jenkins-bot:
Remember the output of failing (blacklisted) tests

https://gerrit.wikimedia.org/r/85952

Gabriel: All patches merged months ago - is there more work left here, or can you close this ticket as RESOLVED FIXED?

We have a good workable solution for now, but I think Gabriel had the enhancement idea of generating selser tests by going through permutations. Gabriel: do you want to create a different enhancement ticket for it and close this one?

(In reply to ssastry from comment #13)

We have a good workable solution for now, but I think Gabriel had the
enhancement idea of generating selser tests by going through permutations.
Gabriel: do you want to create a different enhancement ticket for it and
close this one?

Gabriel: ping?

Lets keep using this bug, but reclassify it as an enhancement.

Our selser test coverage can be improved further. Generating permutations systematically still seems to be a promising candidate solution for doing so.