• Bug 37531 has been marked as a duplicate of this bug. ***

On Jun 14, 2012, at 9:37 PM, Roan Kattouw wrote:

I'm not sure if the code exists in AFT5, but it should still be in AFT4. It's called the lottery. The principle is very simple:

if ( articleID % 1000 < lotteryOdds ) {

article wins lottery, gets AFT

} else {

article does not win lottery; do not pass go, do not collect $200

}

lotteryOdds is the number of articles out of a thousand that get AFT (so if lotteryOdds=15, then 1.5% of all pages get AFT, specifically the pages where the last three digits of the page ID are between 000
and 014 inclusive). Basing it on the page ID ensures that the set of pages that get AFT for a certain value of lotteryOdds is stable, and that articles are never dropped from, but only added to, this set as the value of lotteryOdds increases.

For deployment, you'd do something like this:

1. Deploy lottery code with lotteryOdds=0, check a few articles

(including one whose page ID ends in 000) to verify no one wins the
lottery

2. Increase the value of lotteryOdds a little
3. Wait for the change to propagate through the caches
4. Check that the lottery is working correctly. When raising

lotteryOdds from, say, 10 to 15, you would check:
4a. a page with a page ID ending in 009 or lower (should already have
won before, should still win now)
4b. a page with a page ID ending in 010 (should not have won before,
should win now)
4c. a page with a page ID ending in 014 (same)
4d. a page with a page ID ending in 015 (should not win just yet)

5. Wait for a while *after the caches have cleared* and keep an eye on

the health of things in Ganglia
5a. If bad things happen, drop lotteryOdds back to its previous value
and give up for now
5b. If things hold up OK, go back to step 2 and repeat until
lotteryOdds has reached 100 (=10.0%)

Notes:

• A reasonable sequence of values for lotteryOdds would be 1, 5, 10, 20, 50, 100
• To find articles whose page ID ends in a certain set of digits and

that are in the main namespace (otherwise AFT shouldn't show up
regardless), you can run the following query on the toolserver: SELECT
page_id, page_title FROM page WHERE page_namespace=0 AND page_id MOD
1000 = 15;

• Please run this on the toolserver (or another non-production

database replica like the one analytics has), don't run it on the
production database. IF you don't have toolserver access, apply for an
account or enlist someone who does (Dario?)

• The value of lotteryOdds should be exported to JS using the

ResourceLoaderGetConfigVars hook (don't use the
MakeGlobalVariablesScript hook for this! Its results are much more
aggressively cached for anons). Variables exported through this hook
have a low cache timeout (5 minutes) but unfortunately invalidation
isn't automatic, you have to trigger it yourself. To do this, run
"touch php-1.20wmfN/resources/startup.js" followed by "sync-file
php-1.20wmfN/resources/startup.js", then wait for the cache to clear.
The server-side cache should clear within 5 minutes, after that your
browser's cache may take up to 5 minutes as well

• To test whether the cache has cleared, check if AFT appears on

articles that didn't win the lottery before but should win it now (see
4a-4d), and test this in incognito mode

• After the caches have cleared, you should wait about 15 minutes

before increasing lotteryOdds again. The reasons for this are:

• After you start getting JS with the new value, it may take up to 5

minutes before everyone gets it. Because different people's browsers'
caches will expire at different moments, the change will gradually
reach readers over the course of about 5 minutes.

• Then once the change has reached everyone, you should wait 10

minutes to see if anything strange happens

• Bad things that might happen include:
  • A noticeable increase in load / network traffic / packet loss that

persists and that, when extrapolated to lotteryOdds=100, would be a
significant increase

• Services going down in Nagios (reported by nagios-wm in either

#wikimedia-tech or #wikimedia-operations , I forget which)

• If something goes down, comes back up, and stays up, it's probably

fine. If it stays down or keeps flapping, it's probably bad. When in
doubt as to whether your changes could've caused a certain Nagios
alert, ask ops

The lottery would be in addition to the whitelists and blacklist:

1. articles on the blacklist should not get AFT even if they win the lottery, and
2. articles on the whitelists should get it even if they lose the lottery.

Roan

[Fabrice note: To be even more specific, this method would honor existing black list and white list restrictions by removing from this 10% sample any article that is in the blacklist, or that is a re-direct. We would also continue to include in the 10% sample any article from there whitelists, which include both the hidden category 'Article_Feedback_5' and this 'Additional_Articles' category (so we can continue to manually add articles to it, as needed for special cases):
http://en.wikipedia.org/wiki/Category:Article_Feedback_5_Additional_Articles ]

Since this is a must-have for wider deployment at the end of June, let's plan to test if it works on en-wiki with a tiny sample this Tuesday, June 19 -- with the wider deployment to 10% the following week.

For this first test, we are aiming for a very tiny percentage: 0.01% of the encyclopedia, which should be about 350 articles total. We'll generate a list of selected article names and URLs, so we can test them quickly. It's just to see if the process works -- and if it impacts the encyclopedia in any way, across a range of pages.

On Jun 15, 2012, at 12:02 PM, Reha Sterbin wrote:

The lottery solution still leaves us with a problem: AFT4, which will appear on every non-blacklisted page unless we have a way for it to detect the presence of AFT5 and shut itself off. It's why we didn't go with a lottery for AFT5 the first time around.

Back then, I played around with detection in the javascript (AFT5 looking for AFT4), and I had some success, but if I recall correctly, there's no way to guarantee module loading order. If you know of a way to make AFT5 load second, we could try that out.

Another option might be to move the decision into successive hooks (e.g., one uses the earlier hook for a lottery, and the other uses the later hook to see what happened). I can try something out for that as well, depending on how much time I have in between other features and fixes.

-Reha

On Jun 15, 2012, at 12:27 PM, Fabrice Florin wrote:

Thanks, Reha, much appreciated!

Roan, what do you think about Reha's ideas below?

If none of these solutions are practical to implement on Monday for pre-deployment on Tuesday, Yoni also proposed a simpler solution, which would be to stick to the original plan of adding 320k articles to the Article Feedback 5 category, but doing it with a DB update script on the back-end, which might make the change less visible to the community.

Might this be a more practical option?

I am deeply aware that we are running out of time and that neither solution is perfect. So we may have to pick the solution which is most practical, even if it is not ideal.

Thanks for any suggestions you might have on this important issue.

All the best,

Fabrice

I've got a few ideas:

1/ Prerequisite: update config $wmgArticleFeedbackv5BlacklistCategories to also include the whitelisted AFTv4 categories.
Whitelisted AFTv4 articles should always show AFTv4 form, even if they're in AFTv5 lottery. So we should AFTv5 blacklist the AFTv4 whitelisted entries to make sure they're not shown twice.

2/ Since AFTv5's check is now in PHP (as well), we can manipulate more data. We could, on applicable pages winning AFTv5 lottery, clear out the AFTv4 lottery, like this (in beforePageDisplay() in ArticleFeedbackv5.hooks.php):

// disable older AFT versions by making the odds 0 (for this request)
$out->addJsConfigVars( 'wgArticleFeedbackLotteryOdds', 0 );

Disadvantage: we have to wait until the page's cache has been rebuilt

3/ Once we're certain that AFTv5 can be loaded (in JS), remove the elements added by AFTv4, like this:

var removeAft = function() {
    $aft = $( '#mw-articlefeedback' );
    if ( $aft.length > 0 ) {
        $aft.remove();
    } else {
        clearInterval( removeAftInterval );
    }
}
var removeAftInterval = setInterval( removeAft, 100 );

Disadvantage: it ain't pretty since we're allowing AFTv4 to first add itself and then remove parts of it. AFTv4 binds aren't removed, etc...

I'd suggest to add in all 3, and after 30 days (when cache should be fully rebuilt) remove 3/ again. This should make sure that cache does not affect our code (option 3/) and our codebase remains "clean" (removing 3/ after 2/ has been picked up)
Any suggestions/remarks on above idea?

Forgot to mention; the idea's above would allow us to make sure AFTv4 & AFTv5 don't collide, allowing us to add lottery in AFTv5.

I would add the lottery in both PHP and Javascript (the legacy checks currently saving us from the cache-problem)
1 thing to note though: talk page links WILL be messed up, they'll only start appearing once the talk page has been edited (or cache has otherwise been cleared), because we can't check the article's page id from javascript.

Commits on Gerrit:

• https://gerrit.wikimedia.org/r/#/c/11866/ (wmf config - 1/)
• https://gerrit.wikimedia.org/r/#/c/11867/ (extension - 2/ & 3/)

Partially pushed to prototype (11867)
Still have to push config part (11866) to prototype - tbd with Roan though first

reha wrote:

From Roan:

On Fri, Jun 15, 2012 at 12:01 PM, Reha Sterbin <reha@omniti.com> wrote:

The lottery solution still leaves us with a problem: AFT4, which will appear
on every non-blacklisted page unless we have a way for it to detect the
presence of AFT5 and shut itself off. It's why we didn't go with a lottery
for AFT5 the first time around.

Yeah, there is that. You could avoid this by reversing the lottery in
either AFT4 or AFT5, i.e. make one trigger on 000-009 and the other
one on 010-999.

Back then, I played around with detection in the javascript (AFT5 looking
for AFT4), and I had some success, but if I recall correctly, there's no way
to guarantee module loading order. If you know of a way to make AFT5 load
second, we could try that out.

You could hack something up using the loading-order-guaranteeing
primitives that we have in RL, but I think the reverse lottery thing
would be better.

tchay wrote:

I'd think the rev. lottery is simplest.

if ( articleID % 1000 < lotteryOdds ) {

unregister AFTv4 hook
gets AFTv5

}

Not sure what "unregister" would be like:

For instance: if AFTv4 code loads before AFTv5 then something like … array_splice($wgHooks[…], array_search(…, $wgHooks[…]), 1)

tchay wrote:

It occurs to me we register hooks in LocalSettings.php so it's probably going to be something more like

// remove AFTv4 hook
if ( articleID % 1000 < lotteryOdds ) {

register AFTv5

} else {

register AFTv4

}

Thanks all for pitching in.

(correct me if I'm wrong but) in LocalSettings.php, no article ID is known yet, it's all init stuff before the request actually gets interpreted and the real code launches.

Unregistering the AFTv4 code is pretty much impossible since:

• Roan said not to rely on extensions to be loaded in a particular order, and since
• the PHP isn't executed until the article gets edited and it's cache is cleared.

The only way I can think of to somewhat "unregister" AFTv4 is option 3/, where the elements added by AFTv4 get removed after they've been added - a nasty workaround only meant to hold up until pretty much all caches have been cleared.

The biggest problem is basically that:

• We can't have the checks in JS because of the talk page link (or we'd have to perform an additional AJAX-call for every talk page request)
• We can't have the lottery in PHP because it takes awhile to propagate to all articles (as caches are only cleared once an article is edited)

Reha will be updating this ticket though with another idea discussed with Roan; if I'm not mistaken, we'll be exposing the article ID to javascript so we can move the lottery to javascript (since we'll then know the article ID on the talk page link too). We'll hold of on the lottery for another week to make sure most caches have cleared and the article ID has been exposed for most articles. Something along those lines.

tchay wrote:

Oops, you're right LocalSettings is loaded first.

Well in any case, you, Reha and Roan figured a way around the problem during the meeting. :-)

I assume that since some of solution is JS-based, it can do a gradual rollout also? Not that this is a requirement.

Here's my general understanding of the plan I think you agreed on, in laymen's terms:

• we would provide article info to the Javascript code, so that it could determine whether or not to show AFT5

• we would expose the page ID on the talk pages this week, as a variable that JavaScript can use for this goal

• starting next week, the Javascript code would start the lottery process for articles that match our categories

I apologize if I got any of this wrong, and defer to Reha to write a more detailed technical description of our new plan.

But I wanted to let folks on this thread know that we are looking at 3 stages for this wider deployment:

1. June 19: deploy code to add page ID to talk page

2. June 26: deploy Javascript code to start the lottery with a tiny sample (0.001% of en-wiki - gradually increase that % if all goes well)

3. July 3: deploy AFT5 to 10% of en-wiki, then start socialization

Does this general plan work for everyone? did I miss anything? or get anything wrong? is it practical?

I also want to let you all know that if this plan proves to be too cumbersome or impractical, Howie is prepared to remove AFT4 from the entire encyclopedia, if it would make things a lot easier for you.

Reha, please let us know what you ended up with today, so that Matthias can complete any remaining tasks while we are all sleeping.

Our plan is to deploy that first version of the code tomorrow, on Tuesday, June 19 at 10am PT.

So now is the time to speak up if you have any serious reservations about this plan. The clock is ticking.

On Jun 18, 2012, at 11:21 PM, Reha Sterbin wrote:

In order to ramp up AFTv5 inclusion gradually, two things need to change:

1. AFTv5 needs to be aware of AFTv4 when selecting based on lottery: if the article "wins" AFTv4, AFTv5 needs to shut itself off. Category-based exclusion will remain as-is: lottery will only apply to articles with neither whitelisted nor blacklisted categories.

2. The lottery needs to be implemented in the javascript, so that changes can propogate within a relatively short timeframe. (Keeping it in the back end means that changes propogate as pages are edited, which could take up to 30 days.)

In order for the lottery to be implemented in the js for talk pages and and the special page, the hook must be updated to send the associated page ID to the javascript. That means that there will be a period where some pages have new javascript (expecting a page ID and performing a lottery check) but old markup (the page ID is not sent) -- in this case, I propose showing it: under the old scheme, it would only have gotten to the javascript if it passed all the required checks.

Matthias, you can find my work in the branch omnitiwork/js-lottery -- I believe I've got everything in, but I haven't had much time to test.

-Reha

On Jun 19, 2012, at 12:38 AM, Fabrice Florin wrote:

Thanks, Reha, nicely done!

I posted your helpful explanation below on Bugzilla as well, so we can track this project there, in a public record.

Matthias, it's all yours.

Please be very careful in your final implementation of this new code. We want to avoid accidentally displaying the feedback tool to a large user base prematurely, as we did a couple weeks ago.

In our initial test, we will want to only add a few hundred more pages through the lottery, beyond the current sample of 22k articles. That is on the order of 0.001% of the english encyclopedia. Once we confirm that this is working well, then we plan to gradually increase that lottery percentage all the way to 10% of en-wiki by July 3.

Also, remember that if you encounter a serious road-block due to having to support AFT4, we are prepared to remove AFT4 from the encyclopedia entirely, if this will reduce the risk and effort on your end.

Last but not least, please send Roan the prioritized list of all gerrit revisions we need him to review at least a couple hours prior to deployment tomorrow.

Thanks!

Fabrice

Fabrice:
It looks like we'll be able to start the lottery on next week's deployment.

I would like to "warn" about the cache though:

• By next week, not all articles will have been edited; the lottery will only start working for those that have been edited in the meantime (regardless of whether or not the talk page has been updated)
• Likewise, the talk page link will only start working at the moment the talk page has been edited (regardless of whether or not the article has been updated)
• Because the category-check has to be in PHP (we can't fetch it from the talk page JS), it suffers from caching; which means that when you white- or blacklist an article, it won't propagate immediately (well, it will on the article page, since you're editing it when adding a category; but it'll take until the talk page is edited for that link to be gone there)

Even though we should perfectly be able to start the lottery next week, it won't be perfect; the white-/blacklisting is a bit cripled as well.
I don't know how important the talk page link is or whether it could be an option to move the talk page link to the article itself (perhaps only for selected users)?
Suppose we were to get rid of the link on the talk page, we could reinstate everything to solely Javascript and we wouldn't suffer from the above-described issues.
We don't have to get rid of it; if it's useful and there's no similar alternative, we should definitely keep it around - just wanted to let you know the situation with the caches.

Thanks, Matthias, nicely done! Your caveats are well noted. I think we can live with a gradual increase as described above, as long as we get a critical mass of frequently-edited stories to surface once we start next week.

Two more questions remain: how will we be able to test this accurately? and how can we gradually increase the lottery? Do we need to schedule 15 mins. every few days to update the configuration file on en-wiki?

As discussed in today's weekly meeting, here is our plan for gradual deployment of AFT5 in coming days:

• Tue. June 25: increase to 1% of en-wiki (from 0.65% now)
• Thu. June 28: increase to 2%
• Tue. July 3: increase to 5%
• Thu. July 5: increase to 10%

Please let us know if you have any questions about this gradual deployment schedule.

Matthias and Reha, please let us know how we can test that the lottery is working as intended on prototype, testing and production.

It would be great if you could give us links to pages that should (or should not) be included in the lottery for prototype, testing and production.

It would also be wonderful if you could share a simple way for us to look up the article ID, and compare it to your formula, so we can tell by the last two digits if it should be in the lottery or not, for each of the sample sizes above. The most critical test, of course, will be the first 1% sample, which we will want to test tomorrow.

Below are some examples of pages in a certain percentile on enwiki. Everything below the percentage we're deploying shouldn't be visible - everything above the percentage should.

The lottery for the special page should work just fine for these articles (they aren't cached).
The article and talk page will have to have been edited since last week though, so the cache has been cleared. As long as the page is not yet edited, the lottery won't affect it and AFTv5 will remain invisible (though AFTv5 should be gone)

As Reha pointed out already, you can manually check an article's id by running "alert( mw.config.get( 'wgArticleId' ) );" in your javascript console. Look at the last 3 digits and see what category it fits.

1%: 990+
2491994 Aerinite
25528993 Agent_Orange_(video_game)

2%: 980+
16167982 Charmed_DVDs
1462983 Cheering

5%: 950+

35979 1_kilometre

16218959 1o_Sector

10%: 900+
2505943 1974_in_art
2453900 1976_NFL_season

20%: 800+
17751803 1989–90_FA_Cup
5371845 1829_in_architecture

50%: 500+
21378538 100_St_Georges_Terrace
11023716 007:_Licence_to_Kill

100%: 0+

 47264 Asteroid_belt
296474 Quarter

Thanks, Matthias, much appreciated!

For the record, I also enclose below Reha's notes about testing this on prototype.

On Jun 25, 2012, at 4:40 PM, Reha Sterbin wrote:

Prototype is ready for lottery testing.

I've set the lottery odds for AFT4 to 80%. This means that AFT5 will appear if the page ID ends in 801-999, and AFT4 will appear if the page ID ends in 000-800 -- unless the page is whitelisted/blacklisted by category.

Example test pages:

http://prototype.wikimedia.org/release-en/Harmony (page ID 1900, AFT5)
http://prototype.wikimedia.org/release-en/Central (page ID 1899, AFT5)
http://prototype.wikimedia.org/release-en/United_States_Census (page ID 1721, AFT4)
http://prototype.wikimedia.org/release-en/Kiribati (page ID 1527, AFT4)

If you'd like to try out other pages, you can get the page ID by typing the following in the javascript console:

alert( mw.config.get( 'wgArticleId' ) );

The AFT5 whitelist / AFT4 blacklist category is Article_Feedback_5. The AFT5 blacklist category Article_Feedback_Blacklist.

-Reha

Another list, now for test.wikimedia.org (previous was for enwiki)

1%: 990+
11995 Skin_Alert_Thingy
34993 Tsybenko

2%: 980+
40982 Page_D
36983 Space_bar

5%: 950+
12962 Thesaurus
32962 Transclusion_test

10%: 900+
36904 Parameters
38920 Planets_of_doom

20%: 800+
37895 Prueba_Wiki
32892 Rhubarb

50%: 500+
13697 Wazup
35684 Whatever

100%: 0+
39361 Tuisblad
12453 Ukraine

Thanks, Matthias!

Here is our revised schedule for wider deployment of AFT5 in July:

• Tue. July 3: deploy to 1%

• Thu. July 5: deploy to 2%

• Tue. July 10: deploy to 3 to 5% (depending on what we learn the previous week)

• Wed. July 11 to July 15: Fabrice, Howie and Oliver are at Wikimania in DC

(mostly unavailable, so no new increase, please)

• Mon. July 16: start central notice promotion (banner on every article = lots of community attention)

• Tue. July 17: deploy to 5% to 10% (depending on what we learn the previous week)

• Thu. July 19: Fabrice on vacation through July 22 (no new increase, please)

• Tue. July 23: deploy to 10% (if not already there)