Page MenuHomePhabricator

Run an A/B split test for VisualEditor enabled by default on enwiki for new accounts
Closed, ResolvedPublic

Description

This is to run from 19 June (end on with the Notifications split test, so we don't much their data up).


Version: unspecified
Severity: enhancement
URL: http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/70737
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=49727

Details

Reference
bz49604

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 1:52 AM
bzimport set Reference to bz49604.

Related URL: https://gerrit.wikimedia.org/r/68845 (Gerrit Change I5c603ece309d61641d32ccc9eff5ea2890d5b816)

Has en.wikipedia been notified?

Related URL: https://gerrit.wikimedia.org/r/68846 (Gerrit Change I94d74fd8abeba465f1f9463890541e8c8f786a43)

(In reply to comment #2)

Has en.wikipedia been notified?

This would be good.

From what I can tell, this test only applies to new users, though it's difficult to figure that out based on the current Bugzilla subject text or the related commit. It currently reads as though this will be an A/B on all users, not just new users.

It seems like auto-opting in a user also requires an auto-opt out mechanism. Otherwise, this is a dramatic departure from user expectations. As I understand it, anonymous users currently get the wikitext editor. Nearly every MediaWiki installation, including the one the user was just using use the standard wikitext editor, so suddenly changing that needs to be done cautiously.

Is there currently an auto-opt mechanism? If not, I consider that a blocker to this bug being resolved.

While I think it's ethically questionable to run experiments on users in this manner, one safeguard is ensuring that it's as easy as possible for users to _escape_ such experiments. Within the VisualEditor interface (and possibly elsewhere in the user interface), there should be one-click "I didn't ask for this" option to disable VisualEditor. In addition to an option on the editor itself, other areas of the site (such as the post-login screen) could be modified. "Welcome to Wikipedia! You've been selected to test our beta software. [opt out]" or similar.

(In reply to comment #0)

This is to run from 19 June (end on with the Notifications split test, so we
don't much their data up).

June 19 is a few days away. Given some of the recent feedback at [[Wikipedia:VisualEditor/Feedback]] regarding severe bugs in VisualEditor, it doesn't seem like VisualEditor is ready to be deployed more widely.

I fully agree with MZMcBride :

  • I think it's ethically questionable to run experiments on users in this manner
  • There are still severe bugs in VisualEditor at this time, so it shouldn't be deployed more widely

And I've more concerns about this kind of experiments :

  • There are still a lot of minor/major bugs in VisualEditor at this time, I don't expect new users to have to handle them when discovering Wikipedia edition.
  • There are still major limitations on what can be done with VisualEditor (templates, references, ...). What will be the experience for new users if they can only edit some parts of articles ?

I believe this kind of experiment should only be done when VisualEditor is fully functional (when users can do almost every modification they want) and without major bugs left.

From https://en.wikipedia.org/w/index.php?title=Wikipedia:Administrators%27_noticeboard&oldid=560544063#VisualEditor_-_A.2FB_test_launch_on_18_June:


Hey all. We're going to postpone the A/B test for several days; I'll post more details as I get them, but I understand it's largely down to known bugs with the existing software - bugs that you reported, and bugs that were crucial in making a go/no go decision. Thank you to everyone for all your hard work poking at the VE, and for all your reports thus far; it's much appreciated :). Okeyes (WMF) (talk) 19:25, 18 June 2013 (UTC)

Thanks to Legoktm for pointing this out to me, otherwise I likely never would've seen it.

This is now deployed and will run for a week or so to help advise on whether the main switch should be thrown for all users. Data to be published as we get it.

https://en.wikipedia.org/w/index.php?title=Minnesota&curid=19590&diff=561394244&oldid=561292962

Dirty diff, HTML comments being stripped, etc.

Will the VisualEditor team be cleaning up these edits?