Page MenuHomePhabricator

Thanks: add instrumentation to Thanks
Open, MediumPublic

Description

User story & summary:

As the Product Manager of the team that maintains Thanks, I want to better understand Thanks usage, so we can make data-informed decisions about any feature improvements in the future.

As a Wikimedian, I want to understand how Thanks is used, because that might help clarify is there is an issue with the current confirmation UX.

Background & research:

This task is important because some editors, especially newcomers, find the current Thanks confirmation confusing. Some even think that Thanks is sent privately after the initial click on Thanks. It's currently difficult to understand how widespread this confusion is, and adding instrumentation should provide at least some general data to help decide if additional effort to improve Thanks is needed.

  • Can we find out how many thanks are not being completed on desktop? That is, the user clicks on a thank button, but does not click on the confirmation button.
  • Does the level of non-completion differ by user experience level (perhaps total number of edits, or perhaps total number of completed thanks)?
  • The ultimate question is whether the confirmation step is likely to be preventing misclicks/unwanted thanks, or if it is mostly stopping wanted thanks. If it stops very few wanted thanks, then it might make sense to duplicate this confirmation step on mobile.

See Also:
T63737: Thank notification on mobile should support click to undo
T73360: Instrument click through tracking on Thank, and confirm steps
T51087: Specify which edit was thanked in Special:Log/thanks, both for private and public records' sake, if configured to do so
T324134: Communication: connecting Thanks to a specific edit
T333027: Growth design: finalize designs for thanks confirmation UX

Instrumentation Details:

Thanks instrumentation should help answer the following questions:

  • Where does Thanks originate?
    • Device: Mobile vs. Desktop vs. Apps
    • Page: Revision history, Diff, Watchlist, Recent changes, Contributions,
  • After Thanks is clicked, how often is does the following happen:
    • user does nothing further (Thanks is not sent)
    • user clicks "Cancel" (Thanks is not sent)
    • user confirms and sends Thanks ("Thank" is clicked again after the "Publicly send thanks?" prompt).
  • How much does the above differ based on a "Thanker's" experience level?
    • TBD: should we use the same tenure buckets as used elsewhere (Turnilo) or should we use edit count buckets?
Acceptance Criteria:

Given I'm reviewing Thanks data,
When I want to evaluate how Thanks is used for a specific project,
Then I have the data I need to understand:

  • the origin of thanks (device & page)
  • how often Thanks is confirmed, cancelled, or abandoned
  • how this differs by the Thanker's experience level

Details

Reference
bz69804

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Legoktm subscribed.

What is the restricted project that this is a part of?

Dumb user question: why is this a research question? Generally, it is standard UX practice to not require confirmation for non-destructive changes, and specifically, no social network I've ever seen requires confirmation for stars/likes/etc. It doesn't seem like it is a good use of research's time/energy to confirm pretty widely understood practice..

(Which is to say, of course, that there should clearly be a bug for "remove the confirmation step in giving thanks".)

Dumb user question: why is this a research question? Generally, it is standard UX practice to not require confirmation for non-destructive changes, and specifically, no social network I've ever seen requires confirmation for stars/likes/etc. It doesn't seem like it is a good use of research's time/energy to confirm pretty widely understood practice..

This has been discussed in the see also bugs. A major difference that has been noted that stars/likes have an undo option whereas thanks don't. And there's a bug for that somewhere.

The confirmation for "Thank" was added because on history pages, it is directly next to the (rollback | undo) links, and was added to the end of that bracketed line (rollback | undo | thank) which led to embarrassing errors, of thanking a vandal. T49658: Thanks workflow needs further thought
As soon as we have a way to fix these accidents, we can remove the confirmation step - possibilities are discussed in T71636: Thanks: Implement an undo feature - e.g. a server-side time-delay before sending the "thank", enabling a user to click cancel (like gmail offers for sending an email - a 20 second window)

Unless there is a rationale for adding analytics beyond just "confirm what we already know", then I would suggest that any programming efforts would best be put towards creating the Thank-Undo code instead of this. I suggest closing this task as Declined.

We have a current problem with embarrassing thanks on mobile. (Steps to reproduce: Navigate to a page with a big green 'thanks' button. Drop your expensive mobile device. Try to catch it before it hits the floor and breaks. Do you have any idea what buttons your fingers might have touched when you caught it?).

Real users who have suffered real problems with this button on mobile were told that the confirmation step would not be implemented on mobile unless there was data proving that people actually misclicked this button. Thus we have a request to get this data. If the users could just have what they asked for, which is a simple confirmation step for a proven problem, then this request could be closed as needless.

Whatareyoudoing-WMF Umm... People have been asking for an undo feature since thanks came out. It wasn't until the WMF team said they didn't want to spend any more time on such a lightweight product that they decided we would have to live with a confirmation step instead. This is still unacceptable and makes thanks not worth the time to use. I do hope the WMF rethinks their position on this as right now it takes me less time to send a wikilove message than it does to thank for an edit. I have to use wikilove for everything else from thanking for blocking a sockpuppet and thanking for patrolling an edit of mine and thanking for protecting a page and thanking for deleting a page in my userspace, might as well use it to thank for edits too since there is no easier alternative....

Technical13, I don't see any problem with providing both, but they haven't actually provided the Undo feature. Since they haven't provided the Undo feature (which requires a lot of work), then the people who are having this problem have requested that they provide the already-existing-on-desktop confirmation step on mobile, too. Porting an existing detail to mobile should not require nearly as much work as writing a new feature for both mobile and desktop.

We have a current problem with embarrassing thanks on mobile. (Steps to reproduce: Navigate to a page with a big green 'thanks' button. Drop your expensive mobile device. Try to catch it before it hits the floor and breaks. Do you have any idea what buttons your fingers might have touched when you caught it?).

Real users who have suffered real problems with this button on mobile were told that the confirmation step would not be implemented on mobile unless there was data proving that people actually misclicked this button. Thus we have a request to get this data. If the users could just have what they asked for, which is a simple confirmation step for a proven problem, then this request could be closed as needless.

Oh, right, the mobile issue, T63737: Thank notification on mobile should support click to undo. Yes, I agreed there that the button is very large (currently not working per T77929), and I agree that a confirmation-step, or undo-step, or opt-out would be helpful. (Maybe a CSS class, so that PamD can hide the button, at the very least?)

I'm not sure how we'd get any kind of proof-via-analytics though, because it would require guessing (or psychic powers) to know why anyone cancelled a Thank, no?

Maybe, but this was the data request that they made. They said that they
won't fix the reported problem until they have "data" to show that it's an
actual problem.

Whatamidoing (WMF)/Sherry Snyder
Community Liaison
Wikimedia Foundation, Inc.

Then I'd suggest that the problem isn't going to be fixed: R&D is currently heavily oversubscribed with work. This is a low-priority request, at the moment, and won't be jumped on until that changes/someone runs out of things to do.

Amire80 subscribed.

(Added the StructuredDiscussions project, because I expect Thanks to be pretty important in Flow.)

The confirmation is not shown/required for Flow thanks.

Correction, there is a confirmation for Flow on mobile, but that seems to be a byproduct of the module registration details (see T89565: In Flow on mobile web thanks should not require confirmation step, like on desktop).

Can we A/B test out a click to undo feature and see what impact it has? I think this would be a really useful reusable component to have. Do we have a task to update the API to do that?

Can we A/B test out a click to undo feature and see what impact it has? I think this would be a really useful reusable component to have. Do we have a task to update the API to do that?

I don't think anyone's really against click to undo if it's done properly. (Obviously, it only makes sense with delayed delivery, since it's somewhat meaningless to undo a notification once the recipient has already seen it)

It doesn't seem worth it to implement the API just to A/B test it. (The current task is not requesting that, just funnel data for the current confirmation feature on desktop). If we're going to implement the undo API, we should just do the undo feature for real and be done with it.

Related: https://gerrit.wikimedia.org/r/#/c/163289/

See T63737 - @Mattflaschen I'm happy to do the frontend legwork if someone can help me add support in the api to cancel a thanks. Is suspect all that needs to happen is when you thank you get an id with which you can use in an api request to cancel in the same session.

I started on the backend cancelability of events in https://gerrit.wikimedia.org/r/#/c/163289/ but never got around to finish it. It needs a trivial API endpoint written that accepts the cancel token and passes it into EchoEventCanceler. Not sure if this also needs some sort of security added ensuring the token belongs to the user making the API request. Also have to adjust Thanks to enable cancelability, and return the generated token from the thanks API request.

Removing Research & Data and moving to Research Ideas board.

KStoller-WMF renamed this task from Thanks tool: How much does the confirmation button cost us? to Thanks: add instrumentation to Thanks.Mar 24 2023, 6:49 PM
KStoller-WMF raised the priority of this task from Lowest to Medium.
KStoller-WMF updated the task description. (Show Details)

One thing to note re: the new acceptance criteria is that the mobile diff UI has a completely different thanking workflow (you click a button, there's a popup saying you are thanking, and a "cancel" link, if you don't click it within a few seconds, the thank gets sent).

The moderator tools team are currently adding instrumentation to diff pages that may be helpful here. Cc @Samwalton9

The moderator tools team are currently adding instrumentation to diff pages that may be helpful here. Cc @Samwalton9

See T326216: Log additional click events on Special:MobileDiff - we'll be tracking Thanks clicks.

Tgr removed Tgr as the assignee of this task.
Tgr claimed this task.
In T71804#8735359, @Tgr wrote:

One thing to note re: the new acceptance criteria is that the mobile diff UI has a completely different thanking workflow (you click a button, there's a popup saying you are thanking, and a "cancel" link, if you don't click it within a few seconds, the thank gets sent).

Flow's is also different; there's no confirmation step or grace period to cancel the thanks.

Moving to unblocked as T321754 was released.
@Tgr please unassign yourself if you don't plan to do this work. Thanks!