Page MenuHomePhabricator
Create Task
Maniphest T72600

"Age of unreviewed changesets by affiliation" shows negative number of changesets
Closed, ResolvedPublic
Actions

Description

In "Age of unreviewed changesets by affiliation" at http://korma.wmflabs.org/browser/gerrit_review_queue.html, the popup window shows the number of patches counted in the median for each affiliation. However, many of the numbers provided are negative, which doesn't make any sense.

I had already reported via email that those numbers looked very low considering the amount of changesets uploaded every month. This was when the numbers were still positive. Seeing negative values suggests that there is indeed something wrong there.

Version: unspecified
Severity: normal
URL: http://korma.wmflabs.org/browser/gerrit_review_queue.html

Details

Reference
bz70600

Related Objects
Search...

StatusSubtypeAssignedTask
 DuplicateQgilT125545 Phabricator Q&A session for Community Liaisons
 ResolvedQgilT116025 Goal: Align Community Liaison and Developer Relations project management practices
 ResolvedQgilT119387 Community Liaison and Developer Relation quarterly goals for January - March 2016
 InvalidNoneT98348 Implement the Wikimedia Foundation Call to Action 2015
 InvalidNoneT98358 WMF to integrate, consolidate, and pause or stop stalled initiatives
 ResolvedAklapperT78639 How to address the long tail of low priority tasks in active projects
 DeclinedNoneT113707 Decision on Wikimedia Foundation service-level agreement on code review times
 DeclinedNoneT113706 Define Cut-off date for abandoning old changesets in our code review tool
 OpenNoneT78768 Agree on and implement actions to prioritize code review of patches submitted by volunteers
 ResolvedQgilT101099 Developer Relations Roadmap
 ResolvedQgilT113030 Developer Relations quarterly review Jul-Sep 2015
 ResolvedQgilT101100 Engineering Community quarterly goals for July-September 2015
 ResolvedQgilT109829 Developer Relations quarterly goals for October-December 2015
 ResolvedAklapperT101686 Goal: Define potential actions to reduce code review queues and waiting times
 ResolvedNoneT113028 Gerrit clean up day - releng
 ResolvedAklapperT113378 Lessons learned from the first Gerrit Cleanup Day
 ResolvedAklapperT88531 Goal: Organize a Gerrit Cleanup Day on September 23, 2015
 ResolvedDicortazarT72600 "Age of unreviewed changesets by affiliation" shows negative number of changesets

Event Timeline

• bzimport raised the priority of this task from to High.Nov 22 2014, 3:45 AM
• bzimport added a project: wikimedia.biterg.io.
• bzimport set Reference to bz70600.
• bzimport added a subscriber: Unknown Object (MLST).
Qgil created this task.Sep 9 2014, 8:26 AM
Qgil added a comment.Sep 9 2014, 8:26 AM

Marking High importance because I would really like to leave the Gerrit Review Queue page absolutely clean.

Qgil added a comment.Sep 9 2014, 9:32 AM

In fact, this graph should show the media age of their most recent uploads, just like we did with "Age of open changesets" and "Ranking of repositories".

Finally, could you change "scr_review_time_pending_ReviewsWaitingForReviewer_days_acc_median" for a human readable string, please?

Acs added a comment.Sep 10 2014, 12:23 PM

Ok, the new metric (upload) is generated also for companies, so we can move to this metric and then, fix the number of open issues per company using this new metric.

The new metric has already a human label, so the other error will be fixed also.

As soon as it is available I will update this ticket.

Acs added a comment.Sep 11 2014, 8:12 AM

Quim, problem fixed. Now the pending number per company is right. The problem was that we are showing the new net pending per month, not the total pendings.

Qgil added a comment.Sep 11 2014, 9:41 AM

Now there are no negative numbers, but they still look low. For August we have:

Unknown: 66
Wikimedia Foundation: 55
Independent: 123
Wikimedia Deutschland: 29
TOTAL: 273

However, according to "Volume of open changesets", in August there are 765 waiting for review.

765 - 273 = 492

Where are these 492 changesets missing?

By the way, now we have "scr_review_time_pending_upload_ReviewsWaitingForReviewer_days_acc_median", which is still not very readable.

Qgil added a comment.Sep 28 2014, 8:50 PM

Álvaro, when do you think we will have a solution for this problem? This is the last bug in http://korma.wmflabs.org/browser/gerrit_review_queue.html stopping me from promoting this page widely.

Acs added a comment.Sep 29 2014, 4:44 AM

I plan to review all labels and help texts today and next, work on this issue. So expect a solution, or an explanation of the numbers, next two days!

Cheers

Acs added a comment.Sep 29 2014, 3:17 PM

Quim, the number of reviews now show the correct number.

The main problem was that there are new identities not linked to Unknown.

With this fixed the numbers are much closer.

We have changed the metric for affiliations to "Time from last patchset". Is it ok or you prefer "Time from submission"?

With this metric:

Unknown: 666
WMF: 36
Independent: 10
WMD: 55
WikiWorks: 0

total: 767.

According to the above graph, "Time from last patchset" number of revisions should be 816. For other dates the error is less.

A deeper analysis could be done, but I am pretty confident the data is correct.

Qgil added a comment.Sep 29 2014, 5:27 PM

Progress!

It is strange to find so many Unknown. Any developer with a @wikimedia.org email address is a WMF employee. Any developer with a @wikimedia.de address is a WMDE employee. Are we applying these rules?

Acs added a comment.Oct 6 2014, 3:20 AM

Quim, reviewed the mapping, now we have:

Unknown 479
WMF 206
Individual 124
WMD 8
Wikia 26

total: 843.

We need to improve the mapping to have less Unknown people in any case.

Qgil added a comment.Oct 6 2014, 12:28 PM

Thanks. Hm, after this change, Wikia has a very different line from all the rest. Are we still looking at "waiting for reviewer"? If so, this is worth looking at. Before there was a link to the raw data, but now I can't find it. Could you paste somewhere the gerrit change numbers that are open and owned by Wikia?

Also, if you paste a list of the unknown at https://www.mediawiki.org/wiki/Talk:Community_metrics , we will help identifying the affiliation of these contributors.

Qgil lowered the priority of this task from High to Low.Jan 8 2015, 11:25 AM
Qgil added a project: ECT-March-2015.Mar 2 2015, 8:15 AM

Proposing this task for the ECT-March-2015 sprint

Qgil claimed this task.Mar 2 2015, 10:05 AM
QChris unsubscribed.Mar 2 2015, 6:48 PM
Qgil raised the priority of this task from Low to Medium.Mar 13 2015, 1:27 PM
Qgil reassigned this task from Qgil to Dicortazar.Mar 13 2015, 1:40 PM
Qgil added a subscriber: Dicortazar.

Ok, here the suspicion was that the values of previous months changed as the graph was updated at the end of each month. Silly me, I didn't capture the current data to check those suspected changes... I have captured the data of the last 6 months now.

The graph is stuck in January. @Dicortazar, can you kick it so it is updated to February, please? Then I will check whether the current values for previous months stay. If everything is correct, we can resolve this task.

Dicortazar added a comment.Mar 17 2015, 10:51 AM

Those are in February indeed.

Do you mind double checking this? In addition, there's also information for March (incomplete), although given that there are two lines in the chart, we typically wait till the end of the month to show such info.

Qgil added a comment.Mar 17 2015, 11:35 AM
In T72600#1124747, @Dicortazar wrote:

Those are in February indeed.

In my browsers the graph ends in January:

Screenshot_from_2015-03-17_12:33:48.png (900×1 px, 159 KB)

Dicortazar added a comment.Mar 26 2015, 1:35 PM

Also in mine. Checking this!. I though you referred to a previous one.

Dicortazar added a project: ECT-April-2015.Apr 1 2015, 3:39 PM
Dicortazar set Security to None.
Qgil added a comment.Apr 6 2015, 3:15 PM
In T72600#1124857, @Qgil wrote:

In my browsers the graph ends in January:

... and this is the case still today. This task was planned for March. Do you have an estimated delivery time?

Dicortazar added a comment.Apr 22 2015, 2:33 PM

The dataset is now updated, but the pop-up does not work. Working on this now.

As an estimation, this should be done by the end of April.

Qgil added a project: ECT-May-2015.Apr 30 2015, 9:40 AM
Qgil added a project: ECT-June-2015.EditedMay 29 2015, 6:58 AM

There are still negative numbers.

And also: the title of this graph says "affilation", "scr_review_time_pending_upload_ReviewsWaitingForReviewer_days_acc_median" should be a human-readable string, and a "?" explaining the details of the metric should be added.

Qgil mentioned this in T39463: Key performance indicator: Gerrit review queue.May 29 2015, 7:03 AM
Qgil added a parent task: T88531: Goal: Organize a Gerrit Cleanup Day on September 23, 2015.May 29 2015, 7:11 AM
Qgil added a parent task: T94578: Most basic Tech Community metrics are published and up to date.
Qgil mentioned this in T100189: "Median time to review for Gerrit Changesets, per month": External vs. WMF/WMDE/etc patch authors.May 29 2015, 7:21 AM
Aklapper removed a parent task: T94578: Most basic Tech Community metrics are published and up to date.Jun 16 2015, 4:11 PM
Aklapper added a project: ECT-July-2015.
Qgil added a comment.Jul 3 2015, 11:51 AM

The WMF Annual Plan 2015-16 includes a goal related to this task:

Set and monitor code review KPIs for all community-sourced contributions

Solving this task will help identifying "community-sourced" contributions.

Acs added a comment.Jul 30 2015, 4:22 PM

http://korma.wmflabs.org/browser/gerrit_review_queue.html

Fixed the name of the metric and the number of pending changesets per affiliation. No more negative numbers (we were showing the net changesets pending, not the total changesets pending used to compute the age).

The help is not shown yet ... we are debugging a problem with it.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 30 2015, 4:22 PM
jayvdb subscribed.Jul 31 2015, 3:33 AM
In T72600#1495009, @Acs wrote:

http://korma.wmflabs.org/browser/gerrit_review_queue.html

Fixed the name of the metric and the number of pending changesets per affiliation. No more negative numbers (we were showing the net changesets pending, not the total changesets pending used to compute the age).

The help is not shown yet ... we are debugging a problem with it.

'GitHub' in the list of affiliations for "Age of open changesets by affilation (monthly snapshot)" looks ... wrong, and especially bad that GitHub is increasing to be the second highest in May 2015...?

Qgil added a comment.Jul 31 2015, 10:31 AM

@Acs, this looks good to me now. Thank you!

@jayvdb, removing the false affiliations is being addressed in T100189: "Median time to review for Gerrit Changesets, per month": External vs. WMF/WMDE/etc patch authors.

Qgil closed this task as Resolved.Jul 31 2015, 10:33 AM
Qgil reopened this task as Open.Aug 11 2015, 9:36 AM

Sorry, I have to reopen. In July, FSF has -2 and Deutsche Telekom has -1. This means that the algorithm is still wrong.

Note that if T100189 would have been fixed, we would have probably not seen these negative numbers... How can we trust that WMF is today "49.4"?

This is an important problem.

Qgil added a project: ECT-August-2015.Aug 11 2015, 9:37 AM
Aklapper added a comment.Aug 24 2015, 12:46 PM

@Dicortazar: Any news here when it comes to fixing the underlying problem?

Aklapper moved this task from Backlog to Ready to Go on the wikimedia.biterg.io board.Aug 27 2015, 10:38 AM
Aklapper raised the priority of this task from Medium to High.Aug 28 2015, 9:36 AM
Aklapper added a project: DevRel-September-2015.
Acs added a comment.Aug 31 2015, 5:55 AM
In T72600#1567113, @Aklapper wrote:

@Dicortazar: Any news here when it comes to fixing the underlying problem?

Fixed. The problem was with the library version for viz. Updated to the last version, including the fix already done for this issue, and now all is working in

http://korma.wmflabs.org/browser/gerrit_review_queue.html

Aklapper closed this task as Resolved.Aug 31 2015, 9:32 AM
In T72600#1588421, @Acs wrote:

Fixed. The problem was with the library version for viz.

Thanks a lot! I'm closing this task as resolved.

I've filed some followup tasks for other smaller/minor issues I ran into with that chart:

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL