Page MenuHomePhabricator

Different results with queries in labs versus production
Closed, ResolvedPublic

Description

Different results with queries in labs versus production around september 21st.

The following query returns different (very different) results in production that it does in labs:

Production:

SELECT count(log_user)

 FROM enwiki.logging
      /* exclude proxy registrations */
WHERE log_type = 'newusers'
      /* only include self-created users, exclude attached and proxy-registered users */
  AND log_action = 'create'
  AND log_timestamp BETWEEN 20140921000000 AND 20140922000000;

Returns: 8027

Labs:

SELECT count(log_user)

 FROM logging
      /* exclude proxy registrations */
WHERE log_type = 'newusers'
      /* only include self-created users, exclude attached and proxy-registered users */
  AND log_action = 'create'
  AND log_timestamp BETWEEN 20140921000000 AND 20140922000000;

Returns: 6842

Halfak did some digging and placed the missing rows in

analytics-store:staging.missing_labs_new_user_20140921

It looks like there's 5 hours of the day where the rows were missing.

mysql:research@analytics-store.eqiad.wmnet [staging]> select LEFT(log_timestamp, 10) as hour, count(*) from missing_labs_new_user_20140921 GROUP BY 1;
+------------+----------+

hourcount(*)

+------------+----------+

201409210883
2014092109336
2014092110304
2014092111344
2014092112118

+------------+----------+
5 rows in set (0.01 sec)


Version: unspecified
Severity: normal
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=72226

Details

Reference
bz72413

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:43 AM
bzimport added a project: Cloud-VPS.
bzimport set Reference to bz72413.

Sync in progress. Cause is not yet confirmed, with https://mariadb.atlassian.net/browse/MDEV-6551 a possibility.

Very interested to hear if anyone observes this with recent (<1week) data.

Krenair subscribed.

Checked db2034.codfw.wmnet, labsdb1001.eqiad.wmnet, labsdb1002.eqiad.wmnet, and labsdb1003.eqiad.wmnet. All return 8027, so assuming this got fixed at some point.