Page MenuHomePhabricator

Requests for 'undefined' page increasing on wikipedias
Closed, ResolvedPublic

Description

Requests matching

http://$WIKI.wikipedia.org/wiki/[uU]ndefined

have risen over the last few days, currently 40M-50M/day.

Since >99% of them come with a referrer from a corresponding wikipedia
page [1], they look wrong ... like some JavaScript requesting page
from a variable, where the variable is undefined.

Browsers identify basically as Chrome (~80%) and Firefox (~19%).
Mostly in recent versions (Chrome 35: ~75%, Firefox 29: ~17%).

Browsers identify the operating system mostly as Windows (~97%).

Requests are mostly (sorted by decreasing number of 'undefined'
requests) for enwiki, ruwiki, frwiki, plwiki, itwiki, eswiki, and to a
lesser extent also other wikis. But some of the wikis with overall many
requests do not seem affected, like jawiki, zhwiki, or dewiki.

It seems noteworthy that the client starts to request the 'undefined'
page only once the originating page and all it's assets have been served
to the client. And the client is not only requesting the 'undefined' page
once, but continuously again and again [2] for somewhere between some
seconds up to some 15 minutes. Then it stops at once, either because a
new page is requested by the client or we stop seeing requests from the
client altogether.

By the affected browsers/OS it looks a bit like it could be related to
bug 66112. But the symptoms are sufficiently different.

[1] Like a request for

http://en.wikipedia.org/wiki/undefined

coming with a referrer of

http://en.wikipedia.org/wiki/Cook_County,_Illinois

, and

http://pl.wikipedia.org/wiki/undefined

coming with a referrer of

http://pl.wikipedia.org/wiki/Eva_Green

.

[2] At a rate between 4-20 times per second. This rate seems to be
constant per client.


Version: unspecified
Severity: normal
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=72102

Details

Reference
bz66352

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 3:10 AM
bzimport set Reference to bz66352.
bzimport added a subscriber: Unknown Object (MLST).

The drastic increase of 'undefined' requests seems to start on 2014-06-02
somewhere between 11:00 and 14:00 UTC.

Hrm. What time are we deploying at these days? Monday is definitely a deploy day, is why I ask.

When I hear 'undefined' I think 'Wikidata', but that's just an idle thought. It might be worth checking out the release notes for whatever went out Monday.

(In reply to Oliver Keyes from comment #2)

What time are we deploying at these days?

15:00 UTC onwards.
I of course checked the wikitech's Deployments page, but saw nothing relevant.

And yes, I of course also checked SAL, which does not seem to have
anything relevant either.

Need collaboration with external team (not sure who) to work on this further.

I am adding Timo and Trevor to this bug because I cannot answer any of their questions. Thanks folks.

[undefined] pageviews to wikipedia as of last month

[undefined] pageviews to wikipedia as of last month. These requests are getting to be 5% of our traffic, that is 100 of million.

Attached:

undefined_pageviews.png (360×648 px, 50 KB)

Request data can be found, for example, here: http://dumps.wikimedia.org/other/pagecounts-raw/2014/2014-10/

Or in stat1002:/a/squid/archive/sampled

Requests for 'undefined' pageviews for the month of october

Requests for 'undefined' pageviews for the month of october

attachment undefined-october.txt.gz ignored as private

Different IPs that send requests to undefined October

Different IPs that send requests to undefined October

attachment ips-sorted-with-counts.txt ignored as private

Change 165350 had a related patch set uploaded by QChris:
Stop counting requests for 'undefined' and 'Undefined'

https://gerrit.wikimedia.org/r/165350

Change 165351 had a related patch set uploaded by QChris:
Release fix that stops counting [uU]ndefined

https://gerrit.wikimedia.org/r/165351

Change 165378 had a related patch set uploaded by QChris:
Stop considering requests for 'undefined' and 'Undefined'

https://gerrit.wikimedia.org/r/165378

Change 165395 had a related patch set uploaded by QChris:
[webstatscollector] Add '[uU]ndefined' condition

https://gerrit.wikimedia.org/r/165395

Change 165350 merged by Ottomata:
Stop counting requests for 'undefined' and 'Undefined'

https://gerrit.wikimedia.org/r/165350

Looked at centranotice campaigns today to find whether there was a campaign scheduled on May and September where we can see the event going on. We found no common campaigns. WSM (wiki summer of monuments) was turned on on Sep 1st to Oct 1st for US,UM (us islands),VI, PR but we see 'undefined' requests across all wikis.

Looking at data for october in detail, there seems to be a bunch of repeats on referrers.

Referers for "wiki/undefined" pageviews for 09/10

Odd, they should be more widespread.

 18 http://es.wikipedia.org/wiki/Historia_del_Sahara_Occidental
 21 http://es.wikipedia.org/wiki/Placa_base
 21 http://pt.wikipedia.org/wiki/SAP_ERP
 24 http://da.wikipedia.org/wiki/Jordsk%C3%A6lvet_i_Baluchistan_2013
 26 http://en.wiktionary.org/wiki/causa
 28 http://en.wikipedia.org/wiki/Douglas_A-26_Invader
 29 http://pt.wikipedia.org/wiki/The_Big_Bang_Theory
 36 http://es.wikipedia.org/wiki/F%C3%ADsica
 39 http://pt.wikipedia.org/wiki/Marginaliza%C3%A7%C3%A3o
 44 http://es.wikipedia.org/wiki/Alonso_N%C3%BA%C3%B1ez_de_Haro
 46 http://en.wikipedia.org/wiki/Neodymium_magnet
 54 http://en.wikipedia.org/wiki/PJM_Interconnection
 55 http://en.wikipedia.org/wiki/Event_horizon
 66 http://en.wikipedia.org/wiki/Title_21_CFR_Part_11
 71 http://en.wikipedia.org/wiki/D%C3%A9tente
 99 http://pt.wikipedia.org/wiki/Yeopgijeogin_geunyeo
137 http://en.wikipedia.org/wiki/Hedda_Gabler
217 http://it.wikipedia.org/wiki/Person_of_Interest_(serie_televisiva)
854 http://en.wikipedia.org/wiki/Mandatory_Palestine

Regarding UA (as Christian noted earlier) windows is the winner:

101 Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36
118 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/537.36
138 Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36
139 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36
140 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36
141 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36
163 Mozilla/5.0 (Windows NT 6.2; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0
174 Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36
178 Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36
182 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36
183 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36
207 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0
217 Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36
236 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36
259 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.102 Safari/537.36
267 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36
354 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36
408 Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36
416 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36
416 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
655 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36
830 Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36
835 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36
879 Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36
934 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36
1912 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36

Change 165351 merged by Ottomata:
Release fix that stops counting [uU]ndefined and redirects

https://gerrit.wikimedia.org/r/165351

These threads might be relevant?

'Complitly' is a recurring theme

http://wizzley.com/url-adding-undefined-on-chrome/

http://stackoverflow.com/questions/11017609/undefined-randomly-appended-in-1-of-requested-urls-on-my-website-since-12-jun

Several theories here, again scan for 'complitly'

https://productforums.google.com/forum/#!topic/chrome/G1snYHaHSOc

"The problem seems to appear for browsers that have a plugin called "complitly" installed (http://www.complitly.com/) .
This plugin installs as a windows software and adds a plugin for the installed browsers (Chrome and IE).
When the browser reads a web page with an input field named "search" (or "recherche" in French and I suppose "search" translated in any language), the plugin tries to retrieve a page named "undefined" (I guess, this is a bug in the plugin).

(In reply to Erik Zachte from comment #23)

'Complitly' is a recurring theme

Great find!

request for [uU]ndefined 2012-2014

Charts for daily requests since Jan 2012, show 'complitly' hit us on 12 June 2012, as expected from http://stackoverflow.com/questions/11017609/undefined-randomly-appended-in-1-of-requested-urls-on-my-website-since-12-jun

Of course that doesn't mean much larger spikes in recent month are necessarily explained.

Note small peak in April 2014 (test run for malware?)

Attached:

We've decided that as a team, we've done what we can do on this. I've let Mark B. know and he's in agreement.

Christian has written a fix which we will deploy this week.

We need to decide what level of reporting updates we want to do.

-Toby

jgage wrote:

Have we tried contacting Compility?

Report a problem:
http://www.complitly.com/Faq.aspx#na36

Opt-out for a site:
http://www.complitly.com/Faq.aspx#na44

Change 165378 merged by Ottomata:
Stop considering requests for 'undefined' and 'Undefined'

https://gerrit.wikimedia.org/r/165378

Fix has been deployed on 2014-10-15 ~19:01 and is effective.

The last affected files are

http://dumps.wikimedia.org/other/pagecounts-raw/2014/2014-10/pagecounts-20141015-200000.gz [1]
http://dumps.wikimedia.org/other/pagecounts-raw/2014/2014-10/projectcounts-20141015-200000 [1]
http://dumps.wikimedia.org/other/pagecounts-all-sites/2014/2014-10/pagecounts-20141015-190000.gz
http://dumps.wikimedia.org/other/pagecounts-all-sites/2014/2014-10/projectcounts-20141015-190000

The first files without the [uU]ndefined counts are

http://dumps.wikimedia.org/other/pagecounts-raw/2014/2014-10/pagecounts-20141015-210000.gz [1]
http://dumps.wikimedia.org/other/pagecounts-raw/2014/2014-10/projectcounts-20141015-210000 [1]
http://dumps.wikimedia.org/other/pagecounts-all-sites/2014/2014-10/pagecounts-20141015-200000.gz
http://dumps.wikimedia.org/other/pagecounts-all-sites/2014/2014-10/projectcounts-20141015-200000

[1] When restarting collector and filter for the C implementation of
webstatscollector, where was a period (<2 minutes) where the new collector and
the old filter have been running. Hence, during this
perioud a few [uU]ndefined made it the 20:00:00 file.

Change 165395 merged by Ottomata:
[webstatscollector] Add '[uU]ndefined' condition

https://gerrit.wikimedia.org/r/165395