bingle-admin wrote:

Prioritization and scheduling of this bug is tracked on Mingle card https://wikimedia.mingle.thoughtworks.com/projects/analytics/cards/cards/1439

Further investigation:

*I went through some of the requestlogs manually and found duplicate requests, about 9-10ms apart, the latter of which had the MIME type and referer stripped. This could be the source of both the MIME type data loss and the referer data loss we've seen with Special:BannerRandom hits. Matt Walker theorises that the problem may be us consuming requestlog data from multiple layers of varnish machines, and thus getting the same requests multiple times. I'm going to yank out the hostnames for the weird hinky hits I've noticed to see.

(In reply to comment #0)

Today, though, I encountered requests without a MIME type.

Requests without a MIME type are fine in many settings.
We're seeing many of them.

Since you seem to be able to reproduce, could you provide a short snippet
that allows to exhibit such a log line?

(I am not asking for the log line itself, but for some chain of actions that
allows us to see a log line that you are concerned about)

Alright, you want to hunt for:

*hits to uri_path /wiki/File:Thailand_Surin_locator_map.svg
*Between 2014-01-20T10:14:00 and 2014-01-20T10:15:00

(hopefully that's anonymised enough)

From that particular example, it looks like the (intact) request was a MISS from the varnish cache's point of view, which explains the immediate repeat of the request. Whether it's also responsible for the lack of referrer data is too network engineer-y for me to know - but it is a potential limiter if we want to use MIME type filtering for say, pageviews. The good news is that, assuming my data sample is representative (and it's probably off, since it's 128k mobile views from a specific date), this only happens about 0.03 percent of the time.

*Blinks* actually, looking at that example, the MIME type is intact, it's the referrer that's vanished. My brain is...clearly not on today.

Closing for now, since there doesn't seem to be any easy link to identify why the referers and such are missing. Blah. I need to do a lot more work before BZing things, I think.