Page MenuHomePhabricator

cp301[34] mobile traffic not in udp2log's mobile-sampled-100 and zero stream
Closed, ResolvedPublic

Description

cp301[34] seem to be getting mobile traffic [1] since 2014-04-18, and
are enabled in pybal [2].

But they are not listed in puppets manifests/role/cache.pp, so their
requests are not included in udp2log's mobile-sampled-100 stream [3].
They also do not show up in udp2log's zero stream.

Kafkatee streams are not affected. We see cp301[34] requests there.

[1]


qchris@stat1002 0 22:29:26
cwd: ~
zgrep '^cp3013' /a/squid/archive/sampled/sampled-1000.tsv.log-20140419.gz | head -n 1 | cut -f 1,3
cp3013.esams.wikimedia.org 2014-04-18T19:03:07


qchris@stat1002 0 22:29:49
cwd: ~
zgrep '^cp3014' /a/squid/archive/sampled/sampled-1000.tsv.log-20140419.gz | head -n 1 | cut -f 1,3
cp3014.esams.wikimedia.org 2014-04-18T19:03:04

[2]
http://noc.wikimedia.org/pybal/esams/mobile

[3]


qchris@stat1002 0 22:30:07
cwd: ~
zgrep -c '^cp301[34]' /a/squid/archive/mobile/mobile-sampled-100.tsv.log-20140419.gz
0


Version: unspecified
Severity: normal

Details

Reference
bz64145

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 3:20 AM
bzimport set Reference to bz64145.
bzimport added a subscriber: Unknown Object (MLST).

bingle-admin wrote:

Prioritization and scheduling of this bug is tracked on Mingle card https://wikimedia.mingle.thoughtworks.com/projects/analytics/cards/cards/1559

Change 127456 had a related patch set uploaded by QChris:
Add cp301[34] to mobile caches

https://gerrit.wikimedia.org/r/127456

(In reply to christian from comment #0)

They also do not show up in udp2log's zero stream.

Not sure where I had my eyes ... cp301[34] requests do show up in zero files.

They are only missing from mobile-sampled-100 stream.

Thanks Christian -- need anything else on this?

-Toby

(In reply to Toby Negrin from comment #4)

Thanks Christian -- need anything else on this?

I do not think we need anything else here.
Ottomata said that the change looks ok.
Waiting for BBlack to chime in on the change.

This bug is a side-effect of bug 64240, and cp301[34] currently acting
as frontend caches only.

BBlack said that he'll start add cp301[34] as backends as well, but
connectivity issues are currently preventing that (see RT 6360).

The plan is to add 301[34] separately on different days to lessen the
impact on cache hits. Probably, one day per machine. This process
“should” start today. BBlack is actively working on it.

Plan looks good to me. But that also means that it'll probably take a
few more days until the issue is fully resolved.

Ops changed plans, and they removed cp301[34] from the set of frontend
varnishes again.

They'll redo the machines differently, and will not turn them on until
they are ready as backends as well.

I checked pybal, an cp301[34] are gone again.

I'll check mobile-sampled-100 tomorrow (when the new tsvs are available),
if it is again as expected.

Checked mobile-sampled-100 from udp2log against previous weeks, and also
against kafkatee files and zero files.

cp301[34] are gone again for relevant requests, and number of lines is now
within expectations.

Change 127456 abandoned by BBlack:
Add cp301[34] to mobile caches

Reason:
These have been added now (separately, and as backends only for now until lvs config issues are sorted out) and should get analytics properly

https://gerrit.wikimedia.org/r/127456