Page MenuHomePhabricator

Route SSL traffic again into webstatscollector's filter process
Closed, ResolvedPublic

Description

When emery got decomissioned, erbium around mid-April 2014 had it's
udp2log instance turned from a multicast consumer into a consumer of
only varnishncsa. Hence, since then, erbium only receives logs for the
traffic the ssl terminators cause on the caches, but not the logs for
traffic arriving at the ssl terminators.

However, to correctly count ssl traffic, webstatscollector's filter
(which runs on erbium) relies on seeing the logs at the ssl terminators.

So as a result, ssl traffic was not counted in webstatscollector's
output since mid-April 2014.

Discussing a bit with ottomata, it seems the best way forward [1] [2]
is to try moving webstatscollector's filter to a host that still
consumes the udp2log multicast.

oxygen consumes the udp2log multicast, and does not look overloaded,
so we could try moving webstatscollector's filter there.


[1] We could instead turn multicast consumption on erbium on again. But
that would also affect all the other udp2log filters that meanwhile got
added to erbium. So it would cause further issues.

[2] We could update webstatscollector to stop dropping local http
requests from the ssl terminators. That would be nice as it would mean
that the overall picture of the webstatscollector pipeline gets
simpler. However, people are afraid to touch webstatscollector code,
and we want to phase it out anyways.


Version: unspecified
Severity: normal
Whiteboard: u=Community c=General/Unknown p=0 s=2014-06-26

Details

Reference
bz67456

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:30 AM
bzimport set Reference to bz67456.

Change 143775 had a related patch set uploaded by QChris:
Feed logs from ssl terminators again into webstatscollector's filter

https://gerrit.wikimedia.org/r/143775

As tomorrow (2014-07-04) is a holiday in the US, today (2014-07-03) is the
last working day of this week, and it's bad karma to deploy on the last
working day of a week. So it means that merging the fix will have to wait
until next week. :-/

Change 143775 merged by Gage:
Feed logs from ssl terminators again into webstatscollector's filter

https://gerrit.wikimedia.org/r/143775

The first hourly webstatscollector files after the above merge contain
ssl requests again. Thanks Gage!