Page MenuHomePhabricator

Requests through ssl terminators hardly present in edits tsvs before 2014-01-14
Closed, DeclinedPublic

Description

It seems fixing bug 58876 made the logged requests from ssl
terminators jump from ~50/day to 1.3M/day [1] ... which is actually good.

As the ~1.3M/day seems more plausible than ~50/day ... Did some of the
(maybe) relevant changes

https://gerrit.wikimedia.org/r/#/c/105449/
https://gerrit.wikimedia.org/r/#/c/105450/
https://gerrit.wikimedia.org/r/#/c/105451/

really unbreak edit.tsv writing for ssl terminators as a side-effect?

If so, how could they have unbroken storing requests from ssl
terminators ... should not x-encoded requests have gotten logged
nonetheless?
(They for sure were in the stream before, as I checked before writing
the changes above, and some are visible in the edit tsvs for the
previous dates.)

Was there some network issue beforehand?
Was udp-filter (see [4] for its invocation) misbehaving?

Once we understand what went on, would we need fixes elsewhere?

Is the 1.3M/day now finally about correct?

Some more pointers:

  • The above gerrit changes got deployed to ssl1001 on 2014-01-14 ~18:56 [2].
  • Heavy ssl traffic in the edit tsvs starts on 2014-01-14T18:59:11.634 and is limited to ssl1001 at first. On 2014-01-14T19:06:23.209 other machines kick in.
  • There was a deployment window on 2014-01-14 19:00-21:00 upgrading group1 1.23wmf9->1.23wmf10 [3], but it looks unrelated, as we're seeing ssl requests pour in for unaffected wikis.

[1]


qchris@stat1002 0 22:13:50
cwd: ~
zgrep -c '^ssl' /a/squid/archive/edits/edits.tsv.log-201401*.gz
[...]
/a/squid/archive/edits/edits.tsv.log-20140112.gz:47
/a/squid/archive/edits/edits.tsv.log-20140113.gz:18
/a/squid/archive/edits/edits.tsv.log-20140114.gz:17
/a/squid/archive/edits/edits.tsv.log-20140115.gz:557830
/a/squid/archive/edits/edits.tsv.log-20140116.gz:1307081
/a/squid/archive/edits/edits.tsv.log-20140117.gz:1336017
/a/squid/archive/edits/edits.tsv.log-20140118.gz:1345793
/a/squid/archive/edits/edits.tsv.log-20140119.gz:1379590
/a/squid/archive/edits/edits.tsv.log-20140120.gz:1343811
/a/squid/archive/edits/edits.tsv.log-20140121.gz:1358968

[2] See
http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-operations/20140114.txt
around 18:54, look for ottomata & qchris.

[3] https://wikitech.wikimedia.org/wiki/Deployments/Archive/2014

[4] https://git.wikimedia.org/blob/operations%2Fpuppet/aa4f6383ac7fc8dc5515d8466574368658f959ed/templates%2Fudp2log%2Ffilters.oxygen.erb#L13


Version: unspecified
Severity: normal

Details

Reference
bz60314

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 2:54 AM
bzimport set Reference to bz60314.
bzimport added a subscriber: Unknown Object (MLST).

bingle-admin wrote:

Prioritization and scheduling of this bug is tracked on Mingle card https://wikimedia.mingle.thoughtworks.com/projects/analytics/cards/cards/1398