It seems fixing bug 58876 made the logged requests from ssl
terminators jump from ~50/day to 1.3M/day [1] ... which is actually good.
As the ~1.3M/day seems more plausible than ~50/day ... Did some of the
(maybe) relevant changes
https://gerrit.wikimedia.org/r/#/c/105449/ https://gerrit.wikimedia.org/r/#/c/105450/ https://gerrit.wikimedia.org/r/#/c/105451/
really unbreak edit.tsv writing for ssl terminators as a side-effect?
If so, how could they have unbroken storing requests from ssl
terminators ... should not x-encoded requests have gotten logged
nonetheless?
(They for sure were in the stream before, as I checked before writing
the changes above, and some are visible in the edit tsvs for the
previous dates.)
Was there some network issue beforehand?
Was udp-filter (see [4] for its invocation) misbehaving?
Once we understand what went on, would we need fixes elsewhere?
Is the 1.3M/day now finally about correct?
Some more pointers:
- The above gerrit changes got deployed to ssl1001 on 2014-01-14 ~18:56 [2].
- Heavy ssl traffic in the edit tsvs starts on 2014-01-14T18:59:11.634 and is limited to ssl1001 at first. On 2014-01-14T19:06:23.209 other machines kick in.
- There was a deployment window on 2014-01-14 19:00-21:00 upgrading group1 1.23wmf9->1.23wmf10 [3], but it looks unrelated, as we're seeing ssl requests pour in for unaffected wikis.
[1]
qchris@stat1002 0 22:13:50
cwd: ~
zgrep -c '^ssl' /a/squid/archive/edits/edits.tsv.log-201401*.gz
[...]
/a/squid/archive/edits/edits.tsv.log-20140112.gz:47
/a/squid/archive/edits/edits.tsv.log-20140113.gz:18
/a/squid/archive/edits/edits.tsv.log-20140114.gz:17
/a/squid/archive/edits/edits.tsv.log-20140115.gz:557830
/a/squid/archive/edits/edits.tsv.log-20140116.gz:1307081
/a/squid/archive/edits/edits.tsv.log-20140117.gz:1336017
/a/squid/archive/edits/edits.tsv.log-20140118.gz:1345793
/a/squid/archive/edits/edits.tsv.log-20140119.gz:1379590
/a/squid/archive/edits/edits.tsv.log-20140120.gz:1343811
/a/squid/archive/edits/edits.tsv.log-20140121.gz:1358968
[2] See
http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-operations/20140114.txt
around 18:54, look for ottomata & qchris.
[3] https://wikitech.wikimedia.org/wiki/Deployments/Archive/2014
Version: unspecified
Severity: normal