Page MenuHomePhabricator
Create Task
Maniphest T74175

Diamond logstash monitor fills /var/log/apache2 access log
Closed, ResolvedPublic
Actions

Description

deployment-logstash1.eqiad.wmflabs ended up with a filled /var/

/var/log/apache2/other_vhosts_access.log was filled with lines such as:

logstash.beta.wmflabs.org:80 127.0.0.1 - - [17/Oct/2014:08:16:52 +0000] "GET /server-status HTTP/1.1" 301 592 "-" "Python-urllib/2.7"
logstash.beta.wmflabs.org:80 127.0.0.1 - - [17/Oct/2014:08:16:52 +0000] "GET /server-status HTTP/1.1" 301 592 "-" "Python-urllib/2.7"
logstash.beta.wmflabs.org:80 127.0.0.1 - - [17/Oct/2014:08:16:52 +0000] "\x16\x03\x01" 301 308 "-" "-"
logstash.beta.wmflabs.org:80 127.0.0.1 - - [17/Oct/2014:08:16:52 +0000] "\x16\x03\x01" 301 308 "-" "-"

`/var/log/diamond/diamond.log had a lot of:

[2014-10-14 19:33:37,250] [Thread-1] Error retrieving HTTPD stats for host 127.0.0.1:80, url '/server-status?auto': [Errno 99] Cannot assign requested address

I guess something is (was?) wrong in the Diamond collector used to monitor logstash.

I have deleted the access.log (freeing up 850MB) file and restarted diamond.

As of Nov 25th, the diamond.log is just fine. There is still a lot of requests made to /server-status though.

Version: unspecified
Severity: normal

Details

Reference
bz72175

Related Objects
Search...

Event Timeline

• bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:45 AM
• bzimport added a project: Beta-Cluster-Infrastructure.
• bzimport set Reference to bz72175.
• bzimport added a subscriber: Unknown Object (MLST).
hashar created this task.Oct 17 2014, 8:20 AM
hashar added a comment.Oct 17 2014, 8:29 AM

+ Yuvi Panda: seems the diamond collector for logstash spams the apache access log with garbage requests such as ""\x16\x03\x01" :D

hashar added a comment.Oct 17 2014, 9:14 AM

That might be due to monitor fatal using an HTTPS connection:

modules/beta/files/monitor_fatals.rb:27:https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/fatalmonitor

And the sequence "\x16\x03\x01" would mean an attempt to establish an SSL connection on port 80 which does not have mod_ssl.

greg triaged this task as Medium priority.Nov 24 2014, 11:26 PM
greg moved this task from To Triage to Next: Maintenance on the Beta-Cluster-Infrastructure board.Nov 24 2014, 11:51 PM
hashar updated the task description. (Show Details)Nov 25 2014, 1:44 PM
hashar set Security to None.
yuvipanda closed this task as Resolved.Mar 5 2015, 1:30 PM
yuvipanda claimed this task.

No such instances are found atm, and monitor_fatals.rb is dead.

yuvipanda moved this task from Next: Maintenance to Done on the Beta-Cluster-Infrastructure board.Mar 5 2015, 1:31 PM
hashar reopened this task as Open.Mar 12 2015, 9:18 AM

Reopening, that is still happening. The monitoring is using the vhost logstash.beta.wmflabs.org thus the spam ends up in /var/log/apache2/other_vhosts_access.log.

Example:

logstash.beta.wmflabs.org:80 127.0.0.1 - - [12/Mar/2015:09:14:24 +0000] "GET /server-status HTTP/1.1" 301 592 "-" "Python-urllib/2.7"
logstash.beta.wmflabs.org:80 127.0.0.1 - - [12/Mar/2015:09:14:24 +0000] "GET /server-status HTTP/1.1" 301 592 "-" "Python-urllib/2.7"
logstash.beta.wmflabs.org:80 127.0.0.1 - - [12/Mar/2015:09:14:24 +0000] "\x16\x03\x01" 301 308 "-" "-"
logstash.beta.wmflabs.org:80 127.0.0.1 - - [12/Mar/2015:09:14:24 +0000] "\x16\x03\x01" 301 308 "-" "-"

Some python script is hitting it improperly. Maybe the ganglia monitor though it does not refers to https.

yuvipanda removed yuvipanda as the assignee of this task.Mar 23 2015, 8:00 PM
greg moved this task from Done to Backlog on the Beta-Cluster-Infrastructure board.Jun 26 2015, 4:58 PM
Krenair subscribed.Apr 11 2016, 9:00 PM
root@deployment-logstash2:/# grep "server-status" etc/* -r
etc/apache2/conf-available/50-server-status.conf:# Only serve /server-status on loopback interface to local requests.
etc/apache2/conf-available/50-server-status.conf:# The default mod_status configuration enables /server-status on all
etc/apache2/conf-available/50-server-status.conf:# a more conservative configuration that makes /server-status accessible
etc/apache2/conf-available/50-server-status.conf:      <Location /server-status>
etc/apache2/conf-available/50-server-status.conf:        SetHandler server-status
etc/ganglia/conf.d/apache_status.pyconf:        value = "http://127.0.0.1:80/server-status"

I tried changing that last file to add ?test to the end of the URL (and then did service ganglia-monitor restart), and now that shows in the log. So it's definitely the ganglia monitor.

Krenair added a comment.Apr 11 2016, 9:03 PM

(and then ran puppet to put it back how it was)
But what do we want to do - disable the ganglia monitor, get rid of access logs more often, or...?

Krenair added a parent task: T71601: Log files on labs instance fill up disk (/var is only 2GB) (tracking).Apr 13 2016, 8:54 PM
hashar lowered the priority of this task from Medium to Low.Aug 22 2016, 8:37 AM
hashar mentioned this in T71601: Log files on labs instance fill up disk (/var is only 2GB) (tracking).
hashar closed this task as Resolved.Sep 26 2016, 10:12 AM
hashar claimed this task.

That no more appear. The main reason was the /var being too small which is no more the case today.

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:57 PM
Restricted Application added a project: Release-Engineering-Team (Kanban). · View Herald TranscriptJun 7 2017, 6:57 PM
• Phabricator_maintenance edited projects, added RelEng-Archive-FY201718-Q1; removed Release-Engineering-Team (Kanban).Sep 26 2017, 11:48 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL