Page MenuHomePhabricator

Logging filling up disk on oxygen
Closed, ResolvedPublic

Description


Version: unspecified
Severity: normal
Whiteboard: u=Community c=General/Unknown p=0 s=2014-08-07
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=69663

Details

Reference
bz69661

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:37 AM
bzimport set Reference to bz69661.
bzimport added a subscriber: Unknown Object (MLST).

Change 154457 had a related patch set uploaded by QChris:
Disable udp2log forward to University of Minnesota

https://gerrit.wikimedia.org/r/154457

Created attachment 16218
Maximum-disk-space-used-oxygen-2014-08-07

Attached:

Maximum-disk-space-used-oxygen-2014-08-07.png (188×497 px, 12 KB)

Since noon 2014-08-16, the disk space on root of oxygen started to
fill up (see attachment Maximum-disk-space-used-oxygen-2014-08-07):

qchris@oxygen:~$ df -h
Filesystem           Size  Used Avail Use% Mounted on
/dev/md126           111G  105G     0 100% /
[...]

It seems udp2log's log is responsible:

qchris@oxygen:~$ ls -lh /var/log/udp2log/udp2log.log
-rw-r--r-- 1 udp2log udp2log 101G Aug 17 11:57 /var/log/udp2log/udp2log.log

. This file contains many lines like

/bin/grep: writing output: Broken pipe

. Since bento.cs.umn.edu (udp2log target for Univesity of Minnesota
stream) currently does not resolve, it seems this filter might be
responsible. The change from comment #1 tries to turn this filter off
for now.

Change 154457 merged by Ottomata:
Disable udp2log forward to University of Minnesota

https://gerrit.wikimedia.org/r/154457

ottomata also cleaned up the udp2log log file on oxygen, and oxygen is all
green again in icinga.
Thanks ottomata!