Raising priority, this killed the beta apaches boxes. /var/log is in the / partition so a full disk cause a lot of issues.

Raising priority again. This has again killed several beta boxes over the last two weeks.

There were several broken links within /data/project/apache which seemed to be making gluster lose its mind. Obviously that's a gluster bug, but I don't have much insight into why it couldn't cope.

I removed the broken files that gluster was complaining about, and replaced them via 'git reset --hard'. This appears to have quelled gluster's fears, and I'm pretty sure the actual files are still the way I found them.

I have a test box set up doing a rotation test with gluster log files. Presuming that test goes well I'll commit that change in a couple of days.

https://gerrit.wikimedia.org/r/#/c/42796/

Seems to rotate fine now :-] Thanks Andrew!

I think they are rotating improperly on the servers now, though. The log files show 0 as their file size. I'd imagine it's still writing to the old inodes there.

Logs on the server should be fixed by

https://gerrit.wikimedia.org/r/#/c/43962/1/files/logrotate/glusterlogs

Looks like like the logrotate does not work anymore:

root@deployment-bastion:/var/log/glusterfs# ll -rt
total 213956
-rw------- 1 root root 0 Feb 19 06:27 home.log
-rw------- 1 root root 0 Feb 19 06:27 data-project.log
-rw------- 1 root root 109185374 Mar 1 18:12 data-project.log.1
-rw------- 1 root root 109894887 Mar 1 18:17 home.log.1

I noticed the instance has two logrorate configuration files which are most probably conflicting:

/etc/logrotate.d/glusterfs-common

/var/log/glusterfs/*.log {
daily
rotate 7
delaycompress
compress
notifempty
missingok
}

And the puppet provided one:

cat glusterlogs

2. THIS FILE IS MANAGED BY PUPPET
3. puppet:///files/logrotate/glusterlogs #################

Rotate client logs

/var/log/glusterfs/*.log {
missingok
rotate 3
weekly
compress
postrotate

		/usr/bin/killall -HUP glusterfs > /dev/null 2>&1 || true
		/usr/bin/killall -HUP glusterd > /dev/null 2>&1 || true

endscript
}

Rotate server brick logs

/var/log/glusterfs/bricks/*.log {
missingok
rotate 3
weekly
compress
postrotate

		/usr/bin/killall -HUP glusterfsd > /dev/null 2>&1 || true

endscript
}

The glusterfs-common package does provide /etc/logrotate.d/glustefs-common and it is missing the HUP signaling :(

I have emailed Andrew to find out the status of this bug.

Note to self: I removed the gluster-installed log rotate on testlabs-abogott-dev; now waiting a few days to see if it shapes up

Yep, removing gluster's file helps. So...

https://gerrit.wikimedia.org/r/#/c/57426/

Seems to works now, and the rotated files get compressed. Thank you Andrew!