deployment-mediawiki02 labs instance only has 2GB of disk which highlight an hhvm issue. The hhvm upstart process emits log to /var/log/upstart/hhvm.log which contains a bunch of php notices. That is also send to syslog ( /var/log/syslog/ ) creating a snowball effect that completely fill up /var/.
The hhvm upstart job should be made to discard anything that is below the 'error' level and the notices be sent to a central syslog.
Version: unspecified
Severity: major
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hashar | T71590 rsync errors to beta cluster, inconsistent state after scap | |||
| Resolved | hashar | T71601 Log files on labs instance fill up disk (/var is only 2GB) (tracking) | |||
| Invalid | None | T71976 HHVM emits logs filling /var/log/upstart/hhvm.log and /var/log/syslog/ filling disk | |||
| Resolved | bd808 | T76119 Get HHVM logs into logstash |
changed importance to "major", this makes beta hard to use for real people and causes a lot of false test failures.
Bryan told me the log issue is apparently fixed in production. Since mediawiki02 does not run puppet, it is not taking in account the recent changes made in operations/puppet.git :-(
I think puppet is now passing on the hhvm instances. I am not sure where the log are written to though.
Reopening, the reason I filled this bug is hhvm emitting lot of message on stdout/stderr which are caught by upstart and fill both:
The small /var on labs instances exacerbate the disk consumption which I believe need to be constrained / fixed since that is surely going to hit us in production as well.
Will rephrase summary.
Only way to increase /var is to re-create instances. New instances have a re-sizable /var