I was looking at disk utilization on integration-slave1006 today and noticed that there were a lot of /tmp/mw-ocg-latexer* directories. When I removed them just under 1G of disk space was freed.
These looked like they may be temporary files for tests that are leaked under some test failure scenario.
Version: unspecified
Severity: critical
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=72011
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hashar | T74011 Jenkins: Figure out long term solution for /tmp management | |||
| Resolved | cscott | T73341 /tmp/mw-ocg-latexer* directories left on Jenkins workers |
Change 163186 had a related patch set uploaded by Cscott:
Increase OCG warning/critical space thresholds.
That does not help Jenkins slaves. The /tmp on Jenkins slaves is filling up with 100s of abandoned mw-ocg-latexer**** directories like:
Oct 1 15:38 mw-ocg-latexer9950bagslz4/
Oct 1 15:38 mw-ocg-latexer9950bu0mq4g/
l mw-ocg-latexer87372ckx34w/
total 72K
drwx------ 4 jenkins-deploy wikidev 4.0K Oct 7 22:10 ./
drwxrwxrwt 319 root root 48K Oct 14 06:07 ../
drwxr-xr-x 3 jenkins-deploy wikidev 4.0K Oct 7 22:10 bundle/
-rw-r--r-- 1 jenkins-deploy wikidev 1.9K Oct 7 22:10 languages.tex
drwxr-xr-x 2 jenkins-deploy wikidev 4.0K Oct 7 22:10 latex/
-rw-r--r-- 1 jenkins-deploy wikidev 1.4K Oct 7 22:10 output.tex
This is one of three different software packages (the other being hhvm's perf-**.map and MW_PHPUnit_TextPassDumperTest files) that is filling up /tmp on Jenkins slaves and not being cleaned up.
Fix this soon (make it configurable, clean it up, disable it, whatever, if this needs modification to the Jenkins job, let me or Antoine know). If this is not addressed soon, we may have to undeploy tests for this project until it's sorted out.
Change 166677 had a related patch set uploaded by Arlolra:
Fix cleaning up while running the tests