| hashar | |
| Aug 1 2013, 9:30 AM |
| F11922: file_52378.txt | |
| Nov 22 2014, 2:11 AM |
| F11921: memcached-usage-month-memc0.png | |
| Nov 22 2014, 2:11 AM |
deployment-memc0 and deployment-memc1 have 16GB of memory. They are being applied the role::memcached class which boot a single memcached instance having 90MB of memory. We end up with only 180MB of memcache space.
ps -u nobody f
PID TTY STAT TIME COMMAND 995 ? SLl 0:08 /usr/bin/memcached -m 89088 -p 11211 -u nobody -l 0.0.0.0 -c 25000 -k -n 5 -f 1.05 -D : -o slab_reassign
Version: unspecified
Severity: normal
Created attachment 13199
ganglia graph for memc0 showing 1 month of memcached memory usage
Apparently memcached grows up by itself as needed:
From top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1058 nobody 20 0 1397m 1.2g 2424 S 0 7.4 48:38.38 memcached
Though according to the ganglia graph, it is limited at 1GB of memory.
Attached:
Created attachment 13200
stats output for memc0
Maximum cache size:
STAT limit_maxbytes 93415538688 ~ 93G
Currently used cache:
STAT bytes 1019756272 ~ 1G
Apparently it is not memory limited.
STAT evictions 0
Stats documentation is in the protocol definition: https://github.com/memcached/memcached/blob/master/doc/protocol.txt
So apparently memcached memory usages grows up and maybe we dont need more memory ?
Attached:
So the real issue is that memcached is passed -m 89088, which would let the instances allocate roughly 89GB of memory when they only have 16GB. Should limit to 15GB.
Change 81905 had a related patch set uploaded by Hashar:
tweak memcached limit on beta (89GB -> 15GB)
Change 81905 merged by Faidon Liambotis:
tweak memcached limit on beta (89GB -> 15GB)
I have restarted memcached on deployment-memc{0,1}
ps shows:
/usr/bin/memcached -m 15000 ..
Fixed!