Page MenuHomePhabricator

apache threads locked in luasandbox_timer_destroy
Closed, ResolvedPublic

Description

Every few days a group of Apache threads on a random box will lock up with the stack trace below. Need to find out what is dying in a semaphore-protected code block somewhere.

No obvious periodic cycle that I've noticed, and not confined to specific boxes. No real site impact either, but sometimes the threads have uncommitted DB transactions which is troublesome for locks and stuff.

Anyway, things shouldn't just freeze up :)

(gdb) thread apply all bt

Thread 2 (Thread 0x7f6d67a6b740 (LWP 25458)):
#0 0x00007f6d666392e3 in ?? () from /lib/x86_64-linux-gnu/librt.so.1
#1 0x00007f6d6b992e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007f6d6b6bfccd in personality () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f6d6c452740 (LWP 25047)):
#0 0x00007f6d6b998fd0 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f6d5f7ded90 in ?? () from /usr/lib/php5/20090626/luasandbox.so
#2 0x00007f6d5f7deff3 in luasandbox_timer_destroy () from /usr/lib/php5/20090626/luasandbox.so
#3 0x00007f6d5f7de12d in ?? () from /usr/lib/php5/20090626/luasandbox.so
#4 0x00007f6d6a03dbcf in zend_objects_store_del_ref_by_handle_ex () from /usr/lib/apache2/modules/libphp5.so
#5 0x00007f6d6a03dbf3 in zend_objects_store_del_ref () from /usr/lib/apache2/modules/libphp5.so
#6 0x00007f6d6a00a0f1 in _zval_ptr_dtor () from /usr/lib/apache2/modules/libphp5.so
#7 0x00007f6d6a03dbb7 in zend_objects_store_del_ref_by_handle_ex () from /usr/lib/apache2/modules/libphp5.so
#8 0x00007f6d6a03dbf3 in zend_objects_store_del_ref () from /usr/lib/apache2/modules/libphp5.so
#9 0x00007f6d6a00a0f1 in _zval_ptr_dtor () from /usr/lib/apache2/modules/libphp5.so
#10 0x00007f6d6a0263f0 in zend_hash_destroy () from /usr/lib/apache2/modules/libphp5.so
#11 0x00007f6d6a039bc9 in zend_object_std_dtor () from /usr/lib/apache2/modules/libphp5.so
#12 0x00007f6d6a039be9 in zend_objects_free_object_storage () from /usr/lib/apache2/modules/libphp5.so
#13 0x00007f6d6a03dbcf in zend_objects_store_del_ref_by_handle_ex () from /usr/lib/apache2/modules/libphp5.so
#14 0x00007f6d6a03dbf3 in zend_objects_store_del_ref () from /usr/lib/apache2/modules/libphp5.so
#15 0x00007f6d6a00a0f1 in _zval_ptr_dtor () from /usr/lib/apache2/modules/libphp5.so
#16 0x00007f6d6a0263f0 in zend_hash_destroy () from /usr/lib/apache2/modules/libphp5.so
#17 0x00007f6d6a017a9f in _zval_dtor_func () from /usr/lib/apache2/modules/libphp5.so
#18 0x00007f6d6a00a0f1 in _zval_ptr_dtor () from /usr/lib/apache2/modules/libphp5.so
#19 0x00007f6d6a03c948 in ?? () from /usr/lib/apache2/modules/libphp5.so
#20 0x00007f6d6a077907 in ?? () from /usr/lib/apache2/modules/libphp5.so
#21 0x00007f6d6a078206 in ?? () from /usr/lib/apache2/modules/libphp5.so
#22 0x00007f6d6a03e93b in execute () from /usr/lib/apache2/modules/libphp5.so
#23 0x00007f6d6a00be7c in zend_call_function () from /usr/lib/apache2/modules/libphp5.so
#24 0x00007f6d6a02efd7 in zend_call_method () from /usr/lib/apache2/modules/libphp5.so
#25 0x00007f6d6a039b02 in zend_objects_destroy_object () from /usr/lib/apache2/modules/libphp5.so
#26 0x00007f6d6a03d64d in zend_objects_store_call_destructors () from /usr/lib/apache2/modules/libphp5.so
#27 0x00007f6d6a00a4bd in ?? () from /usr/lib/apache2/modules/libphp5.so
#28 0x00007f6d6a018737 in ?? () from /usr/lib/apache2/modules/libphp5.so
#29 0x00007f6d69fc568d in php_request_shutdown () from /usr/lib/apache2/modules/libphp5.so
#30 0x00007f6d6a0a9247 in ?? () from /usr/lib/apache2/modules/libphp5.so
#31 0x00007f6d6c4a7508 in ap_run_handler ()
#32 0x00007f6d6c4a797e in ap_invoke_handler ()
#33 0x00007f6d6c4b7570 in ap_process_request ()
#34 0x00007f6d6c4b4398 in ?? ()
#35 0x00007f6d6c4adfa8 in ap_run_process_connection ()
#36 0x00007f6d6c4bc1d0 in ?? ()
#37 0x00007f6d6c4bc93a in ?? ()
#38 0x00007f6d6c4bd4e7 in ap_mpm_run ()
#39 0x00007f6d6c4924a4 in main ()


Version: unspecified
Severity: normal
OS: Linux

Details

Reference
bz62303

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 3:03 AM
bzimport set Reference to bz62303.
bzimport added a subscriber: Unknown Object (MLST).

If PHP did a longjmp() out of a signal handler, as in a request timeout, and wmerrors somehow failed to detect the situation, then this could happen. luasandbox_timer_destroy() could be interrupted by the signal and then re-entered during request shutdown. If this is what happened, there may possibly be a correlated syslog line when the deadlock began.

Krinkle set Security to None.
Krinkle removed a subscriber: Unknown Object (MLST).
Springle claimed this task.

I havn't seen this issue in a while, and we've since switched to HHVM anyway which presumably changes a bunch of variables anyway.