Page MenuHomePhabricator
Create Task
Maniphest T63102

Soften qdel behaviour from KILL
Closed, ResolvedPublic
Actions

Description

At the moment, qdel KILLs the job; this is a bit rude.

If jsub would call "qsub -notify", SGE would signal the job before KILLing it.

The signal is set by "execd_param"'s NOTIFY_KILL; default is SIGUSR1, I would favour SIGTERM (or a SIGHUP -> SIGINT -> SIGTERM cascade) as I suppose more programs will already have a suitable handler for that.

The queue parameter "notify" defines the interval between signals; given that many jobs in Tools use database and other network connections, I would be fairly generous here and propose 60 s (that means in the worst case of a SIGHUP -> SIGINT -> SIGTERM -> SIGKILL cascade 180 s which I find acceptable; for special cases, roots can always log into the exec node and kill at will).

Version: unspecified
Severity: enhancement

Details

Reference
bz61102

Event Timeline

• bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 2:53 AM
• bzimport added a project: Toolforge.
• bzimport set Reference to bz61102.
scfc created this task.Feb 9 2014, 7:34 AM
scfc added a comment.Apr 24 2014, 1:47 PM
  • Bug 63878 has been marked as a duplicate of this bug. ***
scfc added a comment.Apr 24 2014, 1:48 PM

We need to use "qsub -notify" in webservice as well.

• bzimport added a comment.Apr 26 2014, 2:57 PM

metatron wrote:

Concerning (non) termination of php-cgi processes:

http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_ModFastCGI

There is an option "kill-signal" in .lighttpd fcgi settings.

"kill-signal": By default lighttpd send SIGTERM to FastCGI processes, which were spawned by lighttpd. Applications, which link libfcgi, need to be killed with SIGUSR1. This applies to php <5.2.1, lua-magnet and others.

I tried setting this value to 9, also to 1. But in neither case, the signal was forwarded to the spawned cgi-processes, while killing with 9 and 1 by hand worked.

This (mis)behaviour seems to matter also in the case of overloaded and dying webservices, as overloaded threads/processes are /not/ terminated as they should be.

scfc added a comment.Apr 26 2014, 3:34 PM

The program flow is different at the moment: On qdel, SGE kills the master lighttpd process with SIGKILL. Thus, lighttpd never has a chance to kill the php-cgi processes. So kill-signal is irrelevant at the moment.

coren added a comment.Aug 27 2014, 11:12 PM

The grid has been adjusted to use SIGTERM by default now; this problem should be solved.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL