Page MenuHomePhabricator

Job queue not running (HTTP 411) due to missing Content-Length: header
Closed, ResolvedPublic

Description

I recently upgraded from 1.20.8 to 1.23.2 and it now appears the job queue is not running, though manual runs of the runJobs script do work. The wiki debug log has messages saying:

[runJobs] Running 1 job(s) via '/index.php?title=Special%3ARunJobs&tasks=jobs&maxjobs=1&sigexpiry=NNN&signature=NNN'
[runJobs] Failed to start cron API: received 'HTTP/1.1 411 Length Required'

It was suggested on the IRC channel that others are seeing this too and that I should file a bug report, so here it is.


Version: 1.23.2
Severity: major
OS: Linux

Details

Reference
bz72274

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:51 AM
bzimport set Reference to bz72274.

FWIW, my servers are running Apache 2.2.22 on Ubuntu 12.04.

Oh and PHP 5.3.10. Sorry for the comment spam.

Not sure if this is the reason but there have been job queue changes in 1.22, see https://git.wikimedia.org/blob/mediawiki%2Fcore.git/REL1_22/RELEASE-NOTES-1.22 and https://gerrit.wikimedia.org/r/#/c/59797/ (and https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 )

What is

$wgJobTypeConf

set to?

You can also set:

$wgDebugLogGroups['runJobs'] = "<some path>"

to get debug output and errors (but maybe that's already what your info in comment 0 comes from?).

$wgJobTypeConf is not set in LocalSettings.php so it's still the default:

array( 'default' => array ( 'class' => 'JobQueueDB', 'order' => 'random' ) )

I haven't used $wgDebugLogGroups, but the output I provided is in the debug log specified by $wgDebugLogFile.

I'll look more closely at the links you've provided, though as I understand it, MW change how the job queue works in 1.22 and then reverted those changes to some degree in 1.23, so I'm not sure how applicable the 1.22 changes could be to a 1.23 wiki.

The problem is that the internal HTTP request to run jobs doesn't include the Content-Length: header (as the HTTP 411 error explains), but it seems to be required[1] (although some webservers, even my apache 2.2.22 on OpenSuSE, doesn't complain about that missing header!):

Relevant code in MediaWiki.php (or Wiki.php before 1.24)

$req = "POST $url HTTP/1.1\r\nHost: {$info['host']}\r\nConnection: Close\r\n\r\n";
...
$bytes = fwrite( $sock, $req );

The solution is simple: Send a Content-Length: header.


[1] http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.13

Do I see it correctly that a Content-Length: 0 should be correct there since the request body is empty?

gerritadmin wrote:

Change 167707 had a related patch set uploaded by Poke:
Add Content-Length header for job queue requests

https://gerrit.wikimedia.org/r/167707

gerritadmin wrote:

Change 167707 merged by jenkins-bot:
Add Content-Length header for job queue requests

https://gerrit.wikimedia.org/r/167707

Fix landed and was confirmed to have solved the issue.

Wondering whether that (merged in master) one-liner patch should be backported to 1.23 tarballs - setting backport flag to "?" and CC'ing Mark y Markus.

gerritadmin wrote:

Change 169734 had a related patch set uploaded by MarkAHershberger:
AuthorDate: 2014-10-20 23:40:20 +0200

https://gerrit.wikimedia.org/r/169734

gerritadmin wrote:

Change 169734 merged by Mglaser:
Add Content-Length header for job queue requests

https://gerrit.wikimedia.org/r/169734

Now that this has been integrated into REL1_23, should I go ahead and backport it to REL1_24 too?

(In reply to Patrick Westerhoff from comment #13)

Now that this has been integrated into REL1_23, should I go ahead and
backport it to REL1_24 too?

Please do.

gerritadmin wrote:

Change 169965 had a related patch set uploaded by Poke:
Add Content-Length header for job queue requests

https://gerrit.wikimedia.org/r/169965

gerritadmin wrote:

Change 169965 merged by jenkins-bot:
Add Content-Length header for job queue requests

https://gerrit.wikimedia.org/r/169965

Backported to 1.23 and 1.24.