Page MenuHomePhabricator

[SF] SECURITY! Infinite recursive RunQuery call to SMWSQLStore2 takes down MediaWiki until PHP times out or consumes maximum memory.
Closed, InvalidPublic

Description

For both MediaWiki 1.17 and 1.18, and for both SMW 1.6 and 1.6.1:

When set in LocalSettings.php:

error_reporting( E_ALL );
ini_set( 'display_errors', 1 );

This error is reported:

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 4081043 bytes) in /home/coin/public_html/w/includes/parser/Parser.php on line 353

Notice that the memory available is 512MB, set in LocalSettings.php with:

ini_set( 'memory_limit', '512M' );

The error first appeared after I had accidentally saved an SMW-heavy template page to an article page. After making the error, I tried to edit and save the correct page text, and the save would not complete. It displayed the memory error after waiting for the script to time out.

The SMW installation has been working fine for months with mostly just me using it. Now, the only pages that are accessible that I know of are these pages:

http://www.coincompendium.com/w/index.php/Special:BlankPage
http://www.coincompendium.com/w/index.php?title=Main_Page&action=history
http://www.coincompendium.com/w/index.php?title=Special:UserLogin

I upgraded to MW 1.18 and SMW 1.6.1, but the problem persists. My web host says this:

Apache is spending a lot of time on this plugin:
SMWSQLStore2

The CPU also drives up doing a ton of SELECT (almost like an infinite loop) statements. How big is this database it is querying?

It seems to be stuck in this loop forever, eating up all the memory. It seems this plugin is broken or this plugin's database is not configured properly?

I say this about the database size:

Uncompressed, the SQL backup file is 129 234 879 bytes = 123.247985 megabytes, which is for something like 1000 to 2000 pages on the site. That's not a lot at all.

SMW_refreshData.php produces this error output:

Refreshing all semantic data in the database!

Some versions of PHP suffer from memory leaks in long-running scripts.
If your machine gets very slow after many pages (typically more than

  1. were refreshed, please abort with CTRL-C and resume this script at the last processed page id using the parameter -s (use -v to display page ids during refresh). Continue this until all pages were refreshed. ---

Processing all IDs from 1 to last ID ...
PHP Fatal error: Out of memory (allocated 1616379904) (tried to allocate 132515949 bytes) in /home/coin/public_html/w/includes/parser/Parser.php on line 353

Fatal error: Out of memory (allocated 1616379904) (tried to allocate 132515949 bytes) in /home/user/public_html/w/includes/parser/Parser.php on line 353
user@nail:~/public_html/w/extensions/SemanticMediaWiki/maintenance$

SMW_setup.php produces this output:

Setting up standard database configuration for SMW ...

Selected storage engine is "SMWSQLStore2" (or an extension thereof)

Checking table wiki_smw_ids ...

Table already exists, checking structure ...
... field smw_id is fine.
... field smw_namespace is fine.
... field smw_title is fine.
... changing type of field smw_iw from 'VARBINARY(32) NOT NULL' to 'VARCHAR(32) binary NOT NULL' ... done.
... changing type of field smw_subobject from 'VARBINARY(32) NOT NULL' to 'VARCHAR(32) binary NOT NULL' ... done.
... field smw_sortkey is fine.
... done.

Checking table wiki_smw_conccache ...

Table already exists, checking structure ...
... field s_id is fine.
... field o_id is fine.
... done.

Checking table wiki_smw_rels2 ...

Table already exists, checking structure ...
... field s_id is fine.
... field p_id is fine.
... field o_id is fine.
... done.

Checking table wiki_smw_atts2 ...

Table already exists, checking structure ...
... field s_id is fine.
... field p_id is fine.
... field value_xsd is fine.
... field value_num is fine.
... done.

Checking table wiki_smw_text2 ...

Table already exists, checking structure ...
... field s_id is fine.
... field p_id is fine.
... field value_blob is fine.
... done.

Checking table wiki_smw_spec2 ...

Table already exists, checking structure ...
... field s_id is fine.
... field p_id is fine.
... field value_string is fine.
... done.

Checking table wiki_smw_subs2 ...

Table already exists, checking structure ...
... field s_id is fine.
... field o_id is fine.
... done.

Checking table wiki_smw_subp2 ...

Table already exists, checking structure ...
... field s_id is fine.
... field o_id is fine.
... done.

Checking table wiki_smw_inst2 ...

Table already exists, checking structure ...
... field s_id is fine.
... field o_id is fine.
... done.

Checking table wiki_smw_redi2 ...

Table already exists, checking structure ...
... field s_title is fine.
... field s_namespace is fine.
... field o_id is fine.
... done.

Checking table wiki_smw_conc2 ...

Table already exists, checking structure ...
... changing type of field s_id from 'INT(8) UNSIGNED NOT NULL KEY' to 'INT(8) UNSIGNED NOT NULL' ... done.
... field concept_txt is fine.
... field concept_docu is fine.
... field concept_features is fine.
... field concept_size is fine.
... field concept_depth is fine.
... field cache_date is fine.
... field cache_count is fine.
... done.

Checking table wiki_sm_coords ...

Table already exists, checking structure ...
... field s_id is fine.
... field p_id is fine.
... field lat is fine.
... field lon is fine.
... done.

Database initialised successfully.

Setting up internal property indices ...

... space for internal properties already allocated.
... writing entries for internal properties ... done.

Internal properties initialised successfully.

Done.


Version: unspecified
Severity: critical

Details

Reference
bz33203

Event Timeline

bzimport raised the priority of this task from to Unbreak Now!.Nov 22 2014, 12:08 AM
bzimport set Reference to bz33203.

Forgot to mark this as critical, since it shuts down the wiki.

I tried running this SMW instance on another system, and I got the same errors, so I'm sure it's not a configuration error at any of my hosts. Trying out after upgrading to SMW 1.7 RC 1 produced similar errors, with more clues in the error messages. The first error message was this:

Fatal error: Call to undefined method SRFGallery::readParameters() in /home/a914/public_html/w/extensions/SemanticResultFormats/Gallery/SRF_Gallery.php on line 20

So, I disabled Semantic Result Formats, and tried again to get this:

MediaWiki internal error.

Exception caught inside exception handler.

Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information.

So, following the instructions in that error message, wgShowExceptionDetails produced this lengthy bit of error info, which once again seems to point to a problem with SMW:

MediaWiki internal error.

Exception caught inside exception handler: exception 'MWException' with message 'FauxRequest::getRequestURL() not implemented' in /home/a914/public_html/w/includes/WebRequest.php:1106
Stack trace:
#0 /home/a914/public_html/w/includes/WebRequest.php(1135): FauxRequest->notImplemented('FauxRequest::ge...')
#1 /home/a914/public_html/w/includes/SkinTemplate.php(1033): FauxRequest->getRequestURL()
#2 /home/a914/public_html/w/includes/SkinTemplate.php(498): SkinTemplate->buildContentNavigationUrls(Object(OutputPage))
#3 /home/a914/public_html/w/includes/OutputPage.php(1856): SkinTemplate->outputPage(Object(OutputPage))
#4 /home/a914/public_html/w/includes/Exception.php(183): OutputPage->output()
#5 /home/a914/public_html/w/includes/Exception.php(209): MWException->reportHTML()
#6 /home/a914/public_html/w/includes/Exception.php(392): MWException->report()
#7 /home/a914/public_html/w/includes/Exception.php(471): MWExceptionHandler::report(Object(DBQueryError))
#8 /home/a914/public_html/w/includes/Wiki.php(536): MWExceptionHandler::handle(Object(DBQueryError))
#9 /home/a914/public_html/w/index.php(57): MediaWiki->run()
#10 {main}

Any advice in debugging this further would be appreciated, especially if it will lead to SMW functioning. Right now, the wiki is down as long as SMW and related extensions are enabled.

This first error:

Error: 1054 Unknown column 'alt' in 'field list' (localhost)

seems to be referring to text in the database. So, I did a search through the text in my database, and there is no "alt" or "field list" that I could find.

No one has any suggestions how to solve this? Is SMW the sole cause of this, or is there some other underlying interaction that could possibly be worked around? None of my trial-and-error attempts have worked so far, other than disabling SMW, so any guidance on what these errors mean and how they might be worked around would be helpful.

I moved this from SMW to Semantic Forms. There were no responses in IRC, here in the bug report, nor in any of the mailing lists, but I finally got some help from Ryan Lane in debugging this, and he helped me narrow down the problem a little more until I could figure out what was happening.

Although the problem is manifesting itself in SMW's code, it's actually something SF is incorrectly calling on SMW to do on pages that have a query form, either embedded, or on the actual Special:RunQuery page.

This did not fail immediately. At first it functioned successfully many times before it finally failed, and then it continued to fail. I did not know what had changed to cause the problem, and debugging was made difficult with a non-functional wiki.

It turns out that someone had accidentally pasted the code for a page that uses an embedded RunQuery, into the RunQuery form. So, the RunQuery Semantic Form was calling itself to run SMW queries repeatedly in an infinite recursion. Since this problem took down the site, I was not able to find the error without digging through a lengthy debugging process, and getting a little advice from Ryan Lane on how to proceed further into new avenues once I had exhausted everything I already knew how to do.

Although it was a simple error that caused the bug to manifest itself, it turns out to be a significant security problem, where a malicious editor could use this bug to potentially shut down a mediawiki site that uses Semantic Forms. Generous php.ini settings for max_execution_time and memory_limit will allow long run times and lots of memory usage, respectively that will exacerbate the unresponsiveness of the server until PHP reaches an error condition, and stops execution. Once execution is stopped, it can be restarted again by the malicious attacker simply reloading the page.

RunQuery Semantic Forms needs to detect a recursive call to itself, and return a suitable error message instead of entering into infinite recursion.

The recursion can be caused either accidentally (in my case), or maliciously, which either way will then effectively shut down the server, and make it difficult to find the problem and fix it before the attacker reloads the page again. Even if you knew exactly where the problem was, its possible this attack could make it impossible to undo the changes that caused it until the attacking page reloads stop - unless you want to edit the database manually to undo the changes.

Could you reproduce the problem on a public wiki?

It looks like you can still see it in one of my test wikis with the login Demo/test:

http://www.coincompendium.net/w/index.php?title=Form:Specimen_find&oldid=14387

When I tried to reproduce it again, template expansion depth limits stopped the problem from reoccurring, so it looks as though I didn't have the bug pinned down as well as I had thought, although the fix did work. Maybe you will be able to come up with more detail on what exactly is causing the bug.

Perhaps it's time to close this bug down? You seem to have used this one bug report to report on two or three different errors, which is confusing. I would close this bug report and, if you encounter any of these problems again, create a new one for it.

You mean I've triggered more than one bug in Semantic Forms via RunQuery? Simply removing the recursive call to a RunQuery form seems to have worked around all the bugs, so if there's more bugs, having only one trigger (that I know of) makes it one bug report. Unfortunately, I have very little understanding of some of the error messages, so I can't divine more from them.

If you have an idea of what's going on, please share it and maybe I can debug further for you. I'd hate to abandon this bug since it was serious enough to shut down my wiki for 2 weeks, and so hard to figure out what happened, especially when the manner of triggering the bug makes it possible for it to be used maliciously.

I can give you administrator priveleges on my test wiki and and SSH login credentials for my test server if that will help.

Well, the error message you got earlier, "Unknown column 'alt' in 'field list'", is definitely unrelated to Semantic Forms.

I would close this down and start over again - it's hard to understand what you're talking about here.

Let's keep it open. I was able to isolate this bug, which I think is very similar:

https://bugzilla.wikimedia.org/show_bug.cgi?id=33424

So, I think I'll be able to get this one eventually too. Maybe the fix for that bug will affect this one? I'm kind of burnt out on bug testing, so I'll have to come back and try it a bit later, to see if the same or similar thing triggers this bug too.

Time to close it? I still don't know what this bug is about.

I think I've found a third related bug, but since I've only isolated 1 of the 3 bugs, I can mark this as invalid to get it off your list until I can divine more details about it. I'll reopen it, if needed, when I've got more information. I suspect the 3 bugs may actually be just one bug being triggered by different circumstances, since the error messages are so similar.