Page MenuHomePhabricator

DPL2's include parameter broken with new parser
Closed, ResolvedPublic

Description

Author: skizzerz

Description:
The page linked to in the URL field contains all the info, mainly with the new parser not parsing the include parameter of DPL. While I am fully aware that it was an intended behavior with the new parser to not let templates/parser functions create other templates, a workaround or hook should be available for things that ''do'' need this functionality. Unfortunately, I don't know enough about how the parser works to write a hook that allows this, which is why I am bringing it up here.

Now you may be asking, "Why should we change the MediaWiki codebase just to support an extension that isn't even used -- and more than likely will never be used -- on Wikimedia Foundation wikis?" The answer is simple: While the WMF may hate this extension due to the enormous load on the databases, there are numerous wikis that utilize this function (and its include parameter) to carry out tasks that would be tedious if not impossible to do by hand. Therefore a hook or something should exist in the parsing process (like ParserParseTemplates or something) that parses templates created by other templates or parserfunctions when called. Option two would be changing the parser to re-allow this sort of thing, but since this was an intentional change to begin with, I highly doubt that will happen. Option three would be to add a hook into the new parser that lets you parse inputted data through the old parser (since we seem to be keeping the old parser around anyway).


Version: unspecified
Severity: major
URL: http://semeb.com/dpldemo/index.php?title=Issue:Including_nested_templates_no_longer_works_with_MW_1.12%27s_new_parser_behaviour

Details

Reference
bz13677

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 21 2014, 10:02 PM
bzimport set Reference to bz13677.

peu wrote:

I consider the DPL extension very useful for wikis I know and use, but I like the new parser functionality of mediawiki too. Should we have to decide to use only one of them? That would be really bad.

roc.no1 wrote:

I agree. I think that DPL and Semantic Mediawiki are the most powerful extensions for Mediawiki. They are great works. They should be used on sites which have large size of data because they rearrange data easily. Especially, DPL is very easy to use. MW's developers should value it.

gero.scholz wrote:

DPL and SMW are two ways to enrich a mediawiki with SEMANTICS.
This is especially useful for medium-size wikis with an elaborated
understanding of the knowledge domain they focus on.
Often people designing and running such kind of wikis are extremely
competent. We should allow them to use the latest MW release
without loosing any functionality.
Therefore I strongly vote for a solution which will allow DPL
to continue the use of the current way of template treatment
(which allows recursive expansion) also in all future MW releases.

brunomarcuzzo wrote:

The dpl extension is fundamental, for one good expression of the wiki.
www.tracciati.org

admin wrote:

Hi,

I am the administrator of [http://www.emuwiki.com EmuWiki.com - The Emulation Encyclopedia]. I use DPL and semantic form in combination to allow users of the wiki to add pages that are automatically included in dynamic lists such as news list, video lists, etc... I believe that DPL and Semantics are the greatest mediawiki extensions and I would really appreciate to have a work around that would allow us to keep our MediaWiki updated without loosing the ability to use DPL.

EmuWikiAdmin

eric.larcher wrote:

Hi,

I'm using today DPL for my personal wiki and I project to use it on professionals wikis. Like Jean-Francois Gariepy (even if I don't know him - really), I think that DPL and Semantics are among the greatest mediawiki extensions and I would really appreciate to have a work around that
would allow us to keep our MediaWiki updated without loosing the ability to use
DPL.

Best regards

skizzerz wrote:

Ok, I'm going to have a look-through in the parser source to see if I can potentially do one or more of the following (I'll try to implement them in core, but if brion or someone has issues with it, I'll at least post a few patches on this bug):

a) add in a function that allows a chunk of wiki text to be processed using the old parser
b) add in compatibility to allow templates to create other templates and view the parameters of other templates
c) other suggestions?

Hopefully I can come up with something stable in a few days, and as such I'm reassigning this bug to myself.

gero.scholz wrote:

This sounds great! Once you have come up with a version that you consider
reasonably stable I could install it on the dpldemo website. There we could go
through all kind of test cases.

I understand that your solution would ..

a) be a side entry for extensions which want to make use of that functionality
b) become a permanent documented official part of future MW releases
c) allow DPL to support its current functionality so that DPL users need not

change their code

d) require some adaptations inside the current DPL code which I would

hopefully be able to make

Am I right?

(In reply to comment #8)

a) add in a function that allows a chunk of wiki text to be processed using the
old parser

I don't like that idea. The point of a new parser is to phase out use of an old one. The code for that one is only going to degrade and become stale.

Additionally, the old parser is very inefficient. Remember that trying to parse a small bit of text with the old parser, will cause anything else inside that, in other words, anything which is included to be parsed with the old parser. There are multiple potential issues with this.

  • You remove the point of the new parser in a section of the page. In other words, anything included by DPL will use inefficient methods of parsing things and drop back to being a server strain.
  • A hook is not used to register Parser Functions and Tags. Because of this, these are only registered for the active parser. If you try to use the old parser for including things, then all included pages will lose all the parserfunctions and tags which are added by extensions. So suddenly anything included by DPL will loose the conditional things done by #if inside of templates.
  • Additionally, the new parser and old parser have some incompatibilities. There are some things which work in the old parser, but not the new one, and some which work in the new but not the old. Part of migrating to the new parser for a wiki is changing things to be compatible with the new parser, which may make them incompatible with the old one. As a result things may render differently depending on what parser is doing the rendering. So as a result things included by DPL may end up looking incorrectly because they do not render the same as they do on their own page.

Rather, the new parser should be improved to support inclusions if possible.

However, I do hear that this is more of a problem with parser tags, not parser functions. And may well be impossible.
Perhaps it may actually be a better idea to improve the workings of #dpl: to cover the cases where one would want to use <dpl/> and instead phase out the use of <dpl/> in favor of #dpl: as Semantic MediaWiki has.

rezyk wrote:

I don't know all of the details/history, but from looking at the code this issue seems more like an incidental effect/change rather than one tied directly to the intended design of the new parser. Even the old parser has code that would cause the same problem, but it is commented out (the line "//$noparse = true;" in Parser_OldPP.php). As long as $noparse was left alone, every parser function result would get its own expansion pass (through a call to replaceVariables), though this might only have been useful for extensions like DPL that might build template calls.

The new parser still makes the necessary equivalent expansion pass on the text from templates, transwiki inclusions, etc, but now always skips it for parser function results (ignoring $noparse altogether). I would think that the control for parser function results to specify whether or not there should be an expansion pass applied should just be put back into Parser::braceSubstitution.

gero.scholz wrote:

Comments #10 and #11 explain that it might make a big difference (from the parser´s point of view) WHERE the recursive parsing mechanism is needed. I think that DPL users could live with a solution which (in that respect) only will support parser function syntax. The majority of the more ambitious applications of DPL most probably use parser function syntax anyway (and not parser tag syntax) because they need parameter expansion - which (currently) only works within parser function calls.

I think DPL users want to (and should) benefit from improvements of the new parser as much as possible. And irritations for the user caused by using two essentially different parsers under the hood are really not desirable. So, if it helps to focus the solution on parser function syntax this should be the preferred way.

rezyk wrote:

adds $noparse and $preprocessFlags as parameters that parser function calls can return

I've attached a patch (tested on the latest 1.13alpha development version) that adds $noparse (default true) and $preprocessFlags (default 0) as parameters that parser function calls can return. With this, an extension like DPL can work correctly by instead returning something like this from its parser function call (ExtDynamicPageList2::dplParserFunction):

return array($dplresult, "noparse" => false);

This also works with the old parser. Parser extension tags are not affected, but they should work fine already. Note that $noparse was present in versions prior to 1.12.0 with a default value of false.

Attached:

rezyk wrote:

Also, to work on everything including 1.12.0 without that patch, an extension could manage the same effect by setting things up like so (when returning from its parser function call):

global $wgVersion;
if (version_compare($wgVersion, '1.12.0')==0 && $parser instanceof Parser) return array($parser->getPreprocessor()->preprocessToObj($dplresult), "isChildObj" => true);
return array($dplresult, "noparse" => false);

I should probably point out that there is a better way to detect the new parser.

If you take a look at ParserFunctions:
http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/ParserFunctions/ParserFunctions.php?revision=HEAD&view=markup

The new parser has a kind of frame/object handling. This test returns true when the parser has support for the new stuff. In other words, returns true when we are using the new parser.
defined( get_class( $parser ) . '::SFH_OBJECT_ARGS' )

It's much cleaner and probably more to the point for testing if the new parser is being used as the existence of the DOM preprocessor is the whole issue here as I understand.

skizzerz wrote:

Implemented patch in r34594

closing as FIXED