Produce/preserve the metadata about additional ResourceLoader modules required by extension tags
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	matmarex
	Jul 4 2014, 4:31 PM

Description

Parsoid must produce/preserve the metadata about additional ResourceLoader modules required by extension tags. (I've heard some gossip that it falls back to PHP parser to do this, so this should be easy to do.)

For example, in PHP parser, "<syntaxhighlight lang=php>$foo</syntaxhighlight>" will add up to two modules to ParserOutput's $mModuleStyles:

'ext.geshi.language.php' (there is such a module for every supported language)
'ext.geshi.local' (if $wgUseSiteCss is true)

This information must somehow be provided by the API used by VisualEditor.

The minimum set of things that have to be provided is the ones provided by OutputPage::addParserOutputContent() in PHP, that is modules, module styles, module scripts, module messages and JS config vars. (These are available via the action=parse&prop=modules API in core.)

Version: unspecified
Severity: major

Details

Reference: bz67540

	Subject	Repo	Branch	Lines +/-
	Include RL style modules from parser functions in <head>	mediawiki/services/parsoid	master	+23 -20
	Expose RL modules and js config vars in action=expandtemplates	mediawiki/core	master	+205 -62

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Open	Release	None	T84936 Release VisualEditor-MediaWiki as "1.0"
Open		None	T50429 [Epic] Support editing parts of a page in VisualEditor-MediaWiki
Open		None	T54365 Explore performance gains from progressive (JIT?) de-alienation in VisualEditor
Open		None	T174303 Copy-pasting linked ISBN numbers from view mode HTML into VisualEditor inserts wikitext links to Special:BookSources (it should turn them into magic links?)
Open	Feature	None	T54091 The read HTML should have hinting to allow full DOM copying (as opposed to just rich copying) from read mode into VE surfaces
Open		None	T55784 [EPIC] Use Parsoid HTML for all page views
Resolved		Tchanders	T118616 [Objective] Improve support for math editing by improving the UX of the Math extension and researching its usability
Resolved		Tchanders	T118618 Use syntax highlighting so that users can understand the LaTeX they type more easily
Resolved		Esanders	T49742 Provide basic source code editor features in VisualEditor SyntaxHighlight plugin
Declined		None	T67243 Flow: Obey the Exlinks gadget (open external links in new tab/window)
Resolved		Catrope	T96855 A15. Flow rendering doesn't render external links correctly
Resolved		Sbailey	T58756 Parsoid doesn't give external links class="external free\|text"
Resolved		• GWicke	T55783 Use CSS-based citation numbering
Resolved		None	T62017 Flow: Reference numbers in Parsoid output aren't superscripted
Resolved		ssastry	T64511 Mass rendering tests / visual diff
Resolved		Arlolra	T53245 Link MediaWiki styles and create Parsoid-specific CSS styling to match MediaWiki's for differing DOM elements
Resolved		Esanders	T69515 VisualEditor: Changing language of a <syntaxhighlight /> node doesn't load styles for new language
Resolved		ssastry	T50812 Get ParserOutput options from expandtemplates or similar action, and encode them as properties
Resolved		• AlexMonk-WMF	T60388 VisualEditor: Pasting a syntaxhighlight block from another VE surface don't load syntaxhighlight's styling
Resolved		Esanders	T45126 Create a VisualEditor plugin tool to add/edit SyntaxHighlight blocks
Declined		None	T73660 Add ZIM format support to OCG
Declined		None	T113736 Include stylesheets and javascript in `mw-ocg-bundler`
Open		None	T105845 RFC: Page components / content widgets
Resolved		Arlolra	T69540 Produce/preserve the metadata about additional ResourceLoader modules required by extension tags
Resolved		• marcoil	T73490 Parsoid should set the prop parameter when calling API action=expandtemplates
Resolved		• marcoil	T86902 Improve Parsoid's loading of CSS modules using ResourceLoader

Event Timeline

• bzimport raised the priority of this task from to High.Nov 22 2014, 3:35 AM

• bzimport added a project: Parsoid.

• bzimport set Reference to bz67540.

matmarex created this task.Jul 4 2014, 4:31 PM

Ex: http://en.wikipedia.org/w/api.php?action=parse&format=json&page=MathML&prop=modules

includes

"modulestyles":["ext.rtlcite","ext.math.styles","ext.geshi.language.xml","ext.geshi.local","ext.geshi.language.html5"]

CSS modules for inclusion in the <head> would be a good start and will help reduce rendering diffs.

Change 160423 had a related patch set uploaded by Marcoil:
WIP: Bug 67540 - Load extension CSS modules

https://gerrit.wikimedia.org/r/160423

There's a WIP patch now at https://gerrit.wikimedia.org/r/160423 that adds the extensions' CSS modules to the style link.

If we also want to load further extension resources, like JS or messages, I suppose we'll have to add loading ResourceLoader's client first and let it load everything else. Is this the way we should go?

I think we'll want the lowlevel information about the modules available (via data-mw?), in addition to having One True Resource Loader URL which will load them all.

In particular two of OCG backends, the ZIM writer and the future parsoid-phantomjs-PDF backend, will want to download the set of resources used by a (potentially large) set of pages. We don't want to have to download the same resources in slight variations over and over again. If there are N different resources used on a wiki, there are 2^N different resource loader URLs which load various combinations of the N resources. We'd rather just load the N resources once, even if that means we have to do a little extra work ourselves to order and merge them.

(12:15:15 PM) gwicke: cscott: we should expose the list separately, but for browsers it's probably more efficient to still do some bundling
(12:15:29 PM) gwicke: good to unbundle the per-page modules from the global ones though
(12:15:35 PM) gwicke: right now those are all mixed afaik
(12:18:03 PM) cscott-free: gwicke: agreed. i'm thinking of exposing the list via data-mw (for example) while still providing the resource url in a <link> tag for browsers.

As discussed on IRC, we'll do a first patch only for CSS support, and then expand on it with the rest of extensions' resources. We can continue using this bug for this and for C. Scott's proposal of exposing the modules in metadata.

Btw, does anyone have examples of pages where an extension CSS causes Parsoid to render differently from PHP? I'd like to test the visual diffs with this patch. Thanks!

Change 160423 merged by jenkins-bot:
Bug 67540 - Load extension CSS modules

https://gerrit.wikimedia.org/r/160423

The merged patch takes care of adding the appropriate CSS modules to the load list, but only for extensions. Some parser functions (e.g. Babel) also have CSS modules, there's a patch for MW that would allow Parsoid to know about them here:
https://gerrit.wikimedia.org/r/162123

• marcoil mentioned this in T86902: Improve Parsoid's loading of CSS modules using ResourceLoader.Jan 15 2015, 11:12 AM

• marcoil added a subtask: T86902: Improve Parsoid's loading of CSS modules using ResourceLoader.

• marcoil closed subtask T86902: Improve Parsoid's loading of CSS modules using ResourceLoader as Resolved.Feb 3 2015, 9:05 AM

ssastry lowered the priority of this task from High to Medium.Feb 3 2015, 5:59 PM

ssastry moved this task from Needs Triage to In Progress on the Parsoid board.

Quiddity mentioned this in T59159: Flow: syntax highlighting tags (GeSHi) don't get styled.Mar 21 2015, 12:08 AM

BTW, i'm trying to move TMH from it's global context into a more isolated one, but i'm guessing that due to the state of this ticket, that will also mean that when that patch lands, it will actually break TMH videos when added by VE right ? Since the global handlers would be gone and the 'local' ones are not loaded yet ?

https://gerrit.wikimedia.org/r/172556
https://gerrit.wikimedia.org/r/172421

A little status update:

What currently works:

Parsoid correctly includes style modules from extensions present in the page that add them through ResourceLoader, so that when requesting the HTML directly from Parsoid it's presented correctly.

What doesn't:

Style modules added by parser functions, as the expandtemplates API doesn't expose them yet. A patch similar to @Jackmcbarn's https://gerrit.wikimedia.org/r/162123, but without including non-RL modules would solve this.
All non-style modules exposed through ResourceLoader (Javascript, global vars, messages…). Parsoid's HTML output doesn't load ResourceLoader client infrastructure, so it's difficult to correctly load those modules.

I'm not sure if it's even a good idea to load non-style modules in Parsoid's HTML <head>, tbh, given that it's not really used independently by anything right now. I like @cscott's proposal of exposing the information in data-mw so that clients can use them as they see fit.

So my proposed next steps would be:

Expose all modules in action=expandtemplates, excluding non-RL modules.
Add the style ones to Parsoid's <head> as we do now with extension ones.
Expose all modules (styles and all others) through data-mw.

That way, for example, when VE receives just the body of a page for editing, it would look into the data-mw and load the modules it sees there, which would fix T69515 and related. Right now VE can't get that data directly as it's in Parsoid's <head>.

ssastry added a project: Parsoid-Rendering.Jun 3 2015, 6:30 PM

ssastry set Security to None.

ssastry moved this task from In Progress to Needs Triage on the Parsoid board.

Change 215874 had a related patch set uploaded (by Marcoil):
Expose ResourceLoader modules in action=expandtemplates

https://gerrit.wikimedia.org/r/215874

gerritbot added a project: Patch-For-Review.Jun 4 2015, 10:32 AM

Change 215883 had a related patch set uploaded (by Marcoil):
Include RL style modules from parser functions in <head>

https://gerrit.wikimedia.org/r/215883

Ricordisamoa subscribed.Jun 4 2015, 4:43 PM

Change 215874 merged by jenkins-bot:
Expose RL modules and js config vars in action=expandtemplates

https://gerrit.wikimedia.org/r/215874

• marcoil mentioned this in rMW8929182a67ef: Expose RL modules and js config vars in action=expandtemplates.Jun 5 2015, 4:17 PM

https://gerrit.wikimedia.org/r/215874 (branch master): MW-1.26-release, WMF-deploy-2015-06-09_(1.26wmf9)

Change 215883 merged by jenkins-bot:
Include RL style modules from parser functions in <head>

https://gerrit.wikimedia.org/r/215883

• marcoil mentioned this in rGPAR7f9ad2472b4c: Include RL style modules from parser functions in <head>.Jun 15 2015, 6:52 PM

TheDJ awarded a token.Jun 16 2015, 10:35 AM

Legoktm removed a subscriber: • Forrestbot.Jun 29 2015, 5:50 PM

There were two requests at wikimania for the rest of the stuff included in this bug. Namely:

Link to JS necessary to render the page contents. (Especially for maps modules, etc)
Some sort of machine-readable enumeration of the modules required (in case you wanted to do manual loading of CSS/JS/etc).

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 21 2015, 10:07 PM

Aklapper removed • marcoil as the assignee of this task.Sep 22 2015, 3:38 PM

cscott mentioned this in T73660: Add ZIM format support to OCG.Sep 25 2015, 2:32 PM

cscott added a parent task: T73660: Add ZIM format support to OCG.

cscott added a parent task: T113736: Include stylesheets and javascript in `mw-ocg-bundler`.

Kelson subscribed.Sep 25 2015, 2:38 PM

Krinkle subscribed.Sep 27 2015, 6:32 PM

• GWicke added a parent task: T105845: RFC: Page components / content widgets.Oct 6 2015, 8:26 PM

Minervaxox mentioned this in T115907: Outreachy proposal for T73660: Add ZIM format support to OCG.Oct 19 2015, 6:27 PM

• Mattflaschen-WMF added a parent task: T59159: Flow: syntax highlighting tags (GeSHi) don't get styled.Oct 23 2015, 3:27 PM

Legoktm removed a project: MW-1.26-release.Oct 28 2015, 6:05 PM

Liuxinyu970226 awarded a token.Aug 24 2017, 2:37 PM