Page MenuHomePhabricator

ResourceLoader: Implement support for Source Maps
Closed, ResolvedPublic

Description

Source maps are a technique for mapping combined and minified JavaScript back to the original files. This could be very useful for debugging ResourceLoader in production mode. This can be necessary since behavior is different from debug mode (in ways other than minification).

This is supported in the Chrome, Firefox 50 and IE 11 debuggers and Closure compiler, and some other stacks have code to generate the maps.

The spec is at https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/edit?hl=en_US&pli=1# and a good overview is at http://www.html5rocks.com/en/tutorials/developertools/sourcemaps/ .

The minified file points to the source map with a line like:

//# sourceMappingURL=/path/to/file.js.map

or a header like:

X-SourceMap: /path/to/file.js.map

If we use a dynamic URL, that should allow doing it in production. It would build the source maps on demand (just like the minification) for people that have them enabled (and are debugging), without slowing the site for anyone else.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/SourceMap

Details

Reference
bz45514
SubjectRepoBranchLines +/-
mediawiki/coremaster+3 -3
operations/mediawiki-configmaster+1 -2
mediawiki/coremaster+1 -1
mediawiki/coremaster+5 -3
mediawiki/coreREL1_40+486 -283
mediawiki/coremaster+23 -21
mediawiki/coremaster+36 -19
mediawiki/coremaster+36 -19
operations/mediawiki-configmaster+5 -0
mediawiki/coremaster+137 -8
mediawiki/coremaster+17 -19
mediawiki/coremaster+551 -185
mediawiki/coremaster+181 -124
mediawiki/coremaster+9 -2
mediawiki/coremaster+795 -318
mediawiki/libs/Minifymaster+458 -15
mediawiki/coremaster+98 -88
mediawiki/coremaster+92 -78
mediawiki/extensions/CodeMirrormaster+42 -58
mediawiki/extensions/Wikibasemaster+33 -49
mediawiki/extensions/WikibaseLexememaster+24 -43
mediawiki/extensions/MobileFrontendmaster+2 -2
mediawiki/skins/Vectormaster+1 -1
mediawiki/extensions/MobileFrontendmaster+63 -106
mediawiki/extensions/Citemaster+13 -45
mediawiki/coremaster+475 -353
mediawiki/extensions/Scoremaster+28 -46
mediawiki/extensions/SyntaxHighlight_GeSHimaster+10 -25
mediawiki/coremaster+32 -5
mediawiki/libs/Minifymaster+189 -0
mediawiki/libs/Minifymaster+1 K -13
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 916070 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/extensions/CodeMirror@master] Migrate ResourceLoaderCodeMirrorModule to a virtual file callback

https://gerrit.wikimedia.org/r/916070

Change 916892 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/extensions/MobileFrontend@master] Migrate ResourceLoaderFileModuleWithMFConfig to a virtual file callback

https://gerrit.wikimedia.org/r/916892

Change 916893 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/extensions/Score@master] Migrate ScoreVeResourceLoaderModule to a virtual file callback

https://gerrit.wikimedia.org/r/916893

Change 916930 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/extensions/SyntaxHighlight_GeSHi@master] Migrate ResourceLoaderSyntaxHighlightVisualEditorModule to a virtual file callback

https://gerrit.wikimedia.org/r/916930

Change 917421 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/extensions/WikibaseLexeme@master] Migrate TemplateModule to a virtual file callback

https://gerrit.wikimedia.org/r/917421

Change 917423 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/extensions/Wikibase@master] Migrate TemplateModule to a virtual file callback

https://gerrit.wikimedia.org/r/917423

Change 917424 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/core@master] ResourceLoader: Unconditionally enable source map links

https://gerrit.wikimedia.org/r/917424

Change 905155 merged by jenkins-bot:

[mediawiki/core@master] ResourceLoader: Allow FileModule scripts to be generated with a callback

https://gerrit.wikimedia.org/r/905155

Change 916930 merged by jenkins-bot:

[mediawiki/extensions/SyntaxHighlight_GeSHi@master] Migrate ResourceLoaderSyntaxHighlightVisualEditorModule to a virtual file callback

https://gerrit.wikimedia.org/r/916930

Change 916893 merged by jenkins-bot:

[mediawiki/extensions/Score@master] Migrate ScoreVeResourceLoaderModule to a virtual file callback

https://gerrit.wikimedia.org/r/916893

Change 916070 merged by jenkins-bot:

[mediawiki/extensions/CodeMirror@master] Migrate ResourceLoaderCodeMirrorModule to a virtual file callback

https://gerrit.wikimedia.org/r/916070

Change 916063 merged by jenkins-bot:

[mediawiki/extensions/Cite@master] Migrate CiteVisualEditorModule to a virtual file callback

https://gerrit.wikimedia.org/r/916063

Change 916892 merged by jenkins-bot:

[mediawiki/extensions/MobileFrontend@master] Migrate ResourceLoaderFileModuleWithMFConfig to a virtual file callback

https://gerrit.wikimedia.org/r/916892

Change 919473 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/skins/Vector@master] Update bundle size to make room for source map links

https://gerrit.wikimedia.org/r/919473

Change 919609 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/extensions/MobileFrontend@master] Update bundle size for source map links

https://gerrit.wikimedia.org/r/919609

Change 919609 merged by jenkins-bot:

[mediawiki/extensions/MobileFrontend@master] Update bundle size for source map links

https://gerrit.wikimedia.org/r/919609

Change 919473 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Update bundle size to make room for source map links

https://gerrit.wikimedia.org/r/919473

Change 917421 merged by jenkins-bot:

[mediawiki/extensions/WikibaseLexeme@master] Migrate TemplateModule to a virtual file callback

https://gerrit.wikimedia.org/r/917421

Change 917423 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Migrate TemplateModule to a virtual file callback

https://gerrit.wikimedia.org/r/917423

Change 940629 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/core@master] ResourceLoader: Factor out the loop body of makeModuleResponse()

https://gerrit.wikimedia.org/r/940629

Change 940629 merged by jenkins-bot:

[mediawiki/core@master] ResourceLoader: Factor out the loop body of makeModuleResponse()

https://gerrit.wikimedia.org/r/940629

Change 944359 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/core@master] ResourceLoader: deliver deprecation warnings as strings

https://gerrit.wikimedia.org/r/944359

Change 944359 merged by jenkins-bot:

[mediawiki/core@master] ResourceLoader: deliver deprecation warnings as strings

https://gerrit.wikimedia.org/r/944359

Change 946668 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/core@master] ResourceLoader: Have FileModule deliver additional path information

https://gerrit.wikimedia.org/r/946668

Currently ResourceLoader::filter() has a cache that is keyed by content hash.

The way source maps are built, minification is done progressively, so there is no content to hash until minification is complete.

Options:

  1. Build the content without minification, hash it, then on a cache miss, throw it away and build it again with minification.
  2. Wrap getOneModuleResponse() with a new getWithSetCallback(). Use the module version as part of the key, it's mostly a content hash anyway.
  3. Same as 2 but use md5( json_encode( $module->getModuleContent() ) ). The v1 patch was similar except that it combined the modules first, regressing T107377.

The ResourceLoader::filter() cache is global. Global caching of source maps might be possible, if relative URLs are used. Not sure if it's worthwhile.

Option 1 needs the least validation work since it's most similar to what we do already. Module::getModuleContent() is memoized so we're just concatenating the bits, it's not really a rebuild. I think I'll benchmark it.

We're not gathering timing data for minification. So to get a sense for how long minification takes, I called ResourceLoader::filter() on ext.codeEditor.ace.modes, which is the largest module in production that has minification enabled. It took ~120ms on mwmaint1002.

I recall Trevor talking about the need for a global minify cache, because there are a lot of small wikis. It's nice to have, but it doesn't seem to be critical from the perspective of user-visible latency or cluster CPU usage. I suppose it helps to reduce cache size.

Change 947488 had a related patch set uploaded (by Krinkle; author: Tim Starling):

[mediawiki/libs/Minify@master] Add IndexMap class

https://gerrit.wikimedia.org/r/947488

Change 947488 merged by jenkins-bot:

[mediawiki/libs/Minify@master] Add IndexMap class

https://gerrit.wikimedia.org/r/947488

Option 1 needs the least validation work since it's most similar to what we do already. Module::getModuleContent() is memoized so we're just concatenating the bits, it's not really a rebuild. I think I'll benchmark it.

I benchmarked this, and found that generating the unminified source string from the cached module content array, for ext.wikiEditor which I am assuming to be representative of a large module, takes ~60µs. Fast enough. So I'm sticking with option 1 for now.

I'm using domain-relative URLs for the source map links and I'm caching the source maps globally (i.e. wiki-independent) in APCu.

Change 947502 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/core@master] ResourceLoader: Implement JavaScript source map support

https://gerrit.wikimedia.org/r/947502

Change 905156 abandoned by Tim Starling:

[mediawiki/core@master] ResourceLoader: Implement JavaScript source map support

Reason:

Split into multiple commits and superseded by I086e275148fdcac89f67a2fa0466d0dc063a17af

https://gerrit.wikimedia.org/r/905156

Change 917424 abandoned by Tim Starling:

[mediawiki/core@master] ResourceLoader: Unconditionally enable source map links

Reason:

Superseded by I086e275148fdcac89f67a2fa0466d0dc063a17af

https://gerrit.wikimedia.org/r/917424

Change 946668 merged by jenkins-bot:

[mediawiki/core@master] ResourceLoader: Have FileModule deliver additional path information

https://gerrit.wikimedia.org/r/946668

Change 947502 merged by jenkins-bot:

[mediawiki/core@master] ResourceLoader: Implement JavaScript source map support

https://gerrit.wikimedia.org/r/947502

Change 954927 had a related patch set uploaded (by Reedy; author: Tim Starling):

[mediawiki/core@REL1_40] ResourceLoader: Allow FileModule scripts to be generated with a callback

https://gerrit.wikimedia.org/r/954927

Change 955000 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] ResourceLoader: adopt SourceMap header instead of inline comment

https://gerrit.wikimedia.org/r/955000

Change 955000 merged by jenkins-bot:

[mediawiki/core@master] ResourceLoader: adopt SourceMap header instead of inline comment

https://gerrit.wikimedia.org/r/955000

Change 772878 abandoned by Krinkle:

[mediawiki/core@master] [WIP] resourceloader: Implement experimental source map support

Reason:

https://gerrit.wikimedia.org/r/772878

Change 954878 had a related patch set uploaded (by Krinkle; author: Tim Starling):

[operations/mediawiki-config@master] Enable source maps on group0 wikis

https://gerrit.wikimedia.org/r/954878

Change 954878 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable source maps on group0 wikis

https://gerrit.wikimedia.org/r/954878

Mentioned in SAL (#wikimedia-operations) [2023-09-13T05:40:24Z] <tstarling@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Enable source maps on group0 wikis T47514 (duration: 07m 14s)

Change 957829 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] mediawiki.page.watch.ajax: Move into separate dir to improve source map

https://gerrit.wikimedia.org/r/957829

Change 957830 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] ResourceLoader: Set 'virtualFilePath' for startup.js

https://gerrit.wikimedia.org/r/957830

Change 957831 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] mediawiki.langauge: Move related modules and virtual files to separate dir

https://gerrit.wikimedia.org/r/957831

Change 957830 merged by jenkins-bot:

[mediawiki/core@master] ResourceLoader: Set 'virtualFilePath' for startup.js

https://gerrit.wikimedia.org/r/957830

Change 957829 merged by jenkins-bot:

[mediawiki/core@master] mediawiki.page.watch.ajax: Move into separate dir to improve source map

https://gerrit.wikimedia.org/r/957829

Change 959071 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] ResourceLoader: Set virtualFilePath for startup.js (take 2)

https://gerrit.wikimedia.org/r/959071

Change 959071 merged by jenkins-bot:

[mediawiki/core@master] ResourceLoader: Set virtualFilePath for startup.js (take 2)

https://gerrit.wikimedia.org/r/959071

Change 957831 merged by jenkins-bot:

[mediawiki/core@master] mediawiki.language: Move related modules and virtual files to separate dir

https://gerrit.wikimedia.org/r/957831

Change 954927 abandoned by Reedy:

[mediawiki/core@REL1_40] ResourceLoader: Allow FileModule scripts to be generated with a callback

Reason:

https://gerrit.wikimedia.org/r/954927

Change 960659 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] ResourceLoader: Enable source maps in DevelopmentSettings.php

https://gerrit.wikimedia.org/r/960659

Do you plan on supporting package files modules when in resource loader debug mode?

Do you plan on supporting package files modules when in resource loader debug mode?

Source maps are essentially a replacement for debug mode. They work transparently and directly in production mode, for all types of modules, including package files modules.

Source maps are currently intentionally disabled in debug mode, however that isn't specific to package files. This applies to debug mode in general. So in the the strictest sense, no, package file modules (or indeed any modules) do not receive source maps to translate between two forms of non-minified code.

In relation to debug mode in general, yes. Package files are supported by both debug=1 and debug=2 today. You can consider source maps an evolution of debug=2. The main difference between debug v1 and v2 was the performance boost (from bundling code for large modules in one request instead of through many serial requests) and production-like behaviour (no global scope, same implementation closures and bundled delivery). Source maps is similar in that regard by going all the way to production mode and providing the underlying source via source maps.

There are no plans to deprecate debug mode. We may want to think about the other indirect behaviours of debug mode and think about how and if we want to expose those in production mode and/or through other means. Perhaps we could transition debug=2 to strip away its delivery logic in favour of normal production mode but leave the boolean mw.config/RLContext flag for runtime conditionals outside RL internals. Or perhaps a debug=3 parameter to make the default one day...

Change 960659 merged by jenkins-bot:

[mediawiki/core@master] ResourceLoader: Enable source maps by default

https://gerrit.wikimedia.org/r/960659

Do you plan on supporting package files modules when in resource loader debug mode?

Source maps are essentially a replacement for debug mode. They work transparently and directly in production mode, for all types of modules, including package files modules.

So in other words, no.

Personally, I have no plans to do any further related work, except for bug fixes and deployments, since I did this project for the performance team and that team no longer exists.

Change 963119 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] ResourceLoader: Bump STORAGE_VERSION

https://gerrit.wikimedia.org/r/963119

Change 963119 merged by jenkins-bot:

[mediawiki/core@master] ResourceLoader: Bump STORAGE_VERSION

https://gerrit.wikimedia.org/r/963119

Change 966945 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[operations/mediawiki-config@master] Enable source maps everywhere

https://gerrit.wikimedia.org/r/966945

Change 966945 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable source maps everywhere

https://gerrit.wikimedia.org/r/966945

Mentioned in SAL (#wikimedia-operations) [2023-10-20T03:15:46Z] <tstarling@deploy2002> Synchronized wmf-config/InitialiseSettings.php: Enable source maps everywhere T47514 (duration: 06m 26s)

tstarling added a project: User-notice.

Works for me.

Request rate is still very low, ~2 req/s.

Suggested Tech News message:

  • Source map support has been enabled on all wikis. When you open the debugger in your browser's developer tools, you should be able to see the unminified JavaScript source code.

A potential follow up is T352167, looks like the source map URL might end up being invalid if the "version" is raised

Source map error: Error: request failed with status 404
Resource URL: https://commons.m.wikimedia.beta.wmflabs.org/w/load.php?lang=es&modules=startup&only=scripts&raw=1&skin=minerva&target=mobile
Source Map URL: /w/load.php?lang=es&modules=startup&only=scripts&skin=minerva&sourcemap=1&version=xhk73

https://commons.m.wikimedia.beta.wmflabs.org/w/load.php?lang=es&modules=startup&only=scripts&skin=minerva&sourcemap=1&version=xhk73 is a 404 saying Can't deliver a source map for the requested version since the version is now 'i1rm6'