It looks like the problem with the scenario described above is that $title is being passed by value, not by reference. That should be an easy fix, if that's all that's needed, and I could back port that to the 1.35 RC.

But it looks like there's a TODO there by @DannyS712 in statelessFetchTemplate that says "TODO rewrite using only RevisionRecord objects". I'm cc'ing him here, perhaps we should take advantage of the opportunity, because anyone springs up to use the BeforeParserFetchTemplateAndTitle hook, to change the $id parameter as well to return a RevisionRecord instead? It would be an opportunity to fix the sad capitalization issue with the Hook class as well.

@DannyS712 to be concrete, the proposal is to replace BeforeParserFetchTemplateAndtitle($parser,$title,&$skip,&$id) (note bad case in hook name) with something like BeforeParserFetchTemplateRevisionRecord($title, &$skip, &$revRecord). That would allow the hook to return a MutableRevisionRecord for the non-existing $title (eg xyz/fr) which a SlotRecord::MAIN redirecting to the fallback language (eg xyz/en) and would eventually (next release or so) allow us to replace $rev_id with $revRecord in the implementation of Parser::statelessFetchTemplate.

That would avoid having to pass the $title by reference; the "new" title would be the title of the returned RevisionRecord (which could be different from the title supplied).

Change 617289 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/core@master] FOR DISCUSSION: Pass $title by reference to BeforeParserFetchTemplateAndtitle

https://gerrit.wikimedia.org/r/617289

Change 617294 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/core@master] FOR DISCUSSION: New BeforeParserFetchTemplateRevisionRecord hook

https://gerrit.wikimedia.org/r/617294

@cscott - Thanks for your patches. I believe they are still a WIP; and not ready for me to test. Will keep an eye on them.

@abi_ yes, I was hoping for some feedback from CPT about preferred approach. Maybe I'll poke @Krinkle today and see what his thoughts are on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/617294

@cscott did u get a chance to chat with @Krinkle?

@tstarling left a comment saying he was fine with my approach on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/617294 so it looks like I just have to update that patch and get it merged.

I just took a look at the patches, and unfortunately they won’t always work. They address the case where the template text is currently something like

{{#switch:<translate></translate>
| =
  <!--
    no garbage, this chunk of wikitext is on a translated page
    (Translate removed the empty translate tags), put the
    actual content here:
  -->
  [uninteresting wikitext removed for clarity]

| #default=
  <!--
    we have some garbage (the literal text “<translate></translate>”),
    so this chunk is on Template:Extension itself;
    let’s transclude the appropriate translation:
  -->
  {{#invoke:Template translation|renderTranslatedTemplate|template=Template:Extension|noshift=1}}
}}

However, in many templates instead of

{{#invoke:Template translation|renderTranslatedTemplate|template=Template:Extension|noshift=1}}

there’s

{{#invoke:Template translation|renderTranslatedTemplate|template=Template:Extension|noshift=1|uselang={{int:lang}} }}

i.e. they use the UI language, not the page language, for example because they’re used on talk pages, which are always English. This should be somehow addressed by the patch.

Further edge cases:

• Templates not strictly following this scheme. They may do some parameter preprocessing in the #default branch, or anything else that breaks with this change.
• Pages (i.e. where the template is used) not using Translate yet, but rather manual translation. This means that Module:Template_translation correctly recognizes its language (based on the subpage name), but Translate thinks it’s English (as the page language Translate uses is the default English).

These can be addressed using the syntax versioning Translate has: do this automatic translation only if the template has syntax version 3, and translation administrators should remove any preprocessing and check if all pages using the template use Translate before upgrading to version 3. (The check could be supported by some tool, as manually going through hundreds of pages is not funny.)

If you want something different from the (planned) normal behavior transcluding translatable pages, would it be acceptable to not use normal template transclusion syntax for that?

I don’t think so. The goal of this #switch hack is to avoid any changes on the template caller site compared to the not internationalized state.

Just came into my mind: what about adding a switch tag (e.g. __USEUILANGUAGE__) to the template?

I do not know how that UI inclusion works right now, but in general I don't think we can vary templates based on user language without breaking caching.

We have to check that there aren't any performance concerns doing syntax version check before enabling automatic translation page transclusion. But thanks for bringing up this switch-thingie now so we can avoid breaking it accidentally.

In T47096#6676496, @Nikerabbit wrote:

I do not know how that UI inclusion works right now, but in general I don't think we can vary templates based on user language without breaking caching.

From the template designer POV, it’s pretty straightforward (although quite hackish): {{int:lang}} transcludes the translation of MediaWiki:lang in UI language. These translations aren’t in MediaWiki core, but rather manually defined on multilingual wikis (c:MediaWiki:Lang/en, mw:MediaWiki:Lang/de, m:MediaWiki:Lang/fr etc.). I have no idea how this affects caching, but since only logged-in users can set their UI language, there are no caching issues for anons.

Change 617289 abandoned by C. Scott Ananian:
[mediawiki/core@master] FOR DISCUSSION: Pass $title by reference to BeforeParserFetchTemplateAndtitle

Reason:
Abandoned in favor of Ia5b5d339706ce4084c16948300e0e3418b11792e

https://gerrit.wikimedia.org/r/617289

@abi_ the latest version of https://gerrit.wikimedia.org/r/c/mediawiki/core/+/617294 should be ready to go; can you verify that it will work for your patch set?

An even more extreme example by @Pols12: it uses UI language in user and user talk namespace, page language everywhere else. I have no idea how to support it. Maybe another switch to turn off this magic and keep the old manual way?

In T47096#6680296, @cscott wrote:

@abi_ the latest version of https://gerrit.wikimedia.org/r/c/mediawiki/core/+/617294 should be ready to go; can you verify that it will work for your patch set?

Thanks for your work, left a comment on the patch.

In T47096#6688455, @abi_ wrote:

Thanks for your work, left a comment on the patch.

I think Title::getPageLanguage() does what you want.

Working with @cscott on getting this done.

Test setup

Two pages:

1. Base Translation Page
2. Translcuded Translation Page (Transcluded in Base Translation Page)

Test case #1

1. Base page is created and translated to 'fr'. When the base page is viewed in fr, the Transcluded Translation Page is displayed in source language. This is OK
2. Continuing from 1, the transcluded page is translated 'fr', TranslateRenderJob is run, but the 'fr' Base Translation Page still shows the content in source language.

Purging the cache for 'fr' version of the Base Page, fixes this issue.

Test case #2

Continuing from Test Case 1

1. Delete the 'fr' translation of the Transcluded Translation Page.
2. See the 'fr' version of the Base Translation Page. It will still display the deleted contents from the 'fr' version of Transcluded Translation Page.

Purging the cache for 'fr' version of the Base Page, fixes this issue.

Summary

So it seems like that updates to the Transcluded page does not seem to cause a re-render of the base page.

I'm not able to reproduce this issue with normal non translatable pages. In this case if the transcluded page is updated or deleted, the base page reflects the change without purging the cache.

I'm not sure whether the issue is with the new hook not registering the base page dependencies properly or something more that we need to do on the Translate end.

Some comments left on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/617294 -- see if you can determine if the $deps array is correct or not.

Test case #1

1. Base page is created and translated to 'fr'. When the base page is viewed in fr, the Transcluded Translation Page is displayed in source language. This is OK
2. Continuing from 1, the transcluded page is translated 'fr', TranslateRenderJob is run, but the 'fr' Base Translation Page still shows the content in source language.

Purging the cache for 'fr' version of the Base Page, fixes this issue.

This was fixed in this patchset: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Translate/+/573450/17..18

We're planning to add a new option when marking a page for translation, that will determine whether that page will be translated when it is transcluded in another page. Since this is not really a change in the syntax, we do not want to update the syntax version.

For example a translatable template B is added to page A. How template B is transcluded will depend on if "Enable page transclusion" option was selected when marking template B for translation.

If user chooses to enable translation aware transclusion for the template, based on the page language, appropriate translation of the template page will be loaded, if available. If the translation is not available then the source language of the template will be loaded.

Here's an example of how this looks:

In T47096#6735895, @abi_ wrote:

We're planning to add a new option when marking a page for translation, that will determine whether that page will be translated when it is transcluded in another page.

Will this option be reversible? As far as I understand you, it will; while the syntax version upgrade isn’t—once you updated to the latest syntax version, there’s no way back. (My personal opinion as a translation administrator is that the more options, the better, so I’m for reversibility.)

In T47096#6737249, @Tacsipacsi wrote:
Will this option be reversible? As far as I understand you, it will; while the syntax version upgrade isn’t—once you updated to the latest syntax version, there’s no way back. (My personal opinion as a translation administrator is that the more options, the better, so I’m for reversibility.)

Yes, this option will be reversible by marking the page for translation again, whereas it would not be reversible if we were to implement this via a syntax version update.

This feature requires the following patch from core: 617294: parser: new BeforeParserFetchTemplateRevisionRecord hook | https://gerrit.wikimedia.org/r/c/mediawiki/core/+/617294

We're not planning to support this feature for older version of MediaWiki (i.e < 1.36)

Test results from the latest changes done for this feature are mentioned below.

Base setup

Two translatable pages: Base Page, Child Page (Transcluded in Base Page)
Both pages are marked for translation.

Cases

• 1. When base page is loaded in a language (fr) to which child page is not translated, the source language is displayed
• 2. When the Child page is translated to that language, and the base page is refreshed the changes should be visible
• 3. Updating the child page language translations, automatically updates the base page as well.
• 4. Deleting a child page language translation, DOES NOT update the base page
  • Need to create a bug for this, but not urgent.
• 5. If the child translatable page is updated, the base page translation shows the child template as fuzzy.
• 6. When child translatable page is updated, the base translation pages for which the child page is not translated, are updated with the update from the child translatable page.

Change 617294 merged by jenkins-bot:
[mediawiki/core@master] parser: new BeforeParserFetchTemplateRevisionRecord hook

https://gerrit.wikimedia.org/r/617294

Change 573450 merged by jenkins-bot:
[mediawiki/extensions/Translate@master] Enable "opt-out" translation aware transclusion for templates

https://gerrit.wikimedia.org/r/573450

Testing report on MediaWiki:

Created a Base Page and Child Page
Translated Base Page to German, and then subsequently the Child Page to German.

It took a while but the base page was eventually updated to show the updated contents of the child page.

Leaving it open for a few days to track any issues reported.

Issue: translation aware transclusion is checked by default on Special:PageTranslation for legacy pages (not marked for translation since the deployment of this change). This will inevitably lead to accidentally breaking things, it should be unchecked by default.

Change 668126 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/Translate@master] Do not select template transclusion when marking page for translation

https://gerrit.wikimedia.org/r/668126

In T47096#6866085, @Tacsipacsi wrote:

Issue: translation aware transclusion is checked by default on Special:PageTranslation for legacy pages (not marked for translation since the deployment of this change). This will inevitably lead to accidentally breaking things, it should be unchecked by default.

The changes to revert this is quite trivial. I've submitted a patch to do this The initial reasoning to have this checked for legacy pages was:

If supports template transclusion was defaulted to false, later even if we flipped the default flag to true, the pages marked as false would not support template transclusion. This would leave a lot of pages with the supportsTransclusion flag as false, even though they could support transclusion.

But I can see why this would cause issues. I'll discuss the pro's and con's of this based on @Tacsipacsi input's here with Niklas and decide how to proceed.

Is it possible to make this option checked by default for pages being marked for translation for the first time, but unchecked for pages being re-marked? The risk of breaking things is quite low in case of a newly translatable template, it’s high only for templates already in use. This way fewer pages would be left with supportsTransclusion set to false, yet the risk of breaking things would be considerably lower.

In T47096#6888462, @Tacsipacsi wrote:

Is it possible to make this option checked by default for pages being marked for translation for the first time, but unchecked for pages being re-marked? The risk of breaking things is quite low in case of a newly translatable template, it’s high only for templates already in use. This way fewer pages would be left with supportsTransclusion set to false, yet the risk of breaking things would be considerably lower.

This is a better approach then what my patch was initially doing. I've updated the patch accordingly.

@Tacsipacsi How many pages are there that would break? And even if they do, it's easy to undo the change (assuming one knows what caused it in the first place).

However, out of caution, I have approved the patch and it will be deployed with the next train.

Change 668126 merged by jenkins-bot:
[mediawiki/extensions/Translate@master] Do not enable supportsTransclusion by default

https://gerrit.wikimedia.org/r/668126

In T47096#6891296, @Nikerabbit wrote:

@Tacsipacsi How many pages are there that would break?

Quite a number of. I have no exact figures, of course, but I’ve reverted nine edits by @Shirayuki so far that broke pages, despite of the fact that he deliberately changed them, just wasn’t aware of the broken pages. Some are templates that use the UI language instead of the page language (using the {{int:lang}} hack), others are templates used on pages translated without the Translate extension, where {{TNT}} recognizes the language suffix in the title, but Translate sees that Title::getPageLanguage() returns English.

And even if they do, it's easy to undo the change (assuming one knows what caused it in the first place).

Yes, my concern is that translation admins may not know what caused the issue, or even become aware that there’s an issue in the first place.

However, out of caution, I have approved the patch and it will be deployed with the next train.

Thanks!

The option is now,

• Checked by default for pages being marked for translation for the first time,
• Unchecked for pages being re-marked unless they already have support transclusion set.

So, in order to determine whether a page is translation-aware when transcluded, is this now a sufficient test to look at this checkbox state in Mark special page?

I mean:
I want to know whether I can safely remove {{TNT}} call (i.e. turning {{TNT|some template}} into {{some template}}).
Can I just go to Special:PageTranslation?do=mark&target=Template:some_template and look at Enable translation aware transclusion for this page checkbox: if it is checked, I can safely remove TNT call, else I can’t. Right?

Change 672242 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/Translate@master] Special:PageTranslation: Add tag for pages that support transclusion

https://gerrit.wikimedia.org/r/672242

In T47096#6912104, @Pols12 wrote:

So, in order to determine whether a page is translation-aware when transcluded, is this now a sufficient test to look at this checkbox state in Mark special page?

That is correct. Eventually we plan to add tags on Special:PageTranslation similar to what we do for pages that use old syntax.

In T47096#6912891, @abi_ wrote:

Eventually we plan to add tags on Special:PageTranslation similar to what we do for pages that use old syntax.

Thanks, I was about to create a new task for it—@Pols12’s trick works well to decide whether an individual page has translation-aware transclusion enabled, but it doesn’t let translation admins keep track of the conversion process.

Change 672242 merged by jenkins-bot:
[mediawiki/extensions/Translate@master] Special:PageTranslation: Add tag for pages that support transclusion

https://gerrit.wikimedia.org/r/672242

We're now highlighting the pages that don't support template transclusion

This is deployed and tested on translatewiki.net; moving this task to done, but leaving it open in case of further queries.

Currently, if I create a translatable template and mark it, and translate it to a language such as Spanish, if I transclude the template on a page like [[Test page/es]] even though it's the /es subpage, it still shows up in English, instead of Spanish. However, if I change the page language under page info, then the translated version is shown. It would be nice if it could show the translated version based on the subpage, as well as the page language. (tested on mediawiki.org and test.wikipedia.org)

Given the page language can be easily set through Special:PageLanguage, I don’t think it is really useful to check subpage name.

In T47096#6982080, @Pols12 wrote:

Given the page language can be easily set through Special:PageLanguage, I don’t think it is really useful to check subpage name.

The issue is that would require us to do that for a lot of pages on MediaWiki.org since we have thousands of translatable pages there.

But most of them already use the Translate extension, and thus have the page language properly set. And for the few that don’t, using {{TNT}} is still an option until they are migrated/have their page language manually set.