usage changes via LinkUpdate
Open, MediumPublic
Actions

Assigned To

None

Authored By

	tstarling
	Jul 29 2010, 2:40 AM

Description

When someone adds or removes a category from a page, the relevant category page changes. And when someone adds or removes an image from a page, the relevant image description page changes, due to the list of file links.

Cache updates for these changes are handled by LinksUpdate::invalidatePages(). However, this function does not update the HTML file cache (i.e. $wgUseFileCache=true), and it does not purge Squid. Thus, anonymous users see the old versions of categories and image description pages after an edit is made.

The fix is made more complicated by the fact that neither of these updates should be done in the same transaction as the main link update. The link update transaction locks a lot of rows, and needs to be closed off as soon as possible.

Version: 1.17.x
Severity: normal

Details

Reference: bz24575

	Subject	Repo	Branch	Lines +/-
	Purge category/file pages out of Cdn/File cache when membership change	mediawiki/core	master	+40 -26

Customize query in gerrit

Related Objects

Mentioned In: T279205: Harden and improve HTTP cache headers and purging in MediaWiki core (Sprint placeholder)
T238426: Creating a file description page with a redirect to an existing image, doesn't purge pages using the recently created redirect
T143852: Caches stuck when uploading images with maintenance script
Mentioned Here: T104711: File description page should be purged after deleting file
T109214: File upload should clear parser and file cache on usage pages
T143852: Caches stuck when uploading images with maintenance script

Event Timeline

• bzimport raised the priority of this task from to Low.Nov 21 2014, 11:08 PM

• bzimport added a project: MediaWiki-Page-editing.

• bzimport set Reference to bz24575.

• bzimport added a subscriber: Unknown Object (MLST).

tstarling created this task.Jul 29 2010, 2:40 AM

Weren't LinksUpdate done in the DeferredUpdates?

(In reply to comment #1)

Weren't LinksUpdate done in the DeferredUpdates?

No, and it wouldn't help if they were. DeferredUpdates is mostly useless, I've migrated most callers away from it. I don't know why Chad bothered to work on it in 1.19.

info wrote:

I'm running MediaWiki 1.19.0 and I have the exact problem. My main page doesn't change too, but that kinda makes sense because it doesn't get edited, the changes are just made through template rotation.

I'm gonna add a shell script (which deletes the category files and the main page from the file cache) to crontab.

Bawolff merged a task: T35238: category pages in file cache do not get updated when I add or remove same category on a page.Apr 7 2015, 8:09 PM

Bawolff added subscribers: Kulmanseidl, MarkAHershberger, aaron, • bzimport.

Is anybody aware of a workaround for this? Not even purging pages seems to help. Only fully clearing the cache (through whatever means) makes a difference.

Is there for example any way I can refresh the caches using a cronjob accessing some php page? (Our project is hosted on a shared host so I dont have many possibilities there)

Seems to be related: https://commons.wikimedia.org/w/index.php?title=Commons:Village_pump&oldid=160031657#There_are_no_pages_or_files_in_this_category

Steinsplitter raised the priority of this task from Low to Medium.May 6 2015, 3:21 PM

Steinsplitter set Security to None.

Could be related but also could be not. Purging category pages did not help in my case, but it is described as a solution in the linked one. I have to clear the entire cache. I am not sure how their configuration would differ from mine to lead to this difference.

Bawolff merged a task: T114160: Non-empty Categories Displaying as Empty When Not Logged-In.Oct 22 2015, 5:57 PM

Bawolff added subscribers: • Rdicerb, zhuyifei1999, Aklapper, PeterMEssex150.

This bug seems easier to fix now then when it was originally filed. I think you can just add an onTransactionIdle callback to squid/file cache purge the pages to the callback that's in SqlDataUpdate::invalidatePages

matmarex merged a task: T145196: Adding or removing files to article should refresh filecache for image pages.Sep 9 2016, 4:27 PM

matmarex added subscribers: Subfader, Ciencia_Al_Poder, Legoktm and 2 others.

Ciencia_Al_Poder added a project: MediaWiki-libs-BagOStuff.Oct 19 2016, 8:23 PM

Krinkle renamed this task from Category and image description pages not purged from file cache or squid cache on link update to Purge Category and File description pages from file cache or squid cache on link update.Apr 7 2017, 10:17 PM

Krinkle added a project: Performance-Team.

Krinkle removed a subscriber: • wikibugs-l-list.

Krinkle moved this task from General to HTTP Cache on the MediaWiki-libs-BagOStuff board.Apr 7 2017, 10:30 PM

Krinkle mentioned this in T143852: Caches stuck when uploading images with maintenance script.Apr 7 2017, 11:00 PM

MarkAHershberger unsubscribed.Apr 8 2017, 2:44 PM

• Gilles moved this task from Inbox, needs triage to Radar on the Performance-Team board.Apr 12 2017, 7:50 PM

Krinkle edited projects, added MediaWiki-Platform-Team-Archived; removed Performance-Team.Apr 12 2017, 7:57 PM

May be related:

CCicalese_WMF moved this task from Inbox to Backlog on the MediaWiki-Platform-Team-Archived board.Jan 9 2018, 2:49 AM

CCicalese_WMF edited projects, added Core-Platform-Team-Old; removed MediaWiki-Platform-Team-Archived.Jul 12 2018, 12:54 AM

CCicalese_WMF moved this task from Inbox to Backlog on the Core-Platform-Team-Old board.Jul 12 2018, 1:02 AM

CCicalese_WMF added a project: Platform Team Legacy (Later).Oct 1 2018, 11:37 PM

CCicalese_WMF edited projects, added Core Platform Team Goals (MCR: Uncategorized); removed Core-Platform-Team-Old.Oct 2 2018, 12:33 AM

CCicalese_WMF edited projects, added Platform Engineering; removed Core Platform Team Goals (MCR: Uncategorized).Oct 2 2018, 2:37 AM

CCicalese_WMF moved this task from Inbox to Triage Meeting Inbox on the Platform Engineering board.

CCicalese_WMF moved this task from Triage Meeting Inbox to Needs Cleaning - Security, stability, performance, and scalability (TEC1) on the Platform Engineering board.Oct 3 2018, 3:58 PM

CCicalese_WMF edited projects, added Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1)); removed Platform Engineering.

Looks like easy according to T26575#1746030

This bit me too. My categories and images weren't updated for users not logged in. Some of the ones I found that were being effected were over a year old. :(

In T26575#5030668, @Ciencia_Al_Poder wrote:

Looks like easy according to T26575#1746030

I think you're radically under-estimating how deeply scary anything related to the Squid/Varnish/ATS CDN caching layer is. This would not remotely be a suitable first bug.

WDoranWMF moved this task from Later to Mop Column on the Platform Team Legacy board.Jul 5 2019, 3:45 PM

WDoranWMF removed a project: Platform Team Legacy (Later).

CCicalese_WMF moved this task from Needs Cleaning - Security, stability, performance, and scalability (TEC1) to Icebox on the Platform Engineering board.Nov 6 2019, 7:45 PM

CCicalese_WMF edited projects, added Platform Engineering; removed Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1)).

Krinkle unsubscribed.Nov 6 2019, 8:18 PM

Krinkle renamed this task from Purge Category and File description pages from file cache or squid cache on link update to Purge Category and File description pages from HTTP/File cache when members/usage changes via LinkUpdate.Dec 3 2019, 10:49 PM

Krinkle mentioned this in T238426: Creating a file description page with a redirect to an existing image, doesn't purge pages using the recently created redirect.Dec 3 2019, 10:53 PM

ahmad subscribed.Mar 20 2020, 5:12 PM

WDoranWMF edited projects, added Platform Engineering (Icebox); removed Platform Engineering.Mar 23 2020, 2:04 PM

Bawolff awarded a token.Mar 24 2020, 10:18 PM

So right now, we're already updating page_touched for these pages, just not clearing them in varnish.

I guess maybe PurgeJobUtils::invalidatePages could also send Cdn/filecache purges too?

Change 583180 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[mediawiki/core@master] Purge category/file pages out of Cdn/File cache when membership change

https://gerrit.wikimedia.org/r/583180

gerritbot added a project: Patch-For-Review.Mar 25 2020, 12:49 AM

Krinkle mentioned this in T279205: Harden and improve HTTP cache headers and purging in MediaWiki core (Sprint placeholder) .Apr 2 2021, 9:25 PM

Lahwaacz subscribed.Apr 22 2021, 8:03 PM

Krinkle edited projects, added MediaWiki-Core-HTTP-Cache; removed MediaWiki-libs-BagOStuff.Jan 22 2022, 11:16 PM

lilydjwg subscribed.Dec 8 2022, 11:30 AM

Restricted Application added a project: Performance-Team. · View Herald TranscriptDec 8 2022, 11:30 AM

Pppery edited projects, added Patch-Needs-Improvement; removed Patch-For-Review.Apr 8 2023, 5:13 PM