URL highly welcome.

sorry, figured it would be clear in context: http://en.wikipedia.beta.wmflabs.org

Ten bucks says this is caused by I9a2ebd50 being merged. Try clearing your browser's cache, if you didn't already.

http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page looks like on your screenshot to me. It looks like the HTML is in a pre-I9a2ebd50 version (with <h5> headings), but the CSS is in a post-I9a2ebd50 version (styles for h3).

Doing a http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page?action=purge didn't fix the issue.

I had tried http://deployment.wikimedia.beta.wmflabs.org/ . ;)

en.wikipedia.beta.wmflabs.org/wiki/Main_Page looks okayish when being logged in with Firefox 16.0.2, but I literally got

<centralnotice-template-welcome>

a few times (not always).

As it loads slowly it might be yet another one of all those CSS off-and-on
errors lately.

I saw similar behavior on test.wiki and Commons after switching to wmf5, but it went away after hard refresh.

The garbled display affects the main pages of http://meta.wikimedia.org and http://wikimediafoundation.org also, as well as certain other pages like http://test.wikipedia.org/wiki/Wikipedia:Create_a_new_page .

The garbled display is visible in Firefox and IE9 but those pages look normal in Chrome.

changing Product/Component to Mediawiki/General since this issue made it out of beta labs to production

(In reply to comment #7)

The garbled display is visible in Firefox and IE9 but those pages look normal
in Chrome.

It's broken for me in Chrome as well. (24.0.1290.1 dev on Ubuntu)

Problem seems to be that ResourceLoader is failing to load some new CSS that is needed unless you use 'debug=true' in the URL.

Fixed with https://gerrit.wikimedia.org/r/#/c/35817/ and
https://gerrit.wikimedia.org/r/#/c/35819

Reopen if there are still any residual issues.

Seeing this on en.wiki while logged in. The HTML isn't cached, as it's displaying the broken items as h3s (the newer tag). The problem seems to be that ResourceLoader isn't delivering the updated CSS. This might just be the 5 minutes of doom, but I'm not sure when the deployment happened.

Fresh screenshot from Chrome

Attached:

I'm not seeing the big headings, but I am seeing the same incorrect misbehaving headings in the left-hand nav: indented "Navigation" , "Interaction" instead of "Support", and no expand/contract arrows on "Interaction" and "Toolbox".

Touched and synced the startup.js and ext.vector.collapsibleNav.css files.
Seems to be working correctly now.

Seems to be fixed for most people. One report of problematic display even after clearing cache:
"I tried to purge the cache but still no change. Using safari on iPad. Monobook."

Otherwise, seems to be resolved. Lowering severity.

Created attachment 11459
Screenshot of header issues (on main page)

It seems to be resolved for me.

For the record, I took the attached (cropped to relevant part) at ~ 2012-12-05 02:49.

Attached:

Swatjester wrote:

I still have the hidden navigation tabs at the top in monobook. I did not have this yesterday. Clearing cache, force reloading, closing and restarting browser, and purging the server cache, all did nothing. According to reports on village pump technical and other places, I'm not the only one.

Chrome, XP Pro.

• Bug 42680 has been marked as a duplicate of this bug. ***

Still appearing for me too (as in, appeared ~5 seconds ago).

We backported the new CSS to wmf4 and deployed it to the remaining wikis today. This should minimize the breakage due to stuck server caches, but I wouldn't be surprised if some people still report the problem tomorrow.

Getting reports of this bug on pl.wiki:
https://pl.wikipedia.org/wiki/Wikipedia:Kawiarenka/Kwestie_techniczne#Menu_nawigacyjne

Specifically: http://iv.pl/images/90157415975042120178.jpg

Note the user is logged in and apparently debug=true fixes the issue (according to folks on IRC).

Reported on German Wikipedia as well: https://de.wikipedia.org/wiki/Wikipedia:Fragen_zur_Wikipedia#Navigationsmen.C3.BC

Just to summarize the situation...

This bug consists of 3 different potential problems, 2 of which have been successfully resolved:

1. Users getting updated CSS, but cached HTML - fixed with https://gerrit.wikimedia.org/r/#/c/35817/ and https://gerrit.wikimedia.org/r/#/c/35819

2. Users getting updated HTML, but outdated client-side-cached CSS ("5 minutes of doom") - fixed by deploying CSS a day before the HTML changes. (Our CSS files are supposed to expire after 5 minutes at the most on the client-side.)

3. Users getting updated HTML, but outdated server-side-cached CSS (that is over 5 minutes old). This is the one we haven't solved. The problem is difficult to reproduce reliably, and seems to fade away on its own. Users who experience this problem state that they are logged in, have cleared their browser cache, and purged the pages. The problem has been reported across a wide range of browsers and from users in both the US and Europe. If we accept these reports as accurate, the only explanation I can come up with is that some WMF servers are holding onto old caches of CSS and JS pages after they have been updated and synced from fenari. (Anecdotal reports from mobile and fundraising developers lead me to believe that this affects JS as well as CSS.) Although the current incarnation of this problem was merely cosmetic, it has the potential to cause major site breakage. We may want to have someone from Ops investigate further if possible.

Missing css on https://hu.wikipedia.org/wiki/Lohr_Ferenc
"Navigation menu" is displayed in lower left corner on every page with HTTPS. With simple http, or with debug=1 it gets correct css and is not displayed.

(In reply to comment #24)
Only with Chrome. Windows 7 64-bit, Vector, logged in and logged out.

I believe I have the solution. Clearing the cache alone won't do it. Everything must be cleared from the browser as reset as if the browser were browsing for the first time. Clear the history, cache, temporary Internet files, browsing data, and cookies. I hope this helps.

(In reply to comment #24)
Resolved deleting the browser cache.

In the case of people fixing the problem by clearing their cache, there are two possible explanations:

1. The CSS files had bad cache headers (didn't expire in 5 minutes)
2. The request for new CSS files happened to hit a server that was correctly serving the new files, rather than a server that was serving the old files.

I suspect the later for 3 reasons: Cyberpower mentions they had to completely reset their browser, including cookies. This would have created a new session and reset whatever server they were stuck to. Also several people have mentioned that clearing their browser cache did not fix the problem for them. Finally, I haven't seen any bad cache headers myself and I've been looking at them frequently.

At 00:10 UTC today (Thursday in the US, Friday in Europe) Ryan Lane cleared all the varnish caches used by bits. If anyone is still noticing this bug, let me know.

Kaldari: Is there anything left to do here, or can we assume that this is FIXED now?

Summarizing recommended actions for affected users again:

• http://en.wikipedia.org/wiki/Wikipedia:Bypass_your_cache
• Add ?action=purge to the web address
• Add ?debug=true to the web address

[Only very few affected users left => removing blocking 38865]

Andre: Unfortunately, I have no idea as I don't yet know what the underlying problem actually was. It would require a dedicated debugging effort to solve and I've already spent too much time on this I'm afraid. This is really an issue that should be handled by platform or ops, but I haven't had any luck getting other people interested in looking at it. I think you can close this bug, but I won't be surprised if it is reopened that next time an interface change is deployed.

Per http://lists.wikimedia.org/pipermail/wikitech-l/2012-December/065126.html this is still happening.

Mido.Architect wrote:

Navigation menu in Japanese wikipedia displayed incorrectly on the side bar

as an example of the bug described here http://lists.wikimedia.org/pipermail/wikitech-l/2012-December/065126.html

Attached:

Mohamed: Please clarify if you've tried all three steps listed in comment 30.

?debug=true always fixes the problem temporarily, but isn't a real solution.

I've also heard that clearing cookies (resetting your session) fixes it for some people.

Gerrit 34702 (bug 42354) changed all h5 to h3, but that breaks the collapsible navigation in Vector for cached html, because gerrit 35817 only contains css.

Please add for 1.21 a backward compatible javascript to avoid the need of mass purge of cached html for [anon] users (also needed for third party wikis).

Gerrit 30361 also contains a javascript change, with is not backported in gerrit 35819 (and that is for 1.21wmf5 only, but cache lives longer than 14 days?)

(In reply to comment #37)

Gerrit change #34702 (bug 42354) changed all h5 to h3, but that breaks the
collapsible
navigation in Vector for cached html, because Gerrit change #35817 only
contains css.

That sounds a lot like bug 43227. Maybe should be handled there.

(In reply to comment #37)

Gerrit change #34702 (bug 42354) changed all h5 to h3, but that breaks the
collapsible
navigation in Vector for cached html,

...and also the hack used by dozen of wikis to workaround bug 708:
https://pt.wikipedia.org/wiki/MediaWiki:Gadget-InterprojectLinks.js
I just fixed two (on ptwiki and ptwikibooks), but there are copies on other wikis...

(In reply to comment #37)

Please add for 1.21 a backward compatible javascript to avoid the need of
mass
purge of cached html for [anon] users (also needed for third party wikis).

I0a05ed39 should fix this.

Okay, I *think* this has been entirely fixed for now, with everything that should support both h3 and h5 supporting them both. If something is still not perfect, do speak up and I'll go and fix it.

Also, MediaWiki core and extension Vector both need releases (Vector to include the fix for bug 43227, MediaWiki to update bundled Vector). I am hoping that people who handle this are CC'd here.

So, unless this is supposed to stay open as a catch-all for further cache-related issues, it should probably be closed. (And the summary badly needs clarifying in either case.)

Closing per comments. New issues along this line to go under new reports.

New incarnation of this bug at bug 43805.

Still has to be backported to 1.20, if needed (for 1.20.4; it didn't make it to 1.20.3).