https://mediawiki.org redirects to http://www.mediawiki.org/ currently. This is wrong and should be fixed.
Version: unspecified
Severity: major
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=68553
Description
Description
Details
Details
- Reference
- bz31369
Related Objects
Related Objects
Event Timeline
Comment Actions
There are lots of these (I think https://wikipedia.org is broken too) that are stuck in Squid cache and need to be purged.
Comment Actions
(In reply to comment #1)
There are lots of these (I think https://wikipedia.org is broken too) that are
stuck in Squid cache and need to be purged.
Are you sure this has to do with Squid cache? Looking at http://noc.wikimedia.org/conf/redirects.conf, it says:
RewriteCond %{HTTP_HOST} mediawiki.org
RewriteRule ^/(.*)$ http://www.mediawiki.org/$1 [R=301,L]I assumed that was the root cause of the issue here, if redirects.conf is up-to-date.
Comment Actions
Like this?
RewriteCond %{HTTPS} off
RewriteCond %{HTTP_HOST} mediawiki.org
RewriteRule ^/(.*)$ http://www.mediawiki.org/$1 [R=301,L]
RewriteCond %{HTTPS} on
RewriteCond %{HTTP_HOST} mediawiki.org
RewriteRule ^/(.*)$ https://www.mediawiki.org/$1 [R=301,L]Comment Actions
(In reply to comment #3)
Like this?
RewriteCond %{HTTPS} off RewriteCond %{HTTP_HOST} mediawiki.org RewriteRule ^/(.*)$ http://www.mediawiki.org/$1 [R=301,L] RewriteCond %{HTTPS} on RewriteCond %{HTTP_HOST} mediawiki.org RewriteRule ^/(.*)$ https://www.mediawiki.org/$1 [R=301,L]
That looks about right. It's unclear whether this bug should be expanded to cover the other (several dozen) similar cases in redirects.conf. It's also unclear whether there's a saner system to do this. %{HTTPS} might be best... but it's going to result in an awful lot of code duplication for one letter.
Comment Actions
This is one of the more evil HTTPS bugs I've seen.. could lead people to believe they're using HTTPS when they're not. Elevating importance and severity.
Comment Actions
Still not fixed. bumping rt ticket as well.
http://rt.wikimedia.org/Ticket/Display.html?id=1668
Comment Actions
For the interested, Ryan Lane says on RT that he hasn't had time to do anything with HTTPS right now, but he is aware of it.
Comment Actions
issue seems to be fixed now. Daniel "mutante" Zahn did some tweaks last week :
With curl (-I show headers, -L follow Location: hints)
Without trailing slash:
$ curl -IL https://www.mediawiki.org 2>/dev/null|grep Location
Location: https://www.mediawiki.org/wiki/MediaWiki
$
With trailing slash:
$ curl -IL https://www.mediawiki.org/ 2>/dev/null|grep Location
Location: https://www.mediawiki.org/wiki/MediaWiki
$
Seems fixed to me so :)
Comment Actions
(In reply to comment #12)
issue seems to be fixed now. Daniel "mutante" Zahn did some tweaks last week :
With curl (-I show headers, -L follow Location: hints)
Without trailing slash:
$ curl -IL https://www.mediawiki.org 2>/dev/null|grep Location
Location: https://www.mediawiki.org/wiki/MediaWiki
$With trailing slash:
$ curl -IL https://www.mediawiki.org/ 2>/dev/null|grep Location
Location: https://www.mediawiki.org/wiki/MediaWiki
$Seems fixed to me so :)
Err, your test case is flawed. This bug is about "https://mediawiki.org", not "https://www.mediawiki.org" (comment 0 and bug summary).
$ curl -Is "https://mediawiki.org" | grep Location
Location: http://www.mediawiki.org/
This behavior is still wrong.
Comment Actions
(In reply to comment #14)
Oh my god.
Well http://wikimedia.org/ has the same issue :-)
You mean https://wikimedia.org, but yes, it appears to have the same issue:
$ curl -Is "https://wikimedia.org/" | grep Location
Location: http://www.wikimedia.org/
Comment Actions
The issue must be in redirects.conf from operations/apache-config
Gitweb:
https://gerrit.wikimedia.org/r/gitweb?p=operations/apache-config.git
Clone:
https://gerrit.wikimedia.org/r/p/operations/apache-config.git
Comment Actions
(In reply to comment #4)
It's unclear whether this bug should be expanded to
cover the other (several dozen) similar cases in redirects.conf. It's also
unclear whether there's a saner system to do this. %{HTTPS} might be best...
but it's going to result in an awful lot of code duplication for one letter.
Well, it's clearer now that there's a mess of bugs about the exact same issue. I'm going to make some noise and dupe them all down to this bug, as it came first and it contains reasonably useful back-and-forth between me and Daniel. I'll aggregate a few test cases and RT links in a single comment as well, I guess.
This means bug 33751 (redirects.conf), bug 35740 (Wikisource), bug 36951 (wikimediafoundation.org), bug 36952 (tracking bug) will be marked as duplicates.
I'm changing this bug's summary from "https://mediawiki.org redirects to http://www.mediawiki.org/" to "Certain Wikimedia redirects improperly go from HTTPS to HTTP; Apache's redirects.conf needs adjustments".
Comment Actions
Test cases
- https://mediawiki.org
- https://wikimedia.org
- https://wikipedia.org
- https://www.wikisource.org
- https://www.wikimediafoundation.org
RT tickets (I believe):
- RT #1668
- RT #2847
- RT #2783
Comment Actions
(In reply to comment #23)
Actually there is dozens and dozens more:
Right. The question is whether these can be solved in a simpler/cleaner/saner way than specifying "RewriteCond %{HTTPS} off" and "RewriteCond %{HTTPS} on" a million times.
According to my reading of http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.30, the Location header must be an absolute URI. Someone suggested that switching these rules to use protocol-relative syntax might resolve this bug, but in addition to the HTTP spec, Apache's rewrite/redirect logic apparently interprets any string beginning with a "/" to be a relative URI path, so that option is a no-go.
As I said earlier, %{HTTPS} might be best... but it's going to result in an awful lot of code duplication for one letter.
Comment Actions
(In reply to comment #24)
(In reply to comment #23)
Actually there is dozens and dozens more:
Right. The question is whether these can be solved in a simpler/cleaner/saner
way than specifying "RewriteCond %{HTTPS} off" and "RewriteCond %{HTTPS} on" a
million times.According to my reading of
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.30, the Location
header must be an absolute URI. Someone suggested that switching these rules to
use protocol-relative syntax might resolve this bug, but in addition to the
HTTP spec, Apache's rewrite/redirect logic apparently interprets any string
beginning with a "/" to be a relative URI path, so that option is a no-go.
Indeed. The spec says it should be absolute and apache doesn't support relative. But, browsers (the ones I've tested) do and always have supported relative Location headers. But anyway, as you say, a no-go.
As I said earlier, %{HTTPS} might be best... but it's going to result in an
awful lot of code duplication for one letter.
We could go for a build process. Where the thing is auto-generated adding one for each with the HTTPS case.
Comment Actions
(In reply to comment #26)
see Gerrit change #13293.
I don't have a Gerrit account, so I'm commenting here. You incorrectly switch "^/(.*)$" to "^/(.*$)" several times in that changeset.
Comment Actions
(In reply to comment #27)
You incorrectly switch "^/(.*)$" to "^/(.*$)" several times in that changeset.
As you just said on IRC that wasn't me.
Anyway, as I just said: there's essentially no difference between those 2 expressions in practice.
Comment Actions
What's the status of https://gerrit.wikimedia.org/r/13293? I see Ryan commented on June 29 to wait for the weekend. It's been a few weekends. Has this been merged/deployed? Can it be?
Comment Actions
(In reply to comment #29)
What's the status of https://gerrit.wikimedia.org/r/13293? I see Ryan
commented on June 29 to wait for the weekend. It's been a few weekends. Has
this been merged/deployed? Can it be?
Bump. The dupes are piling up here. Copying Sumana (Andre is already on the CC list). It'd be awfully nice to get this bug moving forward. :-)
https://gerrit.wikimedia.org/r/13293 had some further activity in September, but it's mostly remained quite quiet. I'm not sure what else is needed here to resolve this bug, though I'm happy to help if I can.
Comment Actions
Gerrit change #13293 needs an update - that's one way to speed this up.
dzahn: Are you actually working on this, or should the assignee be reset?
Comment Actions
Andre the change is pending review and there are a few issues in the patch :-D I don't think anyone is specially assigned to review it.
Comment Actions
(In reply to comment #35)
I don't think anyone is specially assigned to review it.
I think Faidon volunteered.
Comment Actions
(In reply to comment #36 by MZMcBride)
I think Faidon volunteered.
Faidon: Is that the case? If so feel free to assign this ticket to you and steal it from Daniel.
Comment Actions
The patch mentioned above has been abandoned. See reason on the gerrit patch set comments.
Comment Actions
I think this bug is quite important. It's been open for almost two years - is there any chance we can get this working with a less beautiful solution (e.g. duplicating the logic for every domain), and then have a follow-up bug to make it more efficient?
Also - the WMF supposedly prioritizes HTTPS...
https://blog.wikimedia.org/2013/08/01/future-https-wikimedia-projects/
Comment Actions
Asher, Ryan L., Faidon, or anyone else in ops who might know:
Gerrit changeset 13293 indicates that these redirects should be moved to Varnish. Is this possible now? If so, do you have any pointers about which file(s) should be changed in order to move forward toward resolving this bug?
As Erik noted in comment 6, this is a pretty evil bug.
Comment Actions
(In reply to comment #40)
Gerrit change #13293 indicates that these redirects should be moved to
Varnish. Is this possible now? If so, do you have any pointers about which
file(s) should be changed in order to move forward toward resolving this bug?
I suppose it may be possible to move some domains to varnish sooner than others but I'm not sure if it's worth it. (We are making recent progress)
"redirects should be moved to varnish" isn't quite right. Redirects will still be served from the same apaches they are now. Just they'll need to go through varnish instead of squid in order to fix this bug. More status below.
We recently got an OTRS ticket about this:
On 2013-08-09 someone wrote:
Subject: FYI: https://wikipedia.org redirects to http://www.wikipedia.org
would make life a lot easier if i could just type https://wikipedia.org to
get started.
Our reply:
Thanks for pointing this out! I'm also looking forward to that being fixed.
That's bug 31369https://bugzilla.wikimedia.org/31369 which is blocked on
https://wikitech.wikimedia.org/wiki/Projects#Switch_.22text.22_to_Varnish
(see also
http://lists.wikimedia.org/pipermail/wikitech-l/2013-August/071086.html )Also, https://blog.wikimedia.org/2013/08/01/future-https-wikimedia-projects/
(which mentions https://www.eff.org/https-everywhere which is probably the
best short-term solution for you)
Comment Actions
MZMcBride, no, text hasn't moved to Varnish yet. It's being worked on but we're not there yet. When that happens, operations/puppet, templates/varnish/text* would be the place to look for.
jeremyb, I think the idea was that we'd move this into VCL and do the redirects before even reaching Apache. I'm not sure if I like the idea. I'll leave it to whoever decides to implement that :) I agree that it's nasty, though.
Comment Actions
(In reply to comment #42)
jeremyb, I think the idea was that we'd move this into VCL and do the
redirects
before even reaching Apache. I'm not sure if I like the idea. I'll leave it
to
whoever decides to implement that :) I agree that it's nasty, though.
I'm not completely sure which I prefer but I thought the general trend was to move stuff out to apache/php where we could (e.g. zero/carriers).
We could always use a switch statement in asm in VCL to decide which redirect code path to take for a given request.
Anyway, no matter which approach we end up going with having the domains listed in redirects.conf move from squid to varnish is a blocker for fixing this bug.
Comment Actions
As this is blocked by the varnish work, can the project page be updated, specifically dates and RT tickets so those interested can follow along?
https://wikitech.wikimedia.org/wiki/Projects#Switch_.22text.22_to_Varnish
Also, upping importance given current WMF priorities and community desires.
Comment Actions
Bump. Any news? If we consider https important than bugs like this shouldn't be left open for almost 2 years.
Comment Actions
Daniel/Asher/Ryan: Is there an RT ticket for this? *seems* mostly in Ops/apache domain vs Platform, right?
Comment Actions
Oops, I forgot about/missed MZ's comment from way up there ^^
(In reply to comment #22)
Test cases
- https://mediawiki.org
- https://wikimedia.org
- https://wikipedia.org
- https://www.wikisource.org
- https://www.wikimediafoundation.org
RT tickets (I believe):
- RT #1668
- RT #2847
- RT #2783
Comment Actions
Apache doesn't handle this well, and we don't really want to do it in varnish. It's likely best for MediaWiki to handle this instead...
Comment Actions
In addition to those listed above (comment #22 and comment #23), please note that as of today, almost 2 weeks after HTTPS has been "forced" for all logged-in users:
also all get redirected to the non-secure "www.*" versions.
Comment Actions
(In reply to comment #50)
Apache doesn't handle this well, and we don't really want to do it in
varnish. It's likely best for MediaWiki to handle this instead...
Are you saying this is a non-ops issue, then? In my view, either Rob L. or Ken S. need to figure out who to assign this ticket to in order to get it resolved. Or we need to adjust the stated priority, but I'd prefer not to do that. If it's best for MediaWiki to handle this, it moves from ops to platform/core, I think.
Clarification about what you mean (or a rough sketch of how you think this should be implemented in Wikimedia's environment) would be wonderful. Firing up MediaWiki to do this type of high-level redirect seems kind of silly to me, but I don't really care about the implementation details. I'll trust your judgment.
Comment Actions
ksnider wrote:
I agree with Ryan here. Fixing this in Mediawiki should be the focus, otherwise we have to mangle the redirect in apache/varnish. Better if the wiki could return the HTTPS redirect directly instead.
If for some reason this isn't feasible, then it'll be on us to find a workaround.
Thanks.
Comment Actions
We had a brief in-office discussion, and the workaround we would like to propose for this is to change the 301 redirect to a 302 redirect in Gerrit change #13293, and remove the 'Header set Vary "X-Forwarded-Proto"'. That should disable caching for the redirect itself without generally disabling or splitting the cache. The downside is that the redirect won't be cached at all, but we're hoping that we don't get too much traffic on this particular set of redirects.
Longer term, we can potentially move this into MediaWiki (perhaps as part of HipHop), but we'd like to avoid doing this too hastily.
Comment Actions
(In reply to comment #54 by Rob Lanphier)
We had a brief in-office discussion, and the workaround we would like to
propose for this is to change the 301 redirect to a 302 redirect in Gerrit
change #13293, and remove the 'Header set Vary "X-Forwarded-Proto"'.
Does any specific MW core developer plan to work on this?
Comment Actions
(In reply to comment #55)
Does any specific MW core developer plan to work on this?
It looks like Rob is proposing to fix it on the Apache/ops side. That Gerrit is in operations/apache-config.
Comment Actions
(In reply to comment #56)
It looks like Rob is proposing to fix it on the Apache/ops side. That Gerrit
is in operations/apache-config.
In the short-term, yes. But Ryan L, Ken S., and Rob L. suggest on this bug that the long-term solution is for MediaWiki to handle these types of redirects. I think Andre may have been asking who might eventually handle that portion.
I'll repeat what I wrote earlier this year:
Clarification about what you mean (or a rough sketch of how you think this
should be implemented in Wikimedia's environment) would be wonderful. Firing
up MediaWiki to do this type of high-level redirect seems kind of silly to me,
but I don't really care about the implementation details.
Comment Actions
Greg set this ticket to highest priority three months ago, so I wonder if that still reflects priorities.
Comment Actions
(In reply to comment #58)
Greg set this ticket to highest priority three months ago, so I wonder if
that still reflects priorities.
It obviously hasn't been Platform's Highest priority for the past 3 months, wrongly or rightly. :/
I'll reassess current situation and priorities amongst Platform soon. Sorry.
Comment Actions
Bug 51700 was an issue we had with http or https redirect being cached, i.e. not varying on X-Forward-Protocol. Squid did not cache the redirect so that wasn't an issue.
The related fix is https://gerrit.wikimedia.org/r/#/c/75583/ "Fix up Vary headers on 30x redirects from Apache".
Comment Actions
Change 96438 had a related patch set uploaded by Tim Starling:
Generate redirects.conf
Comment Actions
(In reply to comment #22, comment #23, and comment #51)
Test cases
- https://mediawiki.org
- https://wikimedia.org
- https://wikipedia.org
- https://www.wikisource.org
- https://www.wikimediafoundation.org
- https://wiktionary.org
- https://wikiquote.org
- https://wikinews.org
- https://wikivoyage.org
- https://dk.wikipedia.org
- https://wikibooks.org
- https://textbook.wikipedia.org
- https://aa.wikiversity.com
- https://quote.wikipedia.org
- https://wikimania.wikimedia.org
This issue isn't fully fixed.
CORRECT (HTTPS --> HTTPS):
$ curl -Is "https://mediawiki.org" | grep Location
Location: https://www.mediawiki.org/
$ curl -Is "https://www.wikimediafoundation.org" | grep Location
Location: https://wikimediafoundation.org/
$ curl -Is "https://dk.wikipedia.org" | grep Location
Location: https://da.wikipedia.org/
(Though I'm not sure why dk redirects to da... assuming that's deliberate.)
$ curl -Is "https://quote.wikipedia.org" | grep Location
Location: https://en.wikiquote.org/
$ curl -Is "https://textbook.wikipedia.org" | grep Location
Location: https://www.wikibooks.org/
$ curl -Is "https://wikimania.wikimedia.org" | grep Location
Location: https://wikimania2014.wikimedia.org/
INCORRECT (HTTPS --> HTTP):
$ curl -Is "https://wikimedia.org" | grep Location
Location: http://www.wikimedia.org/
$ curl -Is "https://wikipedia.org" | grep Location
Location: http://www.wikipedia.org/
$ curl -Is "https://www.wikisource.org" | grep Location
Location: http://wikisource.org/
$ curl -Is "https://wiktionary.org" | grep Location
Location: http://www.wiktionary.org/
$ curl -Is "https://wikiquote.org" | grep Location
Location: http://www.wikiquote.org/
$ curl -Is "https://wikinews.org" | grep Location
Location: http://www.wikinews.org/
$ curl -Is "https://wikivoyage.org" | grep Location
Location: http://www.wikivoyage.org/
$ curl -Is "https://wikibooks.org" | grep Location
Location: http://www.wikibooks.org/
PROBABLY IRRELEVANT:
$ curl -I "https://aa.wikiversity.com"
curl: (51) SSL peer certificate or SSH remote key was not OK
Comment Actions
Change 106109 had a related patch set uploaded by Jeremyb:
final (I hope!) fix for protorel redirects
Comment Actions
(In reply to comment #65)
(In reply to comment #63)
Please use curl -vs | grep rather than curl -Is
(make that curl -vs 2>&1 >/dev/null | grep)
Comment Actions
(In reply to comment #66)
(In reply to comment #65)
(In reply to comment #63)
Please use curl -vs | grep rather than curl -Is(make that curl -vs 2>&1 >/dev/null | grep)
Why?
Comment Actions
(In reply to comment #67)
Why?
Most UAs encountering these redirects will be sending GETs. Your tests used HEAD. Why not use what most UAs will be using?
Comment Actions
(In reply to comment #68)
Most UAs encountering these redirects will be sending GETs. Your tests used
HEAD. Why not use what most UAs will be using?
Shouldn't the response header for a HEAD request be identical to that for a GET request?
Comment Actions
(In reply to comment #69)
Shouldn't the response header for a HEAD request be identical to that for a
GET
request?
Probably the same most of the time. Not saying HEAD testing is worthless but the first thing to test should be something that more closely approximates actual usage in the wild.
Comment Actions
As I suggested on IRC, I think this is an area where defined test cases would be very helpful, arguably essential.
Comment Actions
For the Toolserver redirects to come, I wrote Gerrit change #108467 if that's useful for this bug.
Comment Actions
(In reply to Gerrit Notification Bot from comment #64)
jeremyb: Patch should be split into several patches - anybody willing to do this?
Greg: You set this to highest priority half a year ago. Is the importance still correct?
Comment Actions
Lowering prio (thanks for the ping, Andre). Letting Jeremy/Ops take it from here. Chris Steipp can review if needed.
Comment Actions
Now fixed, based on testing the prior examples from this thread, plus a couple extras.
aa.wikipedia.com still doesn't work, but that's really bug 40998.
Reopen if you find one that still has the issue.
Comment Actions
(In reply to Matthew Flaschen from comment #76)
aa.wikipedia.com still doesn't work, but that's really bug 40998.
I meant to say aa.wikiversity.com (the one mentioned earlier), but actually neither work.
Comment Actions
sumanah wrote:
Thanks to everyone who raised or worked on this issue - you have helped keep our users safer.