Ori suggested assigning to Faidon for triage/resolution.

It looks like Varnish is caching the 301 response under some circumstances:

< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.1.19
< Date: Mon, 27 Jan 2014 21:12:13 GMT
< Content-Type: text/html; charset=iso-8859-1
< Transfer-Encoding: chunked
< Connection: keep-alive
< Location: https://logstash.wikimedia.org/
< Vary: Accept-Encoding
< X-Varnish: 2005611465 2005610948
< Age: 102
< Via: 1.1 varnish
< X-Cache: cp1044 hit (450)

Maybe a cache-control header needs to be added when sending the 301 to switch protocols?

I think there are two possible solutions to this:

1. Send a Cache-Control header along with the 301 response to keep Varnish from caching the redirect
2. Add the X-Forwarded-Proto header to the hash inside Varnish
   
   sub vcl_hash { if (req.http.X-Forwarded-Proto) { set req.hash += req.http.X-Forwarded-Proto; } }

Can X-Forwarded-Proto by used with a Vary header?

(In reply to comment #4)

Can X-Forwarded-Proto by used with a Vary header?

*be used

Change 110393 had a related patch set uploaded by BryanDavis:
varnish: Add X-Forwarded-Proto to Vary on redirects

https://gerrit.wikimedia.org/r/110393

Change 110396 had a related patch set uploaded by Ori.livneh:
kibana: Set Vary: X-Forwarded-Proto on HTTP -> HTTPS 301s

https://gerrit.wikimedia.org/r/110396

Change 110396 merged by Ori.livneh:
kibana: Set Vary: X-Forwarded-Proto on HTTP -> HTTPS 301s

https://gerrit.wikimedia.org/r/110396

Change 110393 abandoned by BryanDavis:
varnish: Add X-Forwarded-Proto to Vary on redirects

Reason:
Ori's workaround in Id2cc29f911fa36805320cdb606a5da1226c9d230 is less yucky

https://gerrit.wikimedia.org/r/110393

Ori fixed this with a cool bit of Apache config:

RewriteRule ^/(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [R=301,E=ProtoRedirect]
Header always merge Vary X-Forwarded-Proto env=ProtoRedirect

This works around an annoying bit of esoteric Apache internals where headers are stripped during redirect generation and have to be reapplied later. See http://stackoverflow.com/a/3711110/8171 for a description of the underlying "feature" that is being addressed.