Now that all uploaded files are served by Varnish (as of late April/early May), we're seeing major issues with streaming of large video files. This is likely due to limited streaming support in Varnish in the 3.0.2 release we're running:

https://www.varnish-software.com/blog/http-streaming-varnish

"Varnish 3.0 has some support for streaming objects, but I can't say that I'm very happy with the semantics. It only allows one client to stream the object and others are placed on hold while the object is being streamed. If this client is a 3G phone downloading a 100 megabyte object other clients are likely to give up waiting and go and do something else."

Mark has enabled this limited streaming support, currently for files > 100 MB.

Whether it's due to these or other limitations, files > 100 MB are currently very unreliably playable, especially on first play, and still sometimes return 503 errors as well.

It takes many seconds to successfully initiate a play of files such as these, or they may fail completely:

http://upload.wikimedia.org/wikipedia/commons/b/b5/Monthly_Metrics_Meeting_March_1%2C_2012.ogv

http://upload.wikimedia.org/wikipedia/commons/e/ec/Monthly_Metrics_Meeting_April_5%2C_2012.ogv

http://upload.wikimedia.org/wikipedia/commons/f/f3/WMF_Analytics_Day_-_HBase.ogv

We'll need to investigate fuller solutions, whether it's backporting the current experimental streaming branch for upload hosts ( https://github.com/mbgrydeland/varnish-cache-streaming ), reverting to Squid, further changing config, etc. We'll also need to make sure these issues aren't exacerbated further with SWIFT.

Version: unspecified
Severity: major