Page MenuHomePhabricator

Video thumbnails don't work (on live Commons?)
Closed, ResolvedPublic

Assigned To
None
Authored By
bzimport
May 4 2011, 4:17 PM
Referenced Files
F7772: 28810.png
Nov 21 2014, 11:33 PM
F7771: uwthumb4.png
Nov 21 2014, 11:33 PM
F7768: uwthumb1.png
Nov 21 2014, 11:33 PM
F7770: uwthumb3.png
Nov 21 2014, 11:33 PM
F7769: uwthumb2.png
Nov 21 2014, 11:33 PM

Description

Author: neilk

Description:
When an OGV video file is uploaded, the requested preview is supposed to be available at a URL like this (the convention is "mid-(filename).jpg", which I believe corresponds to the middle keyframe, extracted without any scaling).

http://commons.wikimedia.org/wiki/Special:UploadStash/thumb/d4zwo6jo2olm526ap1h6d2v1hmphn91.ogx/mid-d4zwo6jo2olm526ap1h6d2v1hmphn91.ogx.jpg

This file is not available where UploadStash expects it, even after several minutes, so this isn't a thumbnailer-flakiness thing.


Version: unspecified
Severity: normal
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=37036

Details

Reference
bz28810

Related Objects

StatusSubtypeAssignedTask
OpenNone
ResolvedNone

Event Timeline

bzimport raised the priority of this task from to High.Nov 21 2014, 11:33 PM
bzimport added a project: UploadWizard.
bzimport set Reference to bz28810.
bzimport added a subscriber: Unknown Object (MLST).

neilk wrote:

However, this seems to work just fine on commons.prototype, with the exact same file, and in fact a nearly identical URL, except for the hostname.

http://commons.prototype.wikimedia.org/wiki/Special:UploadStash/thumb/d4zwo6jo2olm526ap1h6d2v1hmphn91.ogx/mid-d4zwo6jo2olm526ap1h6d2v1hmphn91.ogx.jpg

(Note that InstantCommons is not an issue since I abandoned both files before publishing).

neilk wrote:

Since Ian is taking this over, I figured I should brain-dump everything I know.

So this is what we know about thumbnails for video

  • ogg thumbnails do work in Special:Upload, and after the file has been uploaded.
  • ogg thumbnails work on commons.prototype.wikimedia.org (and properly configured local machines) for everything, even Special:UploadWizard

So, what are the differences?

  • different ffmpeg config on the cluster? (unsure)
  • files are mounted in NFS on the cluster (definitely a big difference)

And what's different between what UploadWizard does and what everything else does:

  • we are doing something slightly unusual in that we are attempting to get a thumbnail for a file that is in the "temp" zone. This goes back to the definition of zones I told you about. Basically, a zone defines a mapping between a filesystem directory and a base URL. Also, there are some security characteristics associated with zones -- deleted and temp files are not normally served to most users.

the standard zones are

  • 'public' -- the usual thing
  • 'temp' -- files in the stash are here
  • 'deleted' -- files that have been "archived"
  • 'thumb' -- thumbnails

Note: the temp zone, is, if I recall correctly, not the same as the temporary directory that files are written into as PHP receives the upload. The 'temp' zone exists in NFS.

I had to do a lot of trickery to make thumbnails work for files in the "temp" zone and also be secure. We don't want people to be able to see thumbnails unless the file is published (which is the moment when presumably it has a good license). So, all thumbnails are public, but I hid the actual thumbnail URL from the user by streaming it through PHP. This is the real purpose of Special:UploadStash -- to stream files while hiding their scaler URL. Listing files is just a bit of functionality I threw in there for fun and debugging.

(I discovered later that img_auth.php (in the root of MediaWiki) does almost *EXACTLY* the same thing, for wikis which define some forms of security.)

ibaker wrote:

Peter is going to upgrade all the scalers either this week or next. He recommends that we see if moving them to the same OS release as all our dev environments fixes it before digging further.

pinging peter in IRC (and on this bug) to see if the image scalars are upgraded.

To confirm -- is the prototype configuration using on-demand thumbnailing, a separate image scaler subcluster, and the $wgUploadStashScalerBaseUrl configuration option?

Things seem to go through a wildly different code path in this configuration, where it performs some sort of manually-proxied HTTP request instead of loading the file from the filesystem.

IIRC img_auth.php is handled by having the proxy configuration send those requests to the image scalers directly, so in the MW code nothing very clever has to happen.

neilk wrote:

(In reply to comment #5)

To confirm -- is the prototype configuration using on-demand thumbnailing,

yes

a separate image scaler subcluster

no

, and the $wgUploadStashScalerBaseUrl
configuration option?

Okay, live, we use this in CommonSettings.php:

$wgUploadStashScalerBaseUrl = "$urlprotocol//upload.wikimedia.org/$site/$lang/thumb/temp";

We could use this with commons.prototype.wikimedia.org but that would just point to the same server again.

Things seem to go through a wildly different code path in this configuration,

Yup.

where it performs some sort of manually-proxied HTTP request instead of loading
the file from the filesystem.

Yes, if I understand the code, img_auth.php cannot be used with scaled images. It chops off any /\d+-px/ prefix. So you get the original no matter what you ask for, via the StreamFile interface.

The whole point here was to get thumbnails, in a way that nobody but the uploading user could access (since it wasn't published yet, hasn't been looked at by anybody).

So, we assume that the scaler has NTP access to the temporary file in the temp zone. Then we request a scaled version via HTTP, and then pass the data to the client.

(Note that the client theoretically could have done this request directly from their browser, but they don't know the tmpnam the server gave their file. That's what makes it reasonably secure.)

What could be simpler? :(

IIRC img_auth.php is handled by having the proxy configuration send those
requests to the image scalers directly, so in the MW code nothing very clever
has to happen.

I don't understand what you mean here.

py wrote:

all image scalers are now on ubuntu 10.04

Thehelpfulonewiki wrote:

So is this bug fixed?

(In reply to comment #8)

So is this bug fixed?

It doesn't look like this has been fixed. When I tried to upload [[File:Folgers.ogv]], I got a broken thumbnail. But, really, I'm not sure if that is a good test.

Thehelpfulonewiki wrote:

Reassign to default assignee per bug 37789

sumanah wrote:

Chris, can you reproduce this on 1.21wmf2 or git master?

I believe this can be closed, but I would like to confirm that I truly understand what the root issue is.

Using a version of UploadWizard from Oct 22 2012 on beta commons, I uploaded an ogv file http://commons.wikimedia.beta.wmflabs.org/wiki/File:Intelligent_risk_taking.ogv and saw correct thumbs in UW itself and on the resulting page in commons. Screen shots attached.

Created attachment 11234
thumbs

Attached:

uwthumb1.png (523×1 px, 45 KB)

Created attachment 11235
thumbs

Attached:

uwthumb2.png (569×1 px, 40 KB)

Created attachment 11236
thumbs

Attached:

uwthumb3.png (439×959 px, 44 KB)

Created attachment 11237
thumbs

Attached:

uwthumb4.png (684×1 px, 67 KB)

Created attachment 11240
UploadWizard with ogv and black box

I tried again. At least in UploadWizard it's the same black box as before. See screenshot. Might be specific to the file.

Attached:

28810.png (900×1 px, 154 KB)

It's browser-specific. The thumb is black for me in FF but normal in Chromium. I tried with both an ogv already on beta commons and also one new to beta commons.

  • This bug has been marked as a duplicate of bug 41403 ***
Gilles raised the priority of this task from High to Unbreak Now!.Dec 4 2014, 10:11 AM
Gilles moved this task from Untriaged to Done on the Multimedia board.
Gilles lowered the priority of this task from Unbreak Now! to High.Dec 4 2014, 11:22 AM