jhecking wrote:

Could someone please at least confirm whether the action=raw method of the index.php API is supposed to work with the variant parameter? If not then we will have to look into either doing the transcoding client-side or scrape the rendered HTML markup from the Wiki servers instead of using the raw markup. Both options don't look very good. :(

action=raw is by definition raw. No variant processing would be applied to output.

(In reply to comment #2)

action=raw is by definition raw. No variant processing would be applied to
output.

I said that initially as well. But considering the alternatives (scraping action=render HTML or translating variants on the client side, both of which are evil), I think it would be best to add a variant parameter to action=raw, however inconsistent that may be. If no variant parameter is supplied, action=raw will still return the raw wikitext straight from the DB/cache, so no one using the action=raw interface the way it currently works will notice the difference.

Variant processing is for output, not raw source text. Performing variant processing on unprocessed source code is meaningless and would simply corrupt the code.

Since Brion says it doesn't belong at the index.php level I think there are a few options:

• Do we really need raw+variant? The variant stuff may well interact with templates or other Wiki markup. Is there any reason people who want this can't get what they need from action=raw? What is the use case?
• If not index.php we could add support to api.php
• How independent is the variant processing code? Could we abstract a function for api.php that takes a string of one variant and transforms it to the other variant?
• What about something like {{variant:xxx}}? We have similar wiki-functions already.

jhecking wrote:

Hi Brion, Andrew,

Here is some background on the user case for which I think raw+variant is needed - if there is a better way to achieve the same please let me know.

We have integrated Wikipedia content into our mobile search product. Wikipedia has very relevant content on a very large number of typical search queries. However from the majority of the mobile devices that our users are using (particularly in emerging markets) this content is not easily accessible since there is no mobile specific version of the Wikipedia web site. So instead of linking directly to the Wikipedia article on <intl>.wikipedia.org we use the raw markup to render a mobile compatible version of the article within our mobile search product. We use the raw markup for this since we need to be able to render the content in different device specific markup languages. Currently we only do this for articles from en.wikipedia.org but we would like to apply this to other languages as well. However if we cannot get the raw markup in the relevant language variant we cannot do this.

Here are a few examples how the integration looks like:
Sample search results pages featuring Wikipedia content: http://us.m.yahoo.com/p/search?p=who+was+albert+einstein, http://us.m.yahoo.com/p/search?p=what+is+dynamite
Albert Einstein article in oneSearch: http://us.m.yahoo.com/p/search/wiki?displayurl=http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FAlbert_Einstein&.done=http%3A%2F%2Fus.m.yahoo.com%2Fp%2Fsearch%3Fp%3Dwho%2Bwas%2Balbert%2Beinstein

Hope this clarifies the intended use case. Let me know what your thoughts on this are.

Thanks,
Jan

Conversion applies to output, not to raw source code. Early conversion would corrupt the markup (particularly for Latin<->Cyrillic, Latin<->Arabic, etc variants).

jhecking wrote:

I'm not quite sure I can follow: How would the conversion corrupt the Wiki markup? The markup only consists of latin characters, right? Those wouldn't be affected by the conversion I assume. And if they are affected then how come the conversion does not affect the final output which is markup as well - HTML markup in this case.

I guess I have to take a closer look at how the variant parameter works for Cyrillic/Arabic languages. Maybe it works differently than for Chinese.

Thanks,
Jan

Template names, magic keywords, tag names, HTML fragments, bla bla bla.

jhecking wrote:

As far as I can see our alternatives are

1. scrape the HTML markup instead of the raw wiki markup,
2. use the raw markup and duplicate the whole transcoding logic on our servers.

Am I missing anything?

Thanks,
Jan

1. scrape the HTML markup instead of the raw wiki markup

This was always going to be easier anyway. From the URLs you posted earlier it looks like all you need is the plain text. There is plenty of existing free code to extract plain text from HTML. It's much easier than parsing wiki templates etc.