I can't help but think we've got something backwards here. Wasn't the point of gerrit 118592 to always include them?

gryllida wrote:

Please see problem description, (1).

How to get "Биология" (and NOT 'Категория:Биология') appear in these results?

What Cirrus does now is always search for pages by their redirects but its always the target of the redirect that is returned and never the redirect itself. Cirrus thinks of redirects as attributes of the target of the redirect and ignores redirect pages themselves. Look at the redirects object in the json blob here: https://en.wikipedia.org/wiki/Barack_Obama?action=cirrusdump

The upshot is that when you search you can find the result via the redirect and it'll come back in the redirect field but it'll never come back as a title. Example:
https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&list=search&format=json&srsearch=O%27bama&srprop=snippet|titlesnippet|redirectsnippet|sectionsnippet&srlimit=10&srbackend=CirrusSearch

I'm honestly not sure what lsearchd does for this. Its similar to Cirrus so far as I can tell:
https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&list=search&format=json&srsearch=O%27bama&srprop=snippet|titlesnippet|redirectsnippet|sectionsnippet&srlimit=10&srbackend=LuceneSearch

but it doesn't seem to always work similarly to Cirrus. If it produced the right result for Wikilinker then it must be different somehow. Its a lot of code to read and I've read a lot of it but I don't recall reading this part.

In any case for now I think the simplest solution for wikilinker is to set srbackend=LuceneSearch to keep the old behavior. That'll certainly buy us a few months of continued working and its reasonably simple.

gryllida wrote:

Yeah, I think it's complicated and I'm sure we'd figure it out. :-)

srbackend=LuceneSearch gives HTTP timeout on Russian Wikinews:

http://ru.wikinews.org/w/api.php?action=query&list=search&srbackend=LuceneSearch&srlimit=5&srprop=&srredirects=1&format=xml&srsearch=биолог*
Истекло время ожидания HTTP-запроса. = HTTP request timed out.

Should I gather community consensus on re-enabling it or can you just do it? I can again file a new bug if necessary.

gryllida wrote:

I repeat: how can I re-enable LuceneSearch on this project?

Enabling Lucene as the default wouldn't change anything that srbackend can't already do.

Two answers to two different questions:

1. If LuceneSearch is timing out or failing in another way we'll need to fix it. At lest for the next few months. The link seems to be working now but I have no doubt that it was failing before. Its pretty difficult to debug it without breaking it worse. That's why we're moving away from it. If it fails again please update the bug or ping us on irc or something.

2. This kind of issue isn't going to make us switch LuceneSearch back to the primary for ruwiki or ruwikinews. We're totally willing to switch back if Cirrus hurts more then it helps but in this case you have a clear work around and you were relying on a behavior that no one else noticed. And the behavior just doesn't seem helpful outside of your use case.

Truthfully I've change Cirrus's behavior to match Lucene's quirks in the past even though it had a pretty nice audience and I'd do it again in this case too but what your asking for would require a huge architectural change to cirrus which just isn't worth it.

If/when we do have to make some huge change I'll keep this is mind so its an option but I honestly don't think I can do more than that at this point.

A (maybe bad) idea! What if we piped the list of redirects that match the query back through the api. You'd still get the non-redirect page back but it'd come with a redirect list.

gryllida wrote:

The link seems to be working now but I have no doubt that it was failing before. Its pretty difficult to debug it without breaking it worse.

Oops. Works now.

2. This kind of issue isn't going to make us switch LuceneSearch back to the primary for ruwiki or ruwikinews.

I am not asking for primary. I would like to have it as an /option/. At the time it was timing out, there was no such option.

A (maybe bad) idea! What if we piped the list of redirects
that match the query back through the api. You'd still
get the non-redirect page back but it'd come with a redirect list.

The way the wikilinker gadget works now, with LuceneSearch, is that it takes the first 3 results and chooses the shortest one. Such rewrite could be a threat to results relevancy or similarity.

Snippet from the gadget:

если в запросе было только одно слово, то выбираем самое короткое название из первых трёх результатов
чтобы для "Аглией" выдавалось "Англия", а не "Англиканство"
if ( requestTokens === 1 ) {

var resar = [];

for ( var j = 0; j <= 4; j++ ) {
    if ( typeof resp.query.search[j] !== 'undefined' && txt.substr( 0, 3 ).toLowerCase() === resp.query.search[j].title.substr( 0, 3 ).toLowerCase()) {
        resar.push( resp.query.search[j].title );
    }
}

resar.sort( compareStringLengths );

if ( typeof resar[0] !== 'undefined' ) {
    pageName = resar[0];
}

}

This could be, in theory, rewritten to pull a list of results + the redirects, and picking the shortest ones. But this would not guarantee the best result, as some of the redirect origins could be rather short but irrelevant. See:

lucene search:

1. [[biology]] (redirect to [[category:biology]])
2. [[category:biology]]
3. ...

cirrus search:

1. [[category:biology]] <- [[biology]] <- [[bio]] <- ...
2. [[biologists discover world's smallest orchid]] <- ...

With cirrus search, the script for "biology" would return "[[bio|biology]]", but it doesn't mean that such wikilink would be accurate.

I have need for "limit search results to non-redirects" and was directed to this phab. The use-case is when trying to resolve page-naming controversies (or more mildly, how to name new pages) I want to look at precedent. And that means the actual pages alone. Conceptually, it would be an option I could add to intitle:, so an additional token in the search-box or a checkbox in the advanced-search mode or an include/exclude-redirects toggle in the results page.

In T73491#7622857, @DMacks wrote:

I have need for "limit search results to non-redirects" and was directed to this phab. The use-case is when trying to resolve page-naming controversies (or more mildly, how to name new pages) I want to look at precedent. And that means the actual pages alone. Conceptually, it would be an option I could add to intitle:, so an additional token in the search-box or a checkbox in the advanced-search mode or an include/exclude-redirects toggle in the results page.

This seems like a valid feature request, but should really be opened in a new ticket.