It should be noted that Gmane will (obviously) only work for public lists.

Lists like checkuser-l and oversight-l are currently entirely un-searchable. Bumping up the severity from enhancement to normal (as though anyone cares about the several level).

For a while we attempted to use htdig search integration; this sort of worked but:
a) The search quality was not very good
b) The UI was terrible
c) It took so long to reindex that it would no longer complete. :P

Additionally, we block the archives from public search engines via robots.txt to reduce requests for removal of archived posts (since it's a huge pain in the butt to remove a post and regenerate the pipermail archives, and tends to be fragile and break things).

Our public mailing lists are generally also mirrored to gmane and other services and are searchable there, but private lists currently have no search of their archives.

mike.lifeguard+bugs wrote:

Searching the archives is incredibly important for private lists like checkuser-l. Currently unless one has a copy of the archives it is next to impossible to find what you're looking for. Restoring the search for such lists should be considered reasonably urgent and important. For such private lists, horrid UI isn't really a concern. I suppose low-quality results is a concern, but I never had real difficulties there. WRT reindex time - if this is re-enabled only for selected private lists, would that issue be reduced to an acceptable level?

fvassard wrote:

There doesn't seem to be many alternative to htdig.
Did anyone look into Swish-e?
Seems to be the more popular way of incorporating searching.
Let me know, and if there are no objection, I will get it setup-ed.

(somewhat nice guide: http://wpkg.org/Integrating_Mailman_with_a_Swish-e_search_engine)

--Fred.

The main thing to keep in mind is that this is most needed for private-access mailing lists, in which case we need to ensure that access to the search is restricted to authenticated list members; from skimming the swish-e integration page I'm not clear how secure that is for this case.

I don't see why it *needs* to be slow to index new posts, with any software.
Reindex just the last month and merge the results from different months when showing to the user.

It would even be acceptable to filter by dates:
Results for your search of Foo from Jan 2009 to June 2009: 5 [link]
Results for your search of Foo on 2008: 1 [link]
No hits found on other dates.

mike.lifeguard+bugs wrote:

(In reply to comment #4)

Let me know, and if there are no objection, I will get it setup-ed.

Did you decide that Swish-e wasn't appropriate for use here, or is it simply on the backburner?

For the record, I contacted Barry Warsaw (lead developer of Mailman) about a week ago. According to him, there is no active development of a search feature for Mailman right now (he invited me to help the project as a developer, but I'm afraid I can't develop the extension all by myself).

The implications are:

1. Either, we should us an existing search plugin (but we need to make sure it's secure enough)
2. Or, we can develop our own secure extension (which we can then send to Barry as well, for inclusion in the next version of Mailman as a core feature)

(In reply to comment #8)

1. Either, we should us an existing search plugin (but we need to make sure

it's secure enough)

2. Or, we can develop our own secure extension (which we can then send to Barry

as well, for inclusion in the next version of Mailman as a core feature)

Or ditch Mailman completely, it does not deserve to be a holy cow.

Enhancement nowadays, preferably to fix in upstream Mailman, however I don't see this happen without volunteers' contributions.

This would really help alleviate some of T59246

FWIW, W3C uses https://www.w3.org/Search/Mail/Devel and http://www.hypermail-project.org/

And FWIW, http://wiki.list.org/HyperKitty is now the upstream Mailman3 tool for searching (and a lot more).