[stats] Monitor amount of messages delivered by mail relays (for enotif and everything)
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	Nemo_bis
	Dec 2 2013, 10:00 PM

Description

As mentioned on bug 52915, we don't have any stats on how many emails we're sending from MediaWiki or otherwise, we have no idea of the impact our choices have though it would of course be quite important to assess it e.g. to have at least a slight idea of how likely we are to be considered spammers. As written elsewhere:

Note that we used to have some very aggregate statistics, made by Mark I believe, on the number of mails sent by our mail relays. They died some years ago though, we only have things like
https://bugzilla.wikimedia.org/show_bug.cgi?id=5505#c2
https://bugzilla.wikimedia.org/show_bug.cgi?id=28026#c55

Restoring mail relay statistics should be easier than proper eventlogging and could give some general clue, but it's just a guess I'm making.

Ori suggests we could just use whatever command to get the numbers and then send them to ganglia as we do for the job queue stats. mutante points to http://linux.die.net/man/8/eximstats which seems to be part of the standard packages http://packages.ubuntu.com/precise-updates/amd64/exim4-base/filelist

Version: unspecified
Severity: enhancement
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=56414

Details

Reference: bz57890

Related Objects

Mentioned In: T58414: Get mail relay out of Yahoo! blacklist: apply to Yahoo for whitelisting bulk mail

Event Timeline

• bzimport raised the priority of this task from to Medium.Nov 22 2014, 2:40 AM

• bzimport added a project: WMF-General-or-Unknown.

• bzimport set Reference to bz57890.

• bzimport added a subscriber: Unknown Object (MLST).

Nemo_bis created this task.Dec 2 2013, 10:00 PM

Turns out we already have a collect_exim_stats_via_gmetric script used in the role misc::fundraising::mail, see https://ganglia.wikimedia.org/latest/?r=year&cs=&ce=&c=Fundraising+eqiad&h=aluminium.wikimedia.org&tab=m&vn=&hide-hf=false&mc=2&z=medium&metric_group=ALLGROUPS_|_process_|_network_|_memory_|_load_|_disk_|_cpu

So hopefully it will be enough to add this same cron to the other mail server(s).

Change 101117 had a related patch set uploaded by Nemo bis:
Enable collect_exim_stats_via_gmetric cron for mail relay

https://gerrit.wikimedia.org/r/101117

Change 101117 merged by Alexandros Kosiaris:
Enable collect_exim_stats_via_gmetric cron for mail relay

https://gerrit.wikimedia.org/r/101117

Thanks akosiaris! Now we have stats for sodium, but not for mchenry: why and what's the best way to fix it?
https://ganglia.wikimedia.org/latest/?r=day&cs=&ce=&c=Miscellaneous+eqiad&h=sodium.wikimedia.org&tab=m&vn=&hide-hf=false&mc=2&z=medium&metric_group=ALLGROUPS
https://ganglia.wikimedia.org/latest/?r=day&cs=&ce=&c=Miscellaneous+pmtpa&h=mchenry.wikimedia.org&tab=m&vn=&hide-hf=false&mc=2&z=medium&metric_group=ALLGROUPS

Then maybe someone can explain if it's normal that exim_queued_messages is over twice of exim_messages_out. :)

The reason sodium has the stats and mchenry doesn't is sodium has the class exim::roled which is where the script is executed, whereas mchenry lacks it.

Including the check in mchenry related classes will add the stats to it too.

https://gerrit.wikimedia.org/r/#/c/68584/ will solve this issue among others, matanya says, but it looks too big and slow a commit. I plan to submit a patch moving the stats to a new class in exim in mail.pp and then add the class to mchenry's site.pp and to exim::roled.

Change 110524 had a related patch set uploaded by Nemo bis:
Split exim stats to own class and add it to mchenry

https://gerrit.wikimedia.org/r/110524

Change 110524 merged by Alexandros Kosiaris:
Split exim stats to own class and add it to mchenry