Get mail relay out of Yahoo! blacklist: apply to Yahoo for whitelisting bulk mail
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Nemo_bis
	Oct 31 2013, 8:19 AM

Description

X-YahooFilteredBulk: 208.80.152.133

(from https://en.wikipedia.org/?oldid=579580929#WMF_notification_email_marked_as_spam_by_Yahoo )

According to several comments, this means that our mail is currently blacklisted by Yahoo!, so it's sent in the "Bulk" folder for users who have it enabled (if not to spam?). WMF needs to send a "Bulk sender application for whitelisting" http://help.yahoo.com/kb/index?page=content&id=SLN9042 but first their guidelines have to be respected: http://help.yahoo.com/kb/index?y=PROD_MAIL_ML&page=content&id=SLN3435

There are several things in there that we don't do; let's add each of them to blockers when we're sure there's something to do.

"Use email authentication such as DKIM": low-hanging fruit, let's do it as soon as possible (bug 56413).
"Be CAN-SPAM compliant": it's USA law, I assume we're respecting it. :) It seems very lenient, from a quick skimming we're ok. http://business.ftc.gov/documents/bus61-can-spam-act-compliance-guide-business
"Consistently manage your lists": no idea, but MediaWiki surely doesn't remove/disable addresses which bounce: I remember Brion mentioning this in some 2005 bug comment or something like that, I don't know if there's an open bug but I'd say it's something worth doing.
"Segregate your mail streams by IP address and/or domain": in theory we do, I can't judge if well enough.
"Using your domain consistently": no idea, is it consistent to send from wiki@wikimedia.org but then use @wikipedia.org, @mediawiki.org and so on to From headers etc.?
"Publish reverse DNS (PTR) records for your sending IPs": looks ok.

$ dig -x 208.80.152.133
;; ANSWER SECTION:
133.152.80.208.in-addr.arpa. 3552 IN PTR wiki-mail.wikimedia.org.

"Secure your mail servers": I hope so.
"Use common-sense settings": it would be nice to track this (Ori, do you know something?), but Echo sends only few thousands notifications per day on en.wiki (https://toolserver.org/~dartar/en/echo/ ) and standard enotif (talk and watchlist) on all wikis is probably only one or two order of magnitudes bigger, it's still a ridiculously low amount of emails, we're not DoS'ing Yahoo!. It's possible that we have peaks, for instance an edit to a heavily watchlisted pages could cause a peak of thousands enotifs in a single second, but I doubt it's relevant. When we enabled enotifwatchlist on all wikis, the increased network activity of mchenry wasn't even visible at all in ganglia.

Version: unspecified
Severity: major
See Also:
T60261: From: address defaults to mediawiki.org domain on all non-Wikipedia wikis
T59890: [stats] Monitor amount of messages delivered by mail relays (for enotif and everything)
T61731: mailman emails taking long time for delivery, getting stuck in sodium
T66795: Email server's DMARC config prevents users from sending emails via Special:EmailUser
T66818: Mitigate strict DMARC policy on the mailing lists

Details

Reference: bz56414

Related Objects
Search...

Status	Assigned	Task
Invalid	None	T99444 Valid email address was unconfirmed for temporary spam blacklisting
Resolved	None	T104871 Mails from tools are being marked as 'spam' by gmail
Invalid	None	T54915 HTML email notifications caught by spam filters
Declined	• chasemp	T61731 mailman emails taking long time for delivery, getting stuck in sodium
Resolved	• chasemp	T58414 Get mail relay out of Yahoo! blacklist: apply to Yahoo for whitelisting bulk mail
Resolved	01tonythomas	T48640 Add proper email bounce handling to MediaWiki (with VERP)
Resolved	01tonythomas	T71019 Review and deploy BounceHandler extension to Wikimedia wikis
Resolved	01tonythomas	T74683 Rename br_user column
Resolved	01tonythomas	T74684 Inconsistent returns in BounceHandler
Resolved	None	T74685 ApiBounceHandler improvements
Resolved	01tonythomas	T74686 BounceHandlerActions::handleFailingRecipient() SQL query
Invalid	None	T71098 Product review BounceHandler extension for deployement
Resolved	• csteipp	T71099 Security review BounceHandler extension for deployement
Resolved	01tonythomas	T71101 Architecture review for BounceHandler extension
Resolved	aaron	T71100 Performance review BounceHandler extension for Deployment
Resolved	01tonythomas	T75083 Use MW style timestamps in the bounce_records table
Resolved	None	T71621 Deploy BounceHandler extension to beta cluster
Declined	01tonythomas	T92876 Deploy BounceHandler to group1 wikis (non-wikipedias)
Resolved	01tonythomas	T92877 Deploy BounceHandler everywhere (adding in wikipedias)
Open	None	T64704 Account creation by email often winds up marked as spam
Resolved	None	T58413 DomainKeys Identified Mail (DKIM) for wikipedia.org (and other projects)
Invalid	None	T106664 Set up role accounts and feedback loops (FBL) with all providers
Resolved	herron	T168467 Improve lists.wikimedia.org DMARC compatibility

Event Timeline

• bzimport raised the priority of this task from to High.Nov 22 2014, 2:25 AM

• bzimport added a project: WMF-General-or-Unknown.

• bzimport set Reference to bz56414.

• bzimport added a subscriber: Unknown Object (MLST).

Nemo_bis created this task.Oct 31 2013, 8:19 AM

RT tracking: https://rt.wikimedia.org/Ticket/Display.html?id=6151

What's the status of the RT?

(In reply to comment #2)

What's the status of the RT?

No activity, and no owner set.

I have upped the importance, because these delayed emails are causing major issues, and mailman is stalling as a result. Just 2 days ago, when mailman kicked back up, I received a flood of almost 50 emails from various lists. Please get this fixed ASAP. I don't want to listen to music and have my email client make incoming mail noises 47 times during one song.

What's the status of the RT now?

(In reply to comment #4)

I don't want to listen to music and have my email client make incoming mail
noises 47 times during one song.

Whether it's IRC pings or SMS dings, a happier life is quieter things. :-)

Need to find out which exact guidelines are violated.

(Re comment 4: I rather recommend to fix your email client.)

Uhm, I hadn't realized this is blocking bug 59731. I'm sorry; resetting to high priority again.

Updates?

My email client flooded again just now and a few hours ago.

(In reply to comment #10)

My email client flooded again just now and a few hours ago.

https://ganglia.wikimedia.org/latest/?r=custom&cs=02%2F08%2F2014+23%3A00+&ce=02%2F09%2F2014+02%3A00+&c=Miscellaneous+eqiad&h=sodium.wikimedia.org&tab=m&vn=&hide-hf=false&mc=2&z=medium&metric_group=ALLGROUPS

the exim_queued_messages graph seems to support that.

I'm seeing this issue getting worse, and there is nothing more annoying than leaving for 15 minutes and coming back to a flooded inbox, containing newly received emails sent from as far back as 7 days ago. This is also blocking another bug where sodium tends to fail to send messages as a result of this. I have therefore upped this bug's severity to blocker.

Please read [[mw:Bugzilla/Fields#Severity]] before changes. Apparently mail is not lost, but just delayed.

(In reply to comment #13)

Please read [[mw:Bugzilla/Fields#Severity]] before changes. Apparently mail
is
not lost, but just delayed.

Well in that case restoring to what it was.

Any updates? I'm getting really close to unsubscribing.

(In reply to comment #15)

Any updates? I'm getting really close to unsubscribing.

Did you write to Yahoo!'s customer service too, out of curiosity? If you can do something without WMF, just do it!

I tries clicking contact us and I just went in circles.

You should see my ISP's! It's easy: https://io.help.yahoo.com/contact/index?page=contact&locale=en_US&y=PROD_MAIL_ML -> Sending and receiving messages (last) -> Unable to receive messages (third last) -> Email

Note however that nowadays it's easier to receive support from email providers on Twitter, I see they're very active there: https://twitter.com/YahooCare

You can also do some lobbying on their uservoice board; the only relevant topic I found is https://yahoo.uservoice.com/forums/210695-yahoo-mail/suggestions/5361666-final-solution-to-prevent-spam which proposes to whitelist wikipedia.org among other things.

And of course I assume you went through https://help.yahoo.com/kb/mail/SLN3517.html

On a related note, I just ran into German article about the problems of people with a Yahoo mail account on http://www.heise.de/newsticker/meldung/DMARC-Policy-Yahoo-killt-Mailinglisten-Mitgliedschaften-2168857.html (and Google Translate to English pretty much works).
For the records, Mailman 2.1.16 (which will be in Ubuntu Trusty) allows setting "FROM_IS_LIST=yes" to "replace the sender with the list address to conform with policies like ADSP and DMARC".

(In reply to Andre Klapper from comment #20)

On a related note, I just ran into German article about the problems of
people with a Yahoo mail account

That's a new thing though, technical explanation at http://www.ietf.org/mail-archive/web/ietf/current/msg87153.html

I currently study the Yahoo and AOL’s recent change to DMARC, and the problem is pretty different than the one described here, basically because this one doesn’t relay emails (as do a mailing list) but sends original mails.

Possibly it could be implemented DMARC(+DKIM+SPF) on wikimedia.org to improve delivrability for this bug, *BUT* this could lead to heavy consequences on other @wikimedia.org emails (like non-delivery) so it must be carefully thought before action.

(In reply to Nemo from comment #21)

(In reply to Andre Klapper from comment #20)

On a related note, I just ran into German article about the problems of
people with a Yahoo mail account

That's a new thing though, technical explanation at
http://www.ietf.org/mail-archive/web/ietf/current/msg87153.html

That was split to bug 64795.

Bug 56413 was fixed, so I'm told this is currently waiting for bug 46640 and mailman 2.1.17+ (for MediaWiki mailer's and mailman's IPs respectively).

Bug 70930 has been marked as a duplicate of this bug. ***

Nemo_bis added a project: acl*sre-team.Dec 3 2014, 10:29 AM

Nemo_bis set Security to None.

Dzahn unsubscribed.Dec 22 2014, 2:04 PM

This is over a year old with no assignee. I am setting to normal priority. Please reassess when resources are available.

01tonythomas closed subtask T48640: Add proper email bounce handling to MediaWiki (with VERP) as Resolved.Mar 30 2015, 5:06 PM

SamB added a project: MediaWiki-Email.Mar 31 2015, 7:28 PM

jeremyb edited projects, added Mail; removed MediaWiki-Email, WMF-General-or-Unknown.Apr 1 2015, 6:54 PM

Aklapper mentioned this in T121744: Mailing list emails bouncing due to DMARC policy violation.Dec 17 2015, 11:45 AM

Krenair merged a task: T121744: Mailing list emails bouncing due to DMARC policy violation.Jan 13 2016, 1:41 AM

Krenair added subscribers: Aklapper, FastLizard4, StudiesWorld.

Qgil unsubscribed.Jan 13 2016, 3:01 AM

Reedy mentioned this in T123068: Emails sent via EmailUser function not working for user on en.wp.Jan 20 2016, 10:53 PM

Reedy mentioned this in T130723: Specific user with Yahoo address cannot use "email this user" function (unsubscribed after 5 bounces).May 31 2016, 9:53 AM

Aklapper mentioned this in T136468: Cannot send any email via "Email this user" on ru.wikipedia.org.May 31 2016, 10:10 AM

czar mentioned this in T134886: Special:EmailUser emails sometimes not delivered due to bounce.Jun 9 2016, 5:12 PM

Aklapper mentioned this in T66795: Email server's DMARC config prevents users from sending emails via Special:EmailUser.Nov 17 2016, 11:00 PM

Liuxinyu970226 subscribed.Dec 9 2016, 2:20 PM

As far as I can see everything is ready here and T66795#2867163 should reduce bounces to/from Yahoo (although it was arguably their fault, not ours); can ops please apply for said whitelisting? Looks like it's now called "reputation review" and the form is linked from https://help.yahoo.com/kb/SLN24439.html#cont3

• OldUser02 awarded a token.Mar 11 2017, 11:40 AM

• OldUser02 subscribed.

FYI, Yahoo! is blocking replies from OTRS:
https://otrs-wiki.wikimedia.org/w/index.php?title=Caf%C3%A9&oldid=74130#Yahoo.21_Mail_is_blocking_our_replies

Liuxinyu970226 updated the task description. (Show Details)Mar 11 2017, 12:41 PM