Some contribution/contacts are entered twice into Civi - generally 1-3 hours apart from one another. This behavior appears to have begun after the upgrade, and is possibly related to credit card payment processing.
See: http://wikimediafoundation.org/wiki/Special:ContributionHistory/en
-or-
do a search for contributions after august 1.
Version: wmf-deployment
Severity: major
After some investigation, it turns out that the problem is due to a change in how PayPal/Payflow handle transactions.
Originally, transactions that we received via PayPal would result in PayPal sending us a message about that transaction through a service we wrote to receive notification about transactions from PayPal (it's our 'IPN Listener').
Transactions received via Payflow were completely handled on our end - we only received transaction information by asking Payflow directly for it.
Now, Payflow transactions as well as PayPal transactions are sent from PayPal to our IPN listener. As a result, we were pulling in transaction information for the same transaction from two different sources, resulting in double transaction entry.
I am working on a stop-gap measure to deal with this while we sort out a better solution to these changes.
The duplicate transactions we're seeing in the queue are missing some of the data bits that we normally attach to transactions (like the essential 'contribution_tracking_id'). So until we figure out a better long-term solution of how to handle the changes to PayPal's infrastructure, I updated the code to ignore transaction messages in the queue that do not contain a 'contribution_tracking_id'.
The payment pipeline is back up, all outstanding contributions/comments that were in the queue are now in Civi and things appear to be running normally.
Marking as resolved.