#1893 - Newletter emails are sent in duplicate
-
By Lhasadreams
- Added
- 9 views
| Identifier | #1893 |
|---|---|
| Issue type | Minor issue (breaks specific functionality) |
| Title | Newletter emails are sent in duplicate |
| Status | Completed |
| Handling member | Chris Graham |
| Addon | General / Uncategorised |
| Description | Hi,
On vwgolfmk1.org.uk we have just sent out a May Newsletter to all of our members using "Send to site members", which shows 29,601. Looking in the database I can see that we have 30,335 members with 10 of them not having email addresses. So I would of expected a worse around 30,325 emails to be sent. I carbon copy all the emails sent and for the Newsletter, Composr sent 44,167 emails. You might ask why I even picked this up !! One of our committee members, hairyarse (Andy Day) reported that he received 3 copies of the same email to the same email address ([email protected]). I checked and he only has one account and his email address only appears once. So it looks like we are sending nearly a third more emails that expected. Please can someone take a look at why this is happening. Cheers Ade |
| Steps to reproduce | |
| Funded? | No |
The system will post a comment when this issue is modified (e.g., status changes). To be notified of this, click "Enable comment notifications".
Comments
Possible cause is if the send batches takes longer than the time between sends, so it is sometimes repeating. I will check the code, if this is the case we must be able to handle it better.
1) If the MySQL record set ordering was not consistent when we were stepping through it. We now hard-specify the ordering. This is an unlikely cause for serious mismatches, but could explain a few miss-ends if DB writes are happening in parallel.
2) For large sets, we were not de-duplicating. This is probably the cause, if you sent to multiple targets. E.g. if a member is on two newsletters and both are targeted, it wouldn't de-duplicate the sends across different batches (basically).
I sent a new Newsletter out last night and have had several reports of people receiving the same email in triplicate - myself included mine all came at 7:27am.
The Newsletter went to all users, i.e. I only ticked the one box.
It also seems to be sending very slowly, the first one when at 9:21pm and some 19604 later it is still sending this morning (8:47am).
Please can you take a look - thanks.
Cheers
Ade
As an update the sending of the Newsletter generated 51533 emails, we only have just over 30000 users in total and fair number of them have opted out of the Newsletter emails. So we are sending far too many duplicates - interestingly those who got duplicates always received 3 copies very close together.
Cheers
Ade
I have looked again.
I can see you have CRON running every minute, and newsletter drip sending sending 100 per minute.
I calculate VERY roughly it was delivering at 1/3rd the rate it should.
If it is waiting for write locks on the database to be freed while reading a next batch to send to, or if there's a capacity issue drawing the data out of the database, it could end up re-sending the batch when another CRON scheduler comes in and runs in parallel. We delete the batches as soon as they are read, and reading should be pretty speedy (much less than a minute), but database write locks or capacity problems can really change that dynamic.
Potentially write locks could have been caused by a MySQL backup script. Or it could have just been some other kind of queuing at some level.
I've therefore added an explicit lock around the whole drip send CRON hook, and it only frees that lock up once it has read and deleted a next batch. Regardless of the exact issue I think it really must help. I don't like adding too many of these extra locks in normally as they make debugging a pain (stuck locks while stepping through code), but this is a special case.