Hi,

Possible cause is if the send batches takes longer than the time between sends, so it is sometimes repeating. I will check the code, if this is the case we must be able to handle it better.

I don't think it could have been a CRON batching issue, but I found a couple of possible causes and fixed them...

1) If the MySQL record set ordering was not consistent when we were stepping through it. We now hard-specify the ordering. This is an unlikely cause for serious mismatches, but could explain a few miss-ends if DB writes are happening in parallel.

2) For large sets, we were not de-duplicating. This is probably the cause, if you sent to multiple targets. E.g. if a member is on two newsletters and both are targeted, it wouldn't de-duplicate the sends across different batches (basically).

Hi Chris,
I sent a new Newsletter out last night and have had several reports of people receiving the same email in triplicate - myself included mine all came at 7:27am.
The Newsletter went to all users, i.e. I only ticked the one box.
It also seems to be sending very slowly, the first one when at 9:21pm and some 19604 later it is still sending this morning (8:47am).
Please can you take a look - thanks.
Cheers
Ade

Hi Chris,
As an update the sending of the Newsletter generated 51533 emails, we only have just over 30000 users in total and fair number of them have opted out of the Newsletter emails. So we are sending far too many duplicates - interestingly those who got duplicates always received 3 copies very close together.
Cheers
Ade

Sorry for the issues.

I have looked again.

I can see you have CRON running every minute, and newsletter drip sending sending 100 per minute.

I calculate VERY roughly it was delivering at 1/3rd the rate it should.

If it is waiting for write locks on the database to be freed while reading a next batch to send to, or if there's a capacity issue drawing the data out of the database, it could end up re-sending the batch when another CRON scheduler comes in and runs in parallel. We delete the batches as soon as they are read, and reading should be pretty speedy (much less than a minute), but database write locks or capacity problems can really change that dynamic.

Potentially write locks could have been caused by a MySQL backup script. Or it could have just been some other kind of queuing at some level.

I've therefore added an explicit lock around the whole drip send CRON hook, and it only frees that lock up once it has read and deleted a next batch. Regardless of the exact issue I think it really must help. I don't like adding too many of these extra locks in normally as they make debugging a pain (stuck locks while stepping through code), but this is a special case.