Send bulk email with MailKit in C#

问题:

We are using MailKit to successfully send emails by creating a client, making a connection and sending a mail. Very standard stuff which works great as we receive a message in our application, we do something and send on an email.
However we want to be able to process 10's of thousands of emails as quickly as possible.
Our destination email server may be unavailable and therefore we could quickly backup messages therefore producing a backlog.

With MailKit, what is the best and quickwest way to process the mails so that they get sent as quickly as possible. For example at the moment each mail may be processed one after the other and if they take a second each to process it could take a long time to send 40000 mails.

We have been using a parallel foreach to spin up a number of threads but this has limitations. Any suggestions or recommendations would be appreciated.

Code sample added: CORRECTION, NEW CODE SAMPLE ADDED. This is much faster but I cannot get it to work creating a new connection each time. Exchange throws errors 'sender already specified'. This is currently sending around 6 mails per second on average.

            var rangePartitioner = Partitioner.Create(0, inpList.Count, 15);

            var po = new ParallelOptions { MaxDegreeOfParallelism = 30 };


            Parallel.ForEach(rangePartitioner, (range, loopState) =>
            {
                using (var client = new SmtpClient(new SlabProtocolLogger()))
                {

                    client.Connect(_appSettings.RelayAddress, _appSettings.RelayPort);
                    client.AuthenticationMechanisms.Remove("XOAUTH2");

                    for (int i = range.Item1; i < range.Item2; i++)
                    {

                        var message = _outboundQueueRepository.Read(inpList[i]).Load();
                        client.Send(message.Body, message.Metadata.Sender, message.Metadata.Recipients.Select(r => (MailboxAddress)r));

                        _outboundQueueRepository.Remove(inpList[i]);
                    };
                }

            });

回答1:

Correct me if I'm wrong, but it looks to me like the way this works is that the Parallel.Foreach is creating some number of threads. Each thread is then creating an SMTP connection and then looping to send a batch of messages.

This seems pretty reasonable to me.

The only advice I can give you that might optimize this more is if many of these messages have the exact same content and the same From address and that the only difference is who the recipients are, you could vastly reduce the number of messages you need to send.

For example, if you are currently doing something like sending out the following 3 messages:

Message #1:

From: [e-mail]
To: Joe The Plumber <[e-mail]>
Subject: We've got a new sale! 50% off everything in stock!

some message text goes here.

Message #2

From: [e-mail]
To: Sara the Chef <[e-mail]>
Subject: We've got a new sale! 50% off everything in stock!

some message text goes here.

Message #3:

From: [e-mail]
To: Ben the Cobbler <[e-mail]>
Subject: We've got a new sale! 50% off everything in stock!

some message text goes here.

Your code might create 3 threads, sending 1 of the messages in each of those threads.

But what if, instead, you created the following single message:

From: [e-mail]
To: undisclosed-recipients:;
Subject: We've got a new sale! 50% off everything in stock!

some message text goes here.

and then used the following code to send to all 3 customers at the same MimeMessage?

var sender = new MailboxAddress (null, "[e-mail]");
var recipients = new List<MailboxAddress> ();
recipients.Add (new MailboxAddress ("Joe the Plumber", "[e-mail]"));
recipients.Add (new MailboxAddress ("Sara the Chef", "[e-mail]"));
recipients.Add (new MailboxAddress ("Ben the Cobbler", "[e-mail]"));

client.Send (message, sender, recipients);

All 3 of your customers will receive the same email and you didn't have to send 3 messages, you only needed to send 1 message.

You may already understand this concept so this might not help you at all - I merely mention it because I've noticed that this is not immediately obvious to everyone. Some people think they need to send 1 message per recipient and so end up in a situation where they try to optimize sending 1000's of messages when really they only ever needed to send 1.

回答2:

So we found a wider ranging fix which improved performance massively in addition to the improvements we found with the Parrallel ForEach loop. Not related to MailKit but I thought I would share anyway. The way that our calling code was creating our inputList was to use DirectoryInfo.GetDirectories to enumerate over all the first in the directory. In some cases our code took 2 seconds to execute over the directory with 40k files in it. We changed this to EnumerateDirectories and it effecitvely freed up the mail sending code to send many many emails.

So to confirm, the Parallel loop worked great, underlying performance issue elsewhere was the real bottleneck.