Too many DNS lookups in an SPF record

问题:

My website needs to send out emails with Google Apps, SendGrid and MailChimp services. Google Apps is used to receive and read incoming email to my domain.

I need to set the SPF record for my domain. The following is syntactically correct (not sure about A and MX tokens):

"v=spf1 a mx include:_spf.google.com include:servers.mcsv.net include:sendgrid.net ~all"

But if I test it with http://www.kitterman.com/getspf2.py I get

PermError SPF Permanent Error: Too many DNS lookups

Similar problem as http://www.onlineaspect.com/2009/03/20/too-many-dns-lookups-in-an-spf-record/

How can I optimize/rewrite my SPF record?

回答1:

So, I've never had to do this before, but based on the article you sent over, this is what I came up with.

We started with:

v=spf1 a mx include:_spf.google.com include:servers.mcsv.net include:sendgrid.net ~all

We get 10 total lookups before we throw the Too many DNS lookups error:

  2 (Initial TXT & SPF Lookups)
  2 (a & mx Lookups)
  1 (_spf.google.com)
  1 (servers.mcsv.net)
 +1 (sendgrid.net)
 -----------------
  7 Lookups

So without even following the included SPF records, we have 7 lookups.

---

Now, let's dive a level deeper.

1. _spf.google.com

The google SPF record evaluates to:

v=spf1 include:_netblocks.google.com include:_netblocks6.google.com ?all

Each of which resolve to the following values:

# _netblocks.google.com
v=spf1 ip4:216.239.32.0/19 ip4:64.233.160.0/19 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:209.85.128.0/17 ip4:66.102.0.0/20 ip4:74.125.0.0/16 ip4:64.18.0.0/20 ip4:207.126.144.0/20 ip4:173.194.0.0/16 ?all

# _netblocks6.google.com
v=spf1 ip6:2607:f8b0:4000::/36 ip6:2a00:1450:4000::/36 ?all

So google gives us 2 more lookups, bringing the total up to 9 Lookups.

2. servers.mcsv.net

Mailchimp is a bit of a doosey because it adds a whole 3 extra lookups:

v=spf1 include:spf1.mcsv.net include:spf2.mcsv.net include:spf.mandrillapp.com ?all

I would imagine that depending on what you're sending through Mailchimp, you might be able to remove one or two of these records (but you'll have to evaluated that yourself).

Anyway, those resolve to the following:

# spf1.mcsv.net
v=spf1 ip4:207.97.237.194/31 ip4:207.97.238.88/29 ip4:207.97.240.168/29 ip4:69.20.10.80/29 ip4:69.20.41.72/27 ip4:74.205.22.1/27 ip4:69.20.90.0/26 ?all

# spf2.mcsv.net
v=spf1 ip4:204.232.163.0/24 ip4:72.26.195.64/27 ip4:74.63.47.96/27 ip4:173.231.138.192/27 ip4:173.231.139.0/24 ip4:173.231.176.0/20 ip4:205.201.128.0/24 ?all

# spf.mandrillapp.com
v=spf1 ip4:205.201.136.0/24 ip4:205.201.137.0/24 ?all

This brings us up to a total of 12 Lookups (Which is two over the limit already).

2. sendgrid.net

SendGrid ends up being the fewest number of additional lookups for us.

v=spf1 ip4:208.115.214.0/24 ip4:74.63.202.0/24 ip4:75.126.200.128/27 ip4:75.126.253.0/24 ip4:67.228.50.32/27 ip4:174.36.80.208/28 ip4:174.36.92.96/27 ip4:69.162.98.0/24 ip4:74.63.194.0/24 ip4:74.63.234.0/24 ip4:74.63.235.0/24 include:sendgrid.biz ~all

So the only additional lookup here is sendgrid.biz, which evaluates to:

v=spf1 ip4:208.115.235.0/24 ip4:74.63.231.0/24 ip4:74.63.247.0/24 ip4:74.63.236.0/24 ip4:208.115.239.0/24 ip4:173.193.132.0/24 ip4:173.193.133.0/24 ip4:208.117.48.0/20 ip4:50.31.32.0/19 ip4:198.37.144.0/20 ~all

This brings our grand total up to 14 lookups.

---

So our grand total is 14 Lookups. We need to get that down to 10. I've outlined a couple of options below, you may need to use more than 1 of them to get it down.

1. Directly include some of the redirected spf records. Now that we know which servers the spf records redirect to, you could cut out the middleman and include them directly. Note: If any of the services end up changing their SPF records, you'll have to go through the process of updating yours manually.

2. Remove some of the services that you're using. Not sure what your use case is for having all of these services, but there's definitely some overlap that you might be able to use. For instance, SendGrid supports (1) transactional outgoing mail, (2) newsletter / marketing emails, and (3) incoming mail. So there may be some reducible redundancy.

3. Remove the MX record if it is redundant. Depending on your setup, the MX lookup can be redundant.

Hope this helps!

回答2:

Swift's answer is excellent.

A technique which is not mentioned above is to look at whether separate subdomains with their own SPF records can be used for systems that send mail via these different routes.

E.g. if the domain is example.com, have google apps send from addresses like [e-mail]. Then there can be a SPF record for gapps.example.com which includes _spf.google.com, and _spf.google.com can be removed from the main example.com SPF record , which reduces lookups by 3.

回答3:

Have a look at SPF-tools* which help with reassembling the SPF record from the original one that uses includes into a static one containing merely ip4 and ip6 fields. It can be easily coupled together with locally-run DNS server or hosted DNS service using their API to keep everything in sync with the upstream includes.

*I am the author (now along with other contributors) and it is open source under Apache 2.0 license.

回答4:

We explored flattening the SPF records into IPs as well a creating subdomains. All of them seemed like a lot of work. We found a service from spfproxy.org which literally takes a couple minutes to setup. They basically mask the DNS lookups behind them using SPF macros. Not sure why more companies don't offer this.

回答5:

A few years ago I wrote hydrate-spf, a tool that looks up includes and merges the result into one giant record. As mentioned in the README, this approach isn't ideal - it removes the ability for your included domains to update their records. However, when you're bumping up against the allowed limit, it will solve the immediate problem, and can be kept somewhat maintainable through regular updates.

回答6:

The 10 lookup limit is a limit for DNS lookups. Flattening the SPF record to include less DNS lookups and substituting them for IPs (flattening) is a way to get around the limit.

You could do this manually, but then you have to update your SPF records every time one of the providers changes their IPs (which happens frequently).

The ideal solution is to use an SPF flattening service. This one is free for low volumes, or cheap for more than 500 emails/month. It regularly polls the SPF records you want to include for updated IPs.

Fraudmarc.com

Disclosure: I am not associated with this company and this is not a referral link

回答7:

This 10-DNS-lookup limit is imposed by SPF implementations to prevent DDoS attacks against the DNS infrastructure.

With DMARCLY's Safe SPF feature, you can lift the limit without rewriting your SPF record.