The mission of the United State's Government's Consumer Product Safety Commission (CPSC) is to protect consumers from injury by products. It's ironic then that the CPSC is playing an unwitting role in most of the largest DDoS attacks seen on the Internet. To understand how, you need to understand a bit about how you launch a high volume DDoS.
Logo of the Consumer Product Safety Commission
Amplification
DDoS attacks are inherently about an attacker sending more traffic to a victim than the victim can handle. The challenge for an attacker is to find a way to generate a large amount of traffic. Launching a DDoS attack is a criminal act, so an attacker can't simply go sign up for large transit contracts. Instead, attackers find ways to leverage other people's resources.
One of the most effective strategies is known as an amplification attack. In these attacks, an attacker can amplify their resources by reflecting them off other resources online that magnify the level of traffic. The most popular amplification vector is known as DNS reflection.
DNS Reflection
We've written about DNS reflection attacks in detail before. The basics are that an attacker generates DNS requests from a network that allows source IP address spoofing. The attacker forges the victim's IP as the source of the DNS requests. The attacker sends these requests to DNS resolvers that aren't locked down and respond to requests from any network. The DNS resolvers receive the requests and then send the responses back to the spoofed IP of the victim.
This technique "amplifies" an attack because a small DNS request will generate a larger response. How much larger depends on the size of the DNS record that's being queried. And here's where the Consumer Product Safety Commission comes in.
#1 Among DDoS Attackers
You see, the CPSC’s DNS server will respond with a very large DNS record when sent a very small DNS query. This DNS query is only 65 bytes:
Substitute X.X.X.X for one of the 13 million currently open DNS resolvers and you'll get back a response that is 4,426 bytes. To generate 68Gbps of traffic to a victim requires only 1Gbps of requests with a spoofed source IP address to open resolvers for the CPSC.gov DNS record. That's an amplification factor of 68x.
Now, lots of people have large DNS records, but the CPSC.gov record is very large and particularly popular among attackers. Based on both data about the large volumetric attacks we see hitting CloudFlare, as well as data from various resolver honey pots we run, approximately 94% of the DNS reflection attack requests we see are for CPSC.gov. (The next most popular are httrack.com which is used by 2.9% of attacker requests and isc.org which is used by 0.5%.)
At one level there's nothing the CPSC can do to prevent attackers from using their DNS record to launch attacks. CPSC's resources aren't being queried directly by the attackers so they can't block the requests themselves. However, they could construct their DNS record more carefully to make it smaller and therefore less attractive to attackers.
To see how, we can walk through the CPSC DNS record entry by entry and make recommendations on how to reduce its size.
The CPSC's Ginormous Zone
Here’s the CPSC.gov's entire 4,426-byte DNS response which you get from running the ANY DNS query above against an open resolver (scroll to the left in the grey boxes below to see the full DNS record):
You may get the records back in a different order, but you should see something similar if you run the same ANY query.
Walking through the DNS zone file line-by-line you can see the mistakes that the CPSC has made that bloat its record and learn a bit about DNS and, in particular, DNSSEC.
The Good
The Good, The Bad, and The Ugly artwork by Billy Perkins
Let's start with the good. Here are the first eight records, which are pretty standard and can't be optimized much.
These first eight records are bread and butter DNS: SOA, A, AAAA, NS, MX, and TXT. If you want a quick refresher on these basic DNS records, read on. Otherwise, you can skip to the next section ("The Bad") to see where the DNSSEC mess begins.
The first record is the SOA record. SOA stands for Start of Authority. Walking through the record left-to-right, the number 18894 — which you'll see appears next to all records — is the TTL (Time to Live) at the moment for the particular resolver I queried. That number represents the seconds until the resolver needs to fetch another result from authoritative DNS server. If I queried the same resolver 10 seconds later, the number would be 18884 — 10 seconds less than it had been before. If you query a different resolver you'll likely get another number that is somewhere between 21600 (the max TTL the CPSC.gov has specified) and 0. When the number reaches 0, the resolver will query the authoritative name server, update its cache, and reset the TTL to 21600 — the maximum TTL specified for the record.
Continuing on the SOA record, next up is "IN" — which also appears on every DNS record in the CPSC zone. IN stands for "Internet Class." DNS predated the Internet, so there are other classes that are possible, but IN is the only class you'll see in common use today. After "IN" comes "SOA" which designates this record type.
The next entries are the contents of the SOA record. The first entry specifies the primary authoritative name server for the domain ("auth00.ns.uu.net."). The second is the email address of the name server's administrator ("hostmaster.uu.net." means hostmaster@uu.net is the name server's administrator's email). "994622" is the zone serial number, which other name servers use to make sure they're in sync with the primary authoritative name server. "1800" is the refresh interval, specifying the number of seconds before non-primary name servers should check to see if the zone has changed. "600" is the retry interval, specifying the number of seconds to wait if a refresh failed. "1728000" is the number of seconds that non-primary name servers can serve a zone if they've failed to reach the primary name server. In practice with modern DNS setups, none of these numbers end up having much of an impact.
The last number in the SOA record, however, is important. "21600" is the length of time in seconds a negative result should be cached by recursive resolvers. For example, if you query "12345.cpsc.gov" — a record that doesn't exist — then the resolver you query will store the fact that the record doesn't exist for 21600 seconds.
SOA records look complicated but, in practice, only the last number — specifying the number of seconds that a negative result should be cached — has an impact on modern DNS implementations.
The other records in this group of 8 are less cryptic than SOA so we can quickly work through how they work.
The A and AAAA records are "anchor" records which specify the IP addresses responsible for the domain. A records are for IPv4 addresses — in this case 63.74.109.2 — and AAAA records are for IPv6 addresses — in this case 2600:803:240::2. The two NS entries define the name servers responsible for the domain ("auth00.ns.uu.net" which is primary as specified by the SOA record, and "auth61.ns.uu.net" which is secondary).
The two MX records specify the mail servers responsible for email sent to the domain ("hormel.cpsc.gov" and "stagg.cpsc.gov" — someone at the CPSC seems to have a sense of humor as Hormel is the maker of the pork product spam and "Stagg" is a brand of chili also made by Hormel). The number "5" in both MX records is the weight of that mail server. The lower the number the more it will be preferred. Since these are equally weighted, email will be load balanced equally between the two.
Not much from the above could be optimized. The next record is special kind of TXT record known as a SPF record. It specifies what domains or IPs are allowed to send email on behalf of CPSC.gov. In this case, three IP addresses (63.74.109.6, 63.74.109.10 & 63.74.109.20), the MX records (Hormel & Stagg), and the A record for the subdomain (lists.cpsc.gov). The "-all" at the end means mail sent from other domains or IPs not specified here should fail. Alternatives could be "~all" which would soft-fail mail sent from other domains/IPs, or "+all" which would allow mail to be sent from any domain/IP — which would effectively render the SPF record useless.
There's a tiny optimization that CPSC.gov could make by listing one CIDR (63.74.109.0/24) rather than each of the three IPs. By doing so they'd save 15 bytes:
However, as you'll see, that's hardly worth the effort compared with other optimizations they could make.
The Bad
The Good, The Bad, and The Ugly artwork by Billy Perkins
The misconfiguration of the CPSC.gov DNS largely involves how they've setup DNSSEC. DNSSEC is the mechanism to sign DNS records and help prevent cache poisoning. Each record in the DNS zone needs to be signed. To accomplish this you need RRSIG records and a simple change in their configuration could cut the size of the DDoS attacks the CPSC is indirectly responsible for nearly in half.
The signatures for each record are included as RRSIG records. Here's an example of the RRSIG record signing CPSC.gov's SOA record (again, scroll to the left in the box to see the whole record):
Let's walk through this RRSIG record. Just like with the previous records "18894" is the TTL in seconds, "IN" stands for "Internet Class", and RRSIG is the type of DNS record. "SOA" here specifies the type of record this signature applies to. That's easy. However, after that, things get cryptic (both literally and metaphorically).
The "7" here specifies the signature algorithm being used. By the DNSSEC specification, the 7th zone signing algorithm is: RSASHA1-NSEC3-SHA1.
Here's a list of the 10 current algorithms (represented by 14 numbers, just to be confusing) that can be used for signing public DNS records along with the number — or range of numbers if the signatures can be of variable lengths — of bytes each signature generates:
RSA/MD5 / 128–512 bytes (deprecated)
[Not used for RRSIG records]
DSA/SHA1 / 41 bytes
[Reserved]
RSA/SHA-1 / 128–512 bytes
DSA-NSEC3-SHA1 / 328 bits
RSASHA1-NSEC3-SHA1 / 128–512 bytes
RSA/SHA-256 / 128–512 bytes
[Reserved]
RSA/SHA-512 / 128–512 bytes
[Reserved]
GOST R 34.10-2001 / 64 bytes
ECDSA Curve P-256 with SHA-256 / 64 bytes
ECDSA Curve P-384 with SHA-384 / 96 bytes
Algorithm 7 allows the signer to specify a key length between 128 and 512 bytes. The CPSC specifies a key length of 256 bytes, which, today, for RSA is considered secure. Unfortunately, that means they have a 256-byte signature for every RRSIG record. Compare that with Algorithm 13 which, for a higher level of security, would produce only a 64-byte signature — one fourth the size per RRSIG record.
Returning to the RRSIG record, the "2" (after the "7") represents the number of "labels" in the domain being signed. In this case, the domain is cpsc.gov, which is 2 labels deep. If it were www.cpsc.gov then it would be 3 labels. a.b.cpsc.gov would be 4 labels.
"21600" represents the TTL that was used when the signature was originally calculated. Since the TTL could be different for each resolver's response, the TTL value needs to be set to a default to ensure that the signature is calculated consistently.
The next two numbers are timestamps. The first (20160824030507) represents the time this signature will expire (24 August 2016 at 03:05:07 UTC). The second (20160817020507) represents the time this signature became valid (17 August 2016 at 02:05:07 UTC).
"53799" is a checksum on the DNSKEY that was used to sign this record. This checksum is used to more quickly find the DNSKEY that was used to sign this particular record when more than one key is returned. (More on DNSKEYs in the next section.)
After the checksum is "cpsc.gov." which is the root domain. This root domain is included regardless of the number of subdomains (e.g., cpsc.gov, www.cpsc.gov, and a.b.cpsc.gov would all have "cpsc.gov." listed as the root domain). This specifies where to find the DNSKEY records that could have signed this record.
Next up is the signature itself. In this case, the signature is encoded in base64 and 256 bytes long. The base64 encoding means that the 256 bytes turns into 344 on the wire.
Stepping back, the easy optimization here for the RRSIG record would be to pick a better signature algorithm. By choosing Algorithm 13, rather than Algorithm 7, the CPSC could save 192 bytes per RRSIG record. We've written previously about why we use Algorithm 13 for CloudFlare's DNSSEC implementation.
Doing so here would add up because there are ten RRSIG records in the CPSC's zone in order to sign each of the other records (e.g., A, AAAA, NS, MX, etc...).
Just by switching to a better encryption algorithm the CPSC could increase the security of their DNS zone and save 1,920 bytes per DNS response. That small change alone would cut the maximum possible DNS reflection attack possible with the CPSC's zone by 44 percent.
While that's the biggest savings, choosing a less optimal outcome can't be considered an actual mistake. Unfortunately, the CPSC's zone has some ugly mistakes as well.
The Ugly
The Good, The Bad, and The Ugly artwork by Billy Perkins
The RRSIG records discussed in the previous section need to be verified with a public key. These public keys are stored as DNSKEY records. For the CPSC, there are four DNSKEYs that each look like this:
Walking through the record they're similarly formatted to RRSIG records. Once again, 18894 is the TTL on the record, the record type is DNSKEY and that's followed by three numbers: 256, 3 and 7. These are the flags field, protocol, and algorithm.
The flags field is 16-bits wide representing a number of potential boolean flags. Only two bits of this 16-bit field are currently assigned for use with the rest reserved for the future. A value of 256 (or 0000000010000000 or only the 9th flag set to true) indicates that this DNSKEY holds a key that can be used to verify RRSIGs. The protocol requires a 16-bit number so there's no optimization possible here.
The protocol field is 3. That is the only valid value for this field in a DNSKEY record. As the RFC says:
Again, that seems a bit arbitrary but it's what the protocol requires so there's no optimization.
Finally, the algorithm field is 7 which indicates RSASHA1-NSEC3-SHA1. So, it should be an RSA public key formatted according to RFC3110.
Passing it through a base64 decoder and hexdump we can take a look inside:
Notice that the first four bytes are 03 01 00 01. That indicates an exponent of 65537 (03 means three bytes of exponent, 010001 in hex is 65537) which is very common for the RSA algorithm. The rest of the record consists of 256 bytes (1,024 bits). These 1,024 bits are the modulus part of the RSA key. Thus this is a 1,024 bit key.
Like with RRSIG, the CPSC could have better security and a shorter record if they specified a more efficient algorithm for their DNSKEY records. Using Algorithm 13 would save 192 bytes per DNSKEY, or 768 bytes across the 4 records.
But there's the rub: the CPSC doesn't need to have 4 DNSKEY records at all. Except during a key rollover, there should generally only be 2 DNSKEY records. In the CPSC's case, one of the DNSKEYs (id 59364) isn't signing anything. Another of the DNSKEYs (id 27793) is redundant. In other words, the CPSC includes 552 bytes of completely worthless data in every DNS record. Just eliminating these two useless records would cut the maximum size of a DNS Amplification DDoS attack using the CPSC's zone by almost 13 percent.
You can visualize the chain of how DNSSEC records are used to sign each other by using a tool provided by DNSVIZ. The diagram below shows how the two unused DNSKEYs don't actually sign any records.
A Better CPSC Zone
There are a handful of other minor optimizations. Add them all up and the CPSC Zone could be much smaller. At CloudFlare, we've worked to optimize the size and performance of our zone records. We used our automated zone generator on the CPSC and got the following zone file:
That is 1,389 bytes and close to as optimized as you could manually create. The actual response may be a bit larger because resolvers can add a 320-byte DS record. Even so, the optimized CPSC zone would be about one third the size without compromising any functionality and actually increasing security.
But we can do better than that. Unlike the CPSC's current DNS provider (Verizon/UU.net), CloudFlare has anti-DNS reflection protections in place. Specifically, we automatically upgrade from UDP to TCP when a DNS response is particularly large (generally, over 512 bytes). Since TCP requires a handshake, it prevents source IP address spoofing which is necessary for a DNS amplification attack.
In addition, we rate limit unknown resolvers. Again, this helps ensure that our infrastructure can't be abused to amplify attacks.
Finally, across our DNS infrastructure we have deprecated ANY queries and have proposed to the IETF to restrict ANY queries to only authorized parties. By neutering ANY, we've significantly reduced the maximum size of responses even for zone files that need to be large due to a large number of records.
Denouement
DNSSEC is complicated and it's easy to get wrong. Unfortunately, getting your DNSSEC configuration wrong creates a real potential harm to the rest of the Internet by making your domain's zone file into a potential weapon to be abused by attackers. That's why, in CloudFlare's implementation, we spent a significant amount of time to optimize our automatic DNSSEC configuration to be as efficient as possible while still being easy and free for all our customers.
As Clint Eastwood says in the final scene of the movie "The Good, the Bad, and the Ugly": "You see in this world there's two kinds of people, my friend, those with loaded guns and those who dig. You dig." Pardon the puns but… we hope the CPSC will fix their zone and unload the giant gun attackers have used to launch some of the biggest DDoS attacks on the Internet. When they do, it'll be a relief to "dig" their domain and get back a reasonably sized answer.
If the instructions above aren't enough on their own, we'd be happy to work with the CPSC to help them get their DNS record under control and ensure they are no longer inadvertently facilitating the Internet's largest DDoS attacks. Of course, when the CPSC does get its DNS issues fixed, undoubtedly attackers will find another misconfigured and bloated zone they can abuse to launch DNS amplification DDoS attacks. And, when that happens, we'll be happy to work with them too — until the whole Internet has the well-configured DNS foundation it needs to be safe and secure.