DNSSEC Error Caused NASA Website To Be Blocked

The hazards of early DNSSEC adoption: A misconfiguration in NASA’s Domain Name System Security Extensions (DNSSEC) implementation on its website caused Comcast’s network to block users from the site last week.

This is a glaring example of the difficulties in today’s mostly manual process of configuring DNS servers to support the new security protocol that prevents attacks that redirect users to malicious websites. The DNSSEC protocol basically ensures DNS entries remain unchanged in transit and are digitally signed to ensure their authenticity.

NASA had incorrectly signed DNSSEC in its implementation of the new security protocol last week, causing Comcast’s newly DNSSEC-enabled service to automatically block access to the site. Comcast earlier this month became one of the first major ISPs in North America to fully run the DNSSEC protocol as part of its services.

By pure coincidence, NASA's website woes occurred the day part of the Web went dark in protest of controversial anti-piracy legislation, leading some users and pundits to inaccurately speculate this was Comcast’s way of protesting the government-based bills. Far from it: Turns out Comcast's newly deployed DNSSEC service did the right thing by detecting an invalid digital signature on NASA’s DNSSEC-signed domain, and then blocking users from accessing what appeared to be a potential security threat.

Jason Livingood, vice president of Internet Systems Engineering for Comcast Cable Communications, says Comcast studied the problem and found it had to do with a domain-signing error. Comcast worked with NASA to quickly remediate it, and it wasn’t the first signing error the ISP has seen: "We’ve seen this same thing a few times before [elsewhere]," he says.

Comcast, with input from NASA, published a detailed report on the issue yesterday (PDF) as a way to help other early DNSSEC adopters.

NASA had failed to conduct a double-signing process. It signed "nasa.gov" with a new key pair, but the upstream chain of the ".gov" domain had an older key pair that the agency didn’t use in the signing process. That was all it took for Comcast’s DNSSEC to detect a problem with the NASA site when its users tried to visit.

Livingood says his company detected other domains in .gov yesterday with the same problem, but it’s unclear so far whether this is related to NASA’s issue or these are new cases. "This happens around key rollover time,” he says. "This is an area we’re focused on, and we will continue to conduct periodic analysis when we observe failures and try to publish them" to help other early DNSSEC adopters, he says.

[Researcher points to fundamental problems in SSL and DNSSEC, and says it's time for users to take control of trust. See Time For A Better Web Of Trust?]

DNSSEC expert Dan Kaminsky says this is an example of the kinds of short-term troubles with the manual processes required in early adoption of the technology. But these manual DNSSEC processes will eventually be fully automated, which will eliminate these types of issues, he says.

Livingood says it’s not surprising that these problems are cropping up as organizations roll out DNSSEC. "DNSSEC is relatively new for a lot of domains," he says. "Maybe they’re doing their first rollover, and it’s probably a process or automation [issue]," he says.

Cricket Liu, vice president of architecture at Infoblox, says it’s telling that a scientific organization could err in its DNSSEC cutover. “If even the rocket scientists can't get it right, what about the rest of us?” Liu quips. “To me, this really reinforces the argument that DNSSEC is so complex that it requires automation.”

But key-signing key (KSK) rollovers are not easy to automate, he notes. “KSK rollovers are difficult to automate completely because one step in the rollover -- submitting your [delegation signer] record to your parent zone -- isn't standardized. But even providing guidance to the administrator as to what to do and when to do it would have been valuable in this case,” Liu says.

As a pioneering adopter of DNSSEC, Comcast basically took the heat here. Liu says that’s unfortunate: “It's too bad that Comcast took the blame for this failure. In my opinion, they should be commended for deploying DNSSEC ahead of most ISPs. This sort of failure will happen occasionally as new zones are signed and administrators learn the ins and outs of managing DNSSEC-signed zones,” Liu says.

NASA had not responded to press inquiries as of this posting.

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.