As I made my way up the long escalator from the ground floor of Caesar's Palace on the first day of Black Hat, I continued to wrestle with my agenda for the next few hours. I'd already made the tough decision to catch Ofir Arkin's promising NAC attack session rather than sit in on either of my second two choices: David Litchfield's database security discussion, and the VoIP hacking talk being conducted by David Endler and Mark Collier. The second slot that morning was much more troubling, and wouldn't you know it, I made the wrong choice. Dropping anchor at Hendrik Scholz's "SIP Stack Fingerprinting and Stack Difference Attacks" would have made life so much easier this week as I covered Cisco's recent spate of vulnerabilities, including the PIX problem Scholz slipped into his presentation at the end. Instead, I was elsewhere and missed being an eyewitness to one of Black Hat's biggest stories. Not to worry, the pieces are starting to come together.When I first heard that a Black Hat presenter had included information about a zero-day Cisco vulnerability in his presentation, my first reaction was to think that, in covering only 10 of the 70 or more sessions, I was bound to miss something. Then I marched over to Cisco's booth at the show and started asking questions. I was given a phone number to call, but ultimately I wasn't given much to work with (other than a handout covering Cisco's vulnerability disclosure policy).

No problem, I thought. I'll just check the CD that I'd been given by Black Hat with slides from most of the event's presentations. No luck. While Scholz's slides covering his SIP research were there, the all-important final slide was missing. This guy was good. Subsequent messages to Black Hat's event staff didn't yield any audio or video recordings of Scholz's session, although I (it being Vegas and all) would have wagered that someone had to have captured the moment, especially after security researcher Michael Lynn's magic moment at a Black Hat show a year ago, when he gave a presentation against the wishes of Cisco and Internet Security Systems, his employer at the time, that proved attackers could take over--rather than simply shut down--routers and switches running Cisco IOS.

So I went straight to the source. What do you know, Scholz was very responsive and helpful, all the while being careful not to provide enough information for anyone who might be thinking about creating a zero-day exploit against Cisco's PIX firewalls. The Freenet Cityline VoIP developer responded to one of my e-mails by stating that he didn't set out to find a Cisco vulnerability. "We discovered the bug while testing other applications," he wrote. "Based on the potential it could be important but as of now the testing did not show a big impact security-wise. Nonetheless incoming phone-calls were rejected which obviously is a show-stopper on a VoIP-installation."

The PIX issue is related to the way the firewall handles SIP traffic, Scholz said. As far as he can tell, the problem isn't related to parsing the message, but rather understanding what to do with it. "The bug shows that even a big company like Cisco has a hard time keeping up with the new VoIP standards and additional features," he added.

The way Scholz explained the situation to me, in order to allow VoIP to work behind network address translation devices and firewalls, these devices have to inspect the Application-layer traffic and "fix a few things every here and there." This usually results in opening up ports to allow media, such as audio files, to flow between the VoIP client on, for example, a company network and some point outside the company network.

Scholz told me that his Black Hat presentation wasn't inspired by Lynn's, after which Cisco sued the security researcher (although the suit was eventually dropped). Lynn made enough of an impression at the show that he was later hired by Cisco rival Juniper Networks. "Not at all," Scholz wrote. "We happen to use Cisco gear in our network and there happened to be a bug."

The researcher commended Cisco's reaction to his Black Hat bombshell. "As far as I can tell (Cisco is investigating) the PIX does some misinterpretation and 'can' open up the wrong ports for inbound traffic. In a nutshell Cisco did a pretty good job on reacting to this case from my point of view."

In case you're wondering where I was when Scholz was at the podium during Black Hat, I was attending Pete Finnegan's "How to Unwrap Oracle PL/SQL" session because I'd been told by an attendee at the show that several Oracle lawyers would be in attendance to make sure Finnegan didn't step out of line. I thought their blue pinstriped suits would stand out amongst the rainbow of hair colors, the glare of the facial piercings, and the black ink of the tattoos. No such luck.