7 Lessons Learned From Designing a DEF CON CTF

Capture-the-flag (CTF) events are both fun and educational, providing cybersecurity professionals with a way to flex their hacking skills while learning new concepts in a constructive and safe environment. Well-designed CTFs expose individuals and teams to operational challenges, novel attack paths, and creative scenarios that can be applied in their work as offensive and defensive security professionals.

But not all CTFs are created equal, and there's a lot more that goes into designing a successful CTF competition than just coming up with the challenges. Along with the technical design challenges, there are also operational considerations involved with setting up the environment and actually running the competition, creative planning required to set up an engaging game, and details to factor for related to gamifying the challenges, such as trade-offs in how scoring structure is set up.

"As a designer I want it [the CTF] to be challenging and fun. I want to reward people who are clever, who really work at it, and who are persistent," says Jenko Hwong, principal researcher on Netskope's Threat Research Labs team and team leader for last year's DEF CON Cloud Village CTF. "It also has to be practical for us to carry out."

Fun and practical was the mindset that Hwong brought to the DEF CON CTF, a massive multiday affair that had over 400 individuals and teams trying their hands at the challenge and a team of 20 working for him to run the event. A veteran researcher and seasoned CTF participant, Hwong had never run a CTF before this event. One of his biggest hopes for his first try at the job was to level up the relevancy and realism of the challenges in the event, which can sometimes be a bugaboo in CTFs today.

"Sometimes in these CTFs you get a really hard challenge, but it's like, what's the point of this? It'll be a decryption or encryption problem, where the event goes, 'Here's something, good luck,' and then you have to jump through all these hoops that may not be completely divorced from reality but don't really fit into a bigger storyline," he says. "So when I got the call, my thought was, 'Let's jump in. Let's figure out a good story and a good set of challenges that will be fun but also make sense and maybe relate to the real world of research penetration testing, defensive measures — what's happening in the real world.'"

As he dove into the project, though, he found it especially challenging to find much information about running CTFs. Most write-ups are from participants who rate events and explain how they solved challenges, but best practices about running an event were hard to find. As a result, Hwong and his team had to do a ton of work creating challenges nearly from scratch.

"The community generally shares a ton, so why aren't we sharing CTF challenges?" he says. "I think we can do better."

In that spirit of security community sharing, Hwong offers some important lessons that he and his team picked up along the way, so that others in charge of CTF design can learn and understand from the process. His goal is to run the event again and build on what they learned last year. He also hopes others will share their best practices, and even technical details, so that the whole security community can improve the quality of CTFs being offered.

Storytelling Is Key

Hwong says that his DEF CON Cloud Village team was very keen on crafting a storyline that was engaging and fun. He says he thought of the story as a movie script with realistic cyber scenarios built in. For the event they chose a theme of "Gnomes," which was fun and funny. But it wasn't just the storyline writing that was important — so were the technical challenges planned within the story.

"The goblin and gnomes storyline wrapped around everything, but the important thing was coming up with reasonable scenarios that you might encounter as a security professional, including attack paths and reasonable defenses you'd encounter," he says. "The more we can do that as CTF designers, the better it is for learning and the more fun the CTF."

Take a Software Development Approach

CTF creators should definitely take a software development approach to designing the technical elements of their challenges, Hwong recommends.

"You've got to think of design, implementation, and testing," he says, explaining that he and his team learned the hard way how difficult it can be to test challenges in a complex CTF environment that can be manipulated by participants in numerous ways.

"What happened — and I'll take the blame as the lead creator for not guiding the testing — is we missed the negative testing pass, as well as the viability checks," Hwong says. "Part of it is we didn't have enough time to test, so I was continuing to lock down some environments as the challenge was underway so [they] wouldn't be too easy and there were no loopholes. I think at one point for an hour or two I ended up making something unsolvable at a certain step."

So one of the big lessons Hwong learned is that CTF designers need to bring software development rigor to the table that goes all the way through testing and viability work.

Operational Rigor … and a Little Bit of Caffeine

Meticulousness in software development isn't the only technical capability that needs to come to the table. The crew running a CTF needs some serious operational rigor as well.

"We had some fabulous people running the servers and the AWS accounts and the Google and Azure accounts and making sure things kept running and that we were monitoring things," Hwong says. "All of that stuff has to be handled. And if you ignore it, it just could mean things fail, break, or you have performance problems."

One of the operational problems they ran into was some collision between participants and challenges, as the team was operating with a constraint in that they couldn't create a standalone environment for every participant across AWS, Google, and Azure.

"Because it was in the same environment, it helped them on other challenges and if you have a challenge that requires changing the environment, then you have people stepping on each other's toes, changing a shared object," Hwong says. He and his team had to reset policies as the CTF rolled forward so participants wouldn't run into one another, he adds.

Hwong and his team are trying to learn from the experience to figure out a practical method — from time, effort, and expense perspectives — to give participants a truly isolated environment without making the whole CTF less viable because things break or take forever to execute.

On the operational front, he says, CTF showrunners also have to be mindful of the constant communication that they'll need to facilitate between team and participants.

"I was on Discord after midnight and I'm like, 'I've got a talk to give in the morning, would you go to sleep?'" Hwong jokes, explaining that participants will have questions and are going to be pinging organizers for tips and pointers at all hours.

Designing Different Difficulty Levels Is Hard

Getting the difficulty levels of challenges right and creating a fair scoring system may be harder than a newbie CTF organizer may initially think, Hwong warns. A few of the levels that his team designed as easier were actually more difficult for participants to complete than they'd anticipated, while some of the more challenging levels were successfully finished by more participants than expected.

Another challenge is figuring out a scoring system that makes sense. After his experience at DEF CON, Hwong is a proponent of using some kind of bell curve scoring system. But the problem isn't as straightforward as instituting a curve, he says. There's also the issue of normalizing and balancing out the advantage that big CTF teams have in racking up challenge points — an issue that one of the participants provided him with feedback about after the event.

"So if your challenges can be divided and done in parallel with multiple players, if I've got 10 people, I will be 10 times as fast. And so there's an advantage," he says. "His point was we need some sort of dynamic scoring to level it a little bit. If there are things that he's really, really good at, he might be the only one who solves it and he'll get maximum points. The bell curve will reward him if it's something in his wheelhouse of expertise, if he's one versus 10. There's some debatable stuff here that we have to work through."

One possibility is making challenges sequential, but the downside of that is it could make the CTF too rigid and linear, and it could create a bottleneck or dependencies that could blow up one or more challenges. Hwong says he'd also love to see more CTFs reward participants on techniques like how stealthily they operate in an environment or dock points if they leave too many footprints and fingerprints. That's an area he'd like to explore as he designs future events.

Regardless, though, dynamic scoring is something that could alleviate some of the leveling issues. Hwong and his team are pursuing that for the coming year.

Blue Teams Need More Fun CTF Challenges

After working through his first CTF, Hwong also increasingly believes these events don't do enough to challenge and really engage blue team participants.

"Blue team exercises tend to go like this: 'We have a misconfigured environment with lots of vulnerabilities. Can you go fix them?'" he says. "And what they do is they just test whether those configurations are changed or whether I can access this public bucket. And as soon as you make it private, we know you fixed it and you get points. It'd be way better to do things on top of that, such as what if you're compromised, there's an attacker in your environment, and you have to find them and kick them out. So you have an incident going on right now, the attacker [has] credentials, and as long as they'll do things, you might be able to detect it. That's your job as a participant. And until you revoke their access, you don't solve it and you don't get maximum points."

Those kinds of scenarios are harder to do, but they're more realistic for defenders and will make CTFs more valuable for them, he says. This is also on his radar for next time.

CTFs Need More Fresh and Relevant Components.

Hwong also challenges CTF designers — and himself — to incorporate more fresh exploit and vulnerability information into their challenges. This was one of the things he wished he had more time to dive into in his first go at DEF CON Cloud Village and which he's resolved to improve for next year.

"This is one of the areas where CTFs can be more of a learning and training tool," he explains. "We would love to use relevant ideas and exploits fresh from researchers occurring earlier in the year or even presented at DEF CON."

CTF 'Building Blocks' to Improve 'Reusability'

Finally, one of the biggest lessons Hwong says he learned is that the industry needs to find more ways to create reusable components for CTF, just like software developers do for applications. He has dreams of helping to organize an open GitHub repository of small exercises in code that can form the building blocks of building out a CTF.

"You're still going to have to customize it and add your own twist, but the idea is, let's get the first 60% out of the way so CTF organizers can focus on really novel things. That way nobody is reinventing the wheel," he says. "And then the remaining 40% can be adding new techniques, scenarios, and storylines."