Revamping FireEye's
Reverse Engineering CTF

FireEye runs a reverse engineering competition called Flare-On (flare-on.com), which started in 2014 and has continued each year since. One of their objectives with this project is to harvest talented recruits for the FireEye Labs Advanced Reverse Engineering (FLARE) team, collecting contact information for as many highly skilled reverse engineers as possible.

In the beginning, the event was run using a mail server. Competitors would send emails to special hidden email addresses to get each challenge. However, the interface was clunky and it yielded poor data — so FireEye decided to look for alternatives. By moving to CTFd, they improved their data, expanded the reach of the competition, and gained them a much friendlier interface.

Challenge

When Flare-On started out, it would announce the first challenge of the event publicly on a website. When a user solved this challenge, they would receive an email address. When they sent an email to that address, the competition’s mail server would respond automatically with the next challenge.

FireEye liked that this system was iterative, requiring users to solve the problem in front of them before moving on to the next. But the interface was not easy to use, and the information it yielded left much to be desired: competition data were limited, painstaking to collect, and not statistically accurate.

“We really had no idea how many people were participating,” said Nick Harbour, Senior Staff Reverse Engineer at FireEye. “Since our old system only captured solves and had no identity management to speak of, we had to make some terrible correlations from web logs to get initial download stats, which were most likely horribly inflated.”

Most importantly, FireEye needed to expand the reach of the competition to serve their recruitment goals.

Solution

FireEye approached CTFd in 2016 about using the CTF platform for Flare-On. Specifically, they wanted to allow competitors:

1. To unlock each challenge one by one
2. To upload resumes (in addition to submitting their contact information)
3. To offer feedback on the competition

We customized the platform to meet those needs, and FireEye began seeing results.

Results

User interface brought happiness.

FireEye was happier interacting with the sleek and friendly CTFd user interface. Competitors too have expressed positive feedback on social media about the competition in general.

Data quality improved.

"We anticipated that the slick interface, easy configuration, and stability would be a big win for us, but what surprised us was what we weren’t expecting: our data got better. We finally got accurate stats, and we were able to seamlessly collect the data we need for global recruitment."

-- Nick Harbour, Senior Staff Reverse Engineer, FireEye

Participation climbed.

Comparing year-to-year data, it’s clear that user interaction increased since FireEye moved Flare-On to CTFd, making it possible to reach more people.

2014 Before CTFd

• 883 (12.37%) - Competitors who completed Challenge 1
• 226 (3.17%) - Competitors who completed Challenge 7 (final)

2017 After CTFd

• 2620 (78.77%) - Competitors who completed Challenge 1
• 251 (7.54%) - Competitors who completed Challenge 7
• 142 (4.26%) - Competitors who completed Challenge 12 (final)

Recruitment goals were supported.

By increasing participation, CTFd supported the business objective that motivated FireEye to launch Flare-On in the first place: to drive recruitment in reverse engineering, expanding their pool of high-performing candidates.

"CTFd has made it easier for us to scout blue-chip prospects and elite veteran tech talent for our company."

-- Nick Harbour, Senior Staff Reverse Engineer, FireEye

Summary

By transitioning Flare-On from a custom mail server to CTFd, FireEye is able to reach a wider audience on a more intuitive interface, while gaining access to accurate, easily processed competition data. More importantly, they’re meeting their primary business objective for this project, to improve recruitment by putting FireEye in touch with an extensive pool of talented reverse engineering candidates.