Knowing When To Call In Reinforcements

Knowing when you're in over your head is important. In the world of the IT security professional, it is especially critical given your knowledge and experience will determine your actions and influence your reports to management. Those reports will, in turn, impact their decisions (or at least they should).For example, what if the intranet server has been infected with malware that could spread to users' workstations? You have to determine the risk to the business. Can you take it down? If you do, will it impact user productivity to the point that your company starts losing money?

Now imagine that one-half to three-quarters of users' machines have already become infected because it wasn't caught soon enough. Yeah, not a fun scenario. I assisted a friend with a client that once to deal with this very situation. The application server in use wasn't properly secured to prevent users from writing to the same shared folders from where they were running the applications.

You can probably guess what happened next. A single user workstation got infected with malware that infects executables. The user's workstation wasn't caught until it had already infected the executables on the application server, which allowed the infection to spread to three-quarters of the client's workstations. Ouch.

The realization was made quickly that there was no way they could clean every system effectively with an sense of confidence that all of the malware was eradicated. Reinforcements were called in to assist with rebuilding every workstation, including an application server, file server, and Active Directory domain controllers.

What impressed me most about the way the incident was handled is they stepped back to look at their critical functions to the business. They then made decisions about how to proceed in order to keep the business functioning during the rebuild. They knew that if some of the systems were infected, then they could still be used reliably for some operational activities without risk to sensitive data because of placement in the network.

Additionally, they even had methods in place where they could use paper if the computers were down. To be honest, I was really surprised they had taken these steps -- though I'm not sure if having a methodology for paper backup was something they did proactively or because they had issues when the computer systems were first deployed.

What made me think of this incident was a recent post over at the Information Security Leaders blog, titled "Why Information Security is the Hardest Career." The post does a good job of covering why it is a hard career because of having to keep up with new technologies but it missed one thing, and that's an understanding of the business and the risks associated with the ever-changing threat landscape.

In other words, you need to stay on top of the changes, make sure you keep your incident response plan updated to address those changes, and don't ever be afraid to call in for backups when things get a bit hairy and you need help to keep the business running.

John H. Sawyer is a senior security engineer on the IT Security Team at the University of Florida. The views and opinions expressed in this blog are his own and do not represent the views and opinions of the UF IT Security Team or the University of Florida. When John's not fighting flaming, malware-infested machines or performing autopsies on blitzed boxes, he can usually be found hanging with his family, bouncing a baby on one knee and balancing a laptop on the other. Special to Dark Reading.