Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


10:30 AM
Jeff Schilling
Jeff Schilling
Connect Directly
E-Mail vvv

5 Pitfalls to Avoid When Running Your SOC

The former head of the US Army Cyber Command SOC shares his wisdom and battle scars about playing offense not defense against attackers.

The perspective you get running an incidence response team is very different from the one you get as a Security Operations Center director. I refer to it as a “press box view” of cyber security, based on my experience as former head of the U.S. Army Cyber Command's SOC and, more recently, as a civilian CSO at FireHost.

In these roles, I wasn’t responsible for securing any one infrastructure, but I came to understand how many of us were “playing defense” in security ops. I started keeping a notebook of root causes for the security breaches my team worked. From these root causes, I developed what I believe are the five most common SOC pitfalls.

Pitfall 1: Centralized planning and execution
Too many large multinational organizations try to accomplish both centralized planning and centralized execution. This causes a big data problem that cannot be solved, no matter how much artificial intelligence and computing power you throw at it.

Our approach in the global U.S. Army SOC was centralized planning and decentralized execution. Our global SOC pushed out our defensive countermeasures based on threat research but left it to the regional security operations centers to apply those threat indicators and manage the alerts from their security stacks in their regions. The regional teams also had the flexibility to exceed the security posture of the global posture if the regional situation dictated that approach. The security information and event management (SIEM) data did flow to the global SOC for analysis. However, the regional SOC took the first swing at the SIEM events to filter and tune out false positives.

Pitfall 2: Outsourcing the SOC mission and responsibility
In particular, I’ve witnessed two major trends: Companies outsource to managed security service providers and/or offshore their SOCs to high-value regions. In and of itself, it’s not a problem to outsource a portion of your SOC’s capabilities. However, you should never transfer the responsibility of security operations to a third-party provider. This model does not work. A company can and should outsource the alerting, and in some cases the managing of their security devices, but the SOC responsibility of overseeing the incident management of those alerts should remain in-house. I suspect that’s the inherent goal when companies outsource SOC capabilities, but invariably the in-house expertise that can drive action based on those alerts winds up mistakenly being let go or reassigned as part of a cost savings strategy. I recommend that any security team make a special effort to retain the incident handling function within the organization.

Pitfall 3: A belief that technology alone provides effective security
While picking the right technology is important, trained personnel and the right processes to leverage the technology are equally essential. I have a saying I use with my team: “Tiger Woods could play scratch golf with my golf clubs, but I cannot play scratch golf with his clubs.” The point is, you have to focus on leveraging the security tools and applying a methodical approach to analyzing the results from those tools. If you unpack every major breach from the past couple of years, you’ll see that often the technology detected the indications of compromise, but the right people and processes weren’t in place to address them. Don’t make training and process development the trade off for buying new technology; it’s a losing strategy.

Pitfall 4: Equating incident management and problem management
Many SOCs that I’ve visited both in my military and civilian career have a lot of activity but no real direction or objectives. They have plenty of data analysis and open tickets, but no discussion on how to get ahead of the threat and reduce their attack surface area. Before I transitioned to a security focus in my military career, I spent time in CIO and infrastructure provider roles and became very familiar with the ITILv3 framework used by most IT service providers. I noticed that most security teams can perform basic incident management, but they have a gap in their understanding of problem management.

If you are tracking the right metrics and doing trend analysis, you can see when security controls and strategies aren’t working. Very few security teams have the bandwidth to do this type of analysis, but it is the reason our threat actors continue to have the same success with the same techniques, tactics, and procedures.

Pitfall 5: Protecting everything (which, in most cases, protectings nothing)
Threat actors are only interested in probably two percent of your data and infrastructure, but they use the other 98 percent to get at that two percent. The smarter strategy I have seen from some of the most innovative CISOs is they treat their networks as “contested space” as opposed to a castle with an impenetrable wall around it. These innovators are aggressively segmenting their networks and moving their most valued data, applications, and VIP users to a more hardened, protected infrastructure. In fact, most of our clients are moving their difficult-to-protect and regulated data and applications to hardened cloud providers so there is a focused effort on guarding these critical assets.

Another thing we do within our security operations is to focus our security and vulnerability management efforts on what we determine is “key terrain.” The threat playbook has not changed much in the past 10 years. Generally speaking, threat actors compromise a host, elevate privileges, and then look for opportunities to use the victim’s infrastructure against them. Our approach is to ensure that infrastructure, such as Active Directory, software distribution systems, and other key terrain are hardened and audited regularly for threat activity.

These five pitfalls are not a comprehensive list by any means. But they are where my security team puts investment and focus. Our goal is to protect our critical assets, quickly know when they have been compromised and respond with immediate action to contain and eradicate the threat. If anyone believes they are going to create the perfect secure environment, let me save you some pain in discovery: It does not exist. However, if you can narrow your attack surface area through smart security operations that fully integrate the right people, the right processes, and good technology, then you drive up the skill required by an attacker to the point where most threat actors will give up and go after easier, softer targets.

Jeff Schilling, a retired U.S. Army colonel, is Armor's chief security officer. He is responsible for the cyber and physical security programs for the corporate environment and customer-focused capabilities. His areas of responsibilities include security operation, governance ... View Full Bio
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Author
12/22/2014 | 3:26:07 PM
Re: Centralized planning &decentralized execution
First, thank you for your service as well.  I offer one clarification to your comment that a SOC is reactionary only.  At FireHost, our security methodology for our SOC is to Protect, Detect, Respond, Recover.  You may recognize that from several DOD and NIST standards as the model for cyber security operations.  I believe that a SOC should be as active in protection or hardening operations (Vulnerability discovery, Patch management direction, Security Control tuning) as they are in detecting and responding.  Some organziations may make this outside of the scope of the SOC, my personal opinion is it belongs in the SOC.


To answer your question, I think I am on the same track as you with my SOC, though I don't see it as a "poor man's" SOC.  I think you have and brilliant idea.  Most SOCs have Security Analysts with basic security experience and skills that track incident handling.  I am filling my SOC positions with people with an Incident Response background, who understand how to do host level and network forensics.  I am in a unique position in that my SOC protects our cloud environment.  Our SOC analysts routinely work with customers to examine their servers they host with us as we discover indicators of compromise.  For me these IR skills are required for my SOC Incident Management team.


As I mentioned in a previous response, recommend you set up an operational process that allows you to daily review the tactical information you receive from your security operations and controls.  We use the OODA targeting process (Observer, Orient, Decide Act).  We do both Daily and weekly OODA huddles and targeting meetings to ensure we are actioning anything we learn through Threat intelligence or discover/detect through security operations.

User Rank: Author
12/22/2014 | 2:25:11 PM
Re: Centralized planning &decentralized execution
Great commentary.  At FireHost, we subscribe to the OODA Targeting process (Obeserve, Orient, Decide, Act).  Think of these as swim lane processes that are constantly in motion on a perpetual basis.  We have twice daily OODA huddles where our Incident Management team goes over tactical infomation exchange to ensure synchronizaiton between the Incident managment team, Security Device managment and the Vulnerability Threat Management team.  Then our weekly OODA Targeting meeting is what I would describe more of a "week in reveiw" and gets into the first stages of problem management if you follow the ITILv3 process.  Our Audit and compliance team manages our GRC process that our weekly OODA meetings feed problems to the risk register for more complex problem resoultation.  In other words, the secruity issue which require more than the folks that work for me to solve.


User Rank: Ninja
12/22/2014 | 2:10:54 PM
Re: Centralized planning &decentralized execution
@Jeff.schilling,... THANK YOU FOR YOUR SERVICE. I'M also U.S. Army, retired... although my day-to-day was not information security or even technology, I got into this after I retired. What you described as a centralized plan development and decentralized execution is at a high level exactly how I was expected to do my job. Sure, the strategic thinking\planning is always done at a higher level, but if I'm the guy on the ground, you have to give me the leeway to make decisions based on what I see locally, not how you see in from the comfort of your "Press Box" (I like that). Also, I find "the military mind" is a perfectthing for InfoSec, we've already been conditioned not to trust anyone :-).

One thing I think people should know and understand about a SOC, it's first role is totally reactionary for the purpose or monitoring, unless it's been designed with some kind of offensive capability. And the response to any emergency is only as good as the preparations, operational policy and procedure... standards, guidelines and how well these defences were prepared and maintained. And the folks of the SOC need to understand those same things because it will give them a better view of the big picture if they know for instance what exactly is being fed into the SIEM or when\what was the last FW rules change and does that CVE just released contain anything that can be used to exploit any critical or user systems.

One question, I just finished a SIEM integration, we cannot afford a SOC, but I'm trying to build a "poor man's SOC", if you will. I want to plug our IR-Teams into the SIEM, get them training on that so they can have that situational awareness on a day-to-day. What else would you recommend?

Thanks again!
User Rank: Ninja
12/22/2014 | 11:56:16 AM
Re: Centralized planning &decentralized execution
Great article Jeff!
As a retired military member, I can relate to all you provided.

My management of regional or field level security teams has always started with communications effort.  LOTS of communications effort.

I have established an ops-tempo for security services around a morning "stand-up" conference call with all of the security team leads.  During this meeting, we discuss the following items:
  • Status of operations including any personnel shortages
  • Current issues and alerts
  • Operational hurdles that need my interaction/intervention
  • Quick lessons learned
  • Project/tasker statuses

The meeting is limited to 30 min.  Anything I think is too complex to resolved quickly is promptly offboarded to separate discussions/communications after the meeting.  The general idea is to establish a constant presence with those people who are in distant locations, even for a few moments. 

The real benefits of this daily stand-up are
  • Dramatically improved situational awareness of security and operations status.
    There are times that I know about things going on that other executives are not even in tune with.
  • Established esprit-de-corps and cross-domain team mindset among all members of the security teams.
  • Stronger establishment of policy and standards practices among all teams.
  • Ramped up knowledge sharing and synergistic activity among all security team members.
  • Biggest win: a demonstrated effort to let other security teams know that I/we are there to help sort thought tough challenges.  Every day, team leads have a chance to bring concerns to the table and I/we always make an effort to give prompt response.
  • Very few timeline "misses".  We still have timeline pushes, but we usually identify those situations early enough to manage them.

Some would say that an every day meeting is too much. 
I say it is not enough.  Too often I have seen things that could have been quickly resolved if they were bought to the fore as soon as they were identified.  The old addage "that will have to wait until our next meeting" has lesser associated risk. 
The perception of the entire team by our customers is "efficient and effective" service.  That will always be a goal.
User Rank: Author
12/19/2014 | 10:58:05 AM
Re: Centralized planning &decentralized execution
Marilyn, thank you for your question.  The connectivity between a Global SOC and regional SOC is a tough problem, but not insurmountable with a good network engineer and a platform integration team who understands how the information flow needs to happen between the organizations.  Believe or not, the bigger challenge is establishing governance and business processes that dictate how you manage this construct.  That is an article in and of itself.  It really comes down to building a common set of security values across the global organization, grounded in policy, procedures and standards.  Once you get that in place, making the electrons flow is less of a problem.

Marilyn Cohodas
Marilyn Cohodas,
User Rank: Strategist
12/19/2014 | 9:56:03 AM
Centralized planning &decentralized execution
Jeff, Great insights. But I wonder what kind of communication exists between the central and regional teams. Can you elaborate? 
Ransomware Is Not the Problem
Adam Shostack, Consultant, Entrepreneur, Technologist, Game Designer,  6/9/2021
How Can I Test the Security of My Home-Office Employees' Routers?
John Bock, Senior Research Scientist,  6/7/2021
New Ransomware Group Claiming Connection to REvil Gang Surfaces
Jai Vijayan, Contributing Writer,  6/10/2021
Register for Dark Reading Newsletters
White Papers
Cartoon Contest
Write a Caption, Win an Amazon Gift Card! Click Here
Latest Comment: Google's new See No Evil policy......
Current Issue
The State of Cybersecurity Incident Response
In this report learn how enterprises are building their incident response teams and processes, how they research potential compromises, how they respond to new breaches, and what tools and processes they use to remediate problems and improve their cyber defenses for the future.
Flash Poll
How Enterprises are Developing Secure Applications
How Enterprises are Developing Secure Applications
Recent breaches of third-party apps are driving many organizations to think harder about the security of their off-the-shelf software as they continue to move left in secure software development practices.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2021-06-18
RIOT-OS 2021.01 before commit 44741ff99f7a71df45420635b238b9c22093647a contains a buffer overflow which could allow attackers to obtain sensitive information.
PUBLISHED: 2021-06-18
SerenityOS contains a buffer overflow in the set_range test in TestBitmap which could allow attackers to obtain sensitive information.
PUBLISHED: 2021-06-18
SerenityOS in test-crypto.cpp contains a stack buffer overflow which could allow attackers to obtain sensitive information.
PUBLISHED: 2021-06-18
SerenityOS before commit 3844e8569689dd476064a0759d704bc64fb3ca2c contains a directory traversal vulnerability in tar/unzip that may lead to command execution or privilege escalation.
PUBLISHED: 2021-06-18
RIOT-OS 2021.01 before commit 85da504d2dc30188b89f44c3276fc5a25b31251f contains a buffer overflow which could allow attackers to obtain sensitive information.