Security Tips For Virtualization Spinning up VMs is easy--too easy, in fact. Fortunately, keeping virtual servers safe doesn't have to be expensive.
Here you are, adding yet another server to your virtualized environment that went from beta to production in the data center equivalent of zero to 60 in 4.5 seconds. That speed means the security policies and processes you routinely applied to physical servers probably went out the window over the past few years. In our latest survey on virtualization security, one IT pro told us that, for questions like "Do you harden your hypervisors?" we should provide the option of answering "No, and no future plans, but we know we should have plans."
We've tracked this area for some years in our practice and recently revisited the InformationWeek Analytics Virtualization Security Survey we first deployed in May 2008 to see how attitudes and practices have evolved. One thing we do know: That commenter is dead on. Yes, virtualization saves money and can increase reliability. But complexity is the enemy of security, and today's virtualized infrastructures are anything but simple, between virtual I/O, hardware appliances custom-built for virtual environments, and virtual software being used for what once was done in hardware. And now, storage, desktop, file, and network virtualization are on the table, too.
If your security policies haven't kept up, it's time to tap the brakes.
In some areas, we see surprisingly little movement from two years ago. For example, we asked how virtual machines are viewed with regard to security risk; 76% of the 423 business technology professionals responding in 2008 said their companies considered VMs as safe or safer compared with physical servers. In our new poll, 75% of 684 respondents answer the same. In 2008, 12% had no VM security provisions in place. Zero. In this survey, it's down a mere 2 points, to 10%. But there are some swings. Most notably, the number of companies with VM-specific security tools in production nearly doubled. Apparently, sellers of virtualization security appliances (VSAs) have gotten their message across.
So, do you need to take some of the savings achieved by server virtualization and pump that cash into VSAs?
No. You need to get back to basics.
Virtualization security is useless if you think only in terms of securing VMs. Virtual servers should be no different from their physical counterparts. They run the same operating systems and require the same security technologies and processes. But there's one very big difference. Physical servers provided for a natural separation of duties within the IT organization; as a result, most enterprises have a distinct network team and systems team. Separation of duties is good security practice. But that's largely gone out the window when it comes to managing virtual environments. Nearly half (47%) of those involved in provisioning and ongoing management of the virtual switching infrastructure are members of the systems team, not the networking team.
At first glance, it might seem that since the hypervisor is running on a server, the systems team should be at the wheel. But in the more than 50 network audits and compliance reviews my team and I have performed--and in interviews for this report--we see a common problem whenever systems teams manage the virtual environment: They ignore everything but the VM itself. The network? Single LAN for all servers--even in virtual desktop deployments. Hardening of the hypervisor? Defaults, ignored, or they didn't even know that the hypervisor could be secured. It may sound harsh, but we've never once walked into an audit and seen a hardened hypervisor or properly secured and architected virtual network.
It doesn't have to be this way. The virtual environment does have some of the same modularity as the physical environment. You can, and should, use virtual switches, create complex network topologies, even implement network-level high availability.
In response to complexity, some enterprises have assigned dedicated engineers to manage virtual systems. These "cross-functional teams" are in use by 22% of our respondents and comprise VMware or Hyper-V experts who, in theory, know how to do virtualization security. But in our experience, these people end up becoming provisioning experts and resource allocation managers--which makes sense because those skills are absolutely required to keep a production virtual environment performing as expected. The upshot, though, is that we've centralized our human and our technology resources, even as virtualization dramatically increases the complexity of implementing security controls.
And therein lies the main problem.
It's Not Easy Being Virtual
Abstracting hardware adds layers of decision-making when implementing security. Where do you put the control? How do you ensure the control is inspecting all virtual traffic? Do you have your VSA compete for the same physical hypervisor resources you want your production systems to use? What if you want to use an existing physical intrusion detection or intrusion prevention system?
It would take a scorecard to track all the network configurations, abstractions, and mappings required for a virtual appliance to work correctly in even a mildly complex enterprise setup. No wonder we consistently see IT teams either delay the decision or give up altogether.
"I plan to look into virtualization security when the field has matured a bit more, and I am more educated on the threats," says one survey respondent. Our response? The field doesn't need to mature. It uses the same technologies and, more important, policies and best practices you use in the physical world.
Of course, given how well baseline security technologies are protecting us, that's not saying much. But that's another report.
If virtualization is really just complex physical computer security, what about all those virtualization-specific risks, such as Blue Pill and hyperjacking?
We attempted to measure our respondents' virtualization-specific security concerns by asking about hyperjacking in particular. Since 2008, the number of respondents who don't know what hyperjacking is decreased by 10 points, down to 26%, but the number who say hyperjacking scares the heck out of their security folks went up a mere three points, to 11%. (Hyperjacking is an attack that compromises a hypervisor and gives an attacker access to all VMs on the physical server.)
Knowledge is power, and many times in the security world, knowledge is used to create FUD. Hyperjacking is a perfect example. Is it a risk? Sure. Is it a highly probable risk? No way. Right now, hyperjacking just doesn't pay big enough, from an attacker's perspective. It's a low-level attack that requires so many variables to align that the probability of a successful break-in is very remote. You'd have to breach the hypervisor directly from an external source, which means the hypervisor itself would have to be directly accessible via the Internet, and we've never seen that in the real world, thankfully. Instead, focus on efforts that will reduce the impact of a security breach, such as properly implementing the vSwitch architecture and adding hypervisor configuration change detection.
A lack of access control presents another problem within virtualization security. Forty-two percent of respondents say virtualization administrators have access and login rights to the VMs they administer. An additional 41% have access, but with different user IDs and passwords. Overall, 83% of respondents say those who manage their virtual environments also manage their VMs. There is no segregation of duties. Any one of these administrators could change a resource, take snapshots, and start up new instances of servers--or even easier, copy the virtual disks to removable media and in effect take entire servers home in their pockets.
Penetration of the hypervisor management system is the second most common concern from our respondents, behind penetration of a guest VM containing sensitive data. A lack of segregation of duties increases this risk, especially if the management system is authenticating against a domain such as Active Directory. Many times we see that compromised administrative logins are the same user names and passwords used to control the entire hypervisor, since the hypervisor authenticates to AD.
In a physical server environment, configuration is hard to standardize. Virtualization helps here and allows for the rapid provisioning of servers; it's what allowed the cloud to be created. But the ability to rapidly spin up VMs also can be a security risk in and of itself. Commonly referred to as VM sprawl, this issue is referenced by most virtualization management vendors when advising IT to adopt centralized management of the virtual infrastructure. Yet, according to our respondents, sprawl isn't seen as a security risk; it's ranked lowest in terms of security concerns.
We agree that VM sprawl itself isn't a security problem. The likelihood that someone will create a rogue VM to steal data or cause a DoS for other VMs is small. What is a problem is that the lack of VM inventory management technology enables copying of raw virtual disks to removable media. Most VM management tools have access-control capabilities to prevent creation or snapshots of VMs, but it's unclear whether these controls are enabled.
If they're not, do it today.
Top Virtualization Threats
There are other problems you should address now to keep your virtual server infrastructure safe. We cover these in more depth in our full report, but at a minimum watch for:
>> Loose controls: Implement strong change management that is auditable and mandates a separation of duties. The logins used to manage the virtual infrastructure must not have access to anything but the virtualization management software. Also, all virtualization infrastructure changes should be logged, and those logs reviewed by someone not on the virtualization team.
Events such as restarting VMs, creating new VMs, and adjusting hardware should always be correlated to reasons they were done.
>> Shoddy virtual network design: Virtual networks are complex, mainly because of the abstraction. Go slow, use Visio, and map it all out so proper segmentation is deployed and security control points (these are where security controls can be installed) are defined before the virtual network is created. Think about firewalls, IPS/IDS, Web application firewalls, and routing.
>> Unsecured physical data stores: Make sure the spot where your VMs and snapshots are stored is not easily accessible. If an attacker or malicious insider can access your virtual disk files, it's game over. This includes backups. Don't put backups of virtual disks on a public file server, ever.
>> Thinking VSAs will solve the problem: You don't need virtual security appliances to manage most virtualization risks. Is there anything wrong with using your physical IDS/IPS or firewall and trunking the VLANs to the virtual switch? We don't think so. Use what you have. If you want to add VSAs to address performance problems, fully test the appliances at adequate load before you buy.
>> Believing what you hear in the news is real risk: Don't be reactionary. Thinking the latest virtualization attack you saw at Black Hat will affect you next week will only result in you having to take some real blue pills to reduce stress. Most of these attacks are still in the theoretical stage and simply aren't practical for attackers because they don't deliver quick ROI.
New Problems Need New Solutions
OK, so we've established that virtualization is complex, therefore virtualization security is more complex than security in a physical server world. What are respondents' companies doing today to cope? Most are still using conventional infrastructure tools that don't support virtualization, but fewer than were in 2008: 44% vs. 54% in 2008.
We do recommend investigating virtualization-aware versions of established security products. McAfee and Citrix have announced such a partnership, and VMware is enabling this effort via its VM-safe API. We like the idea of these systems, as do our respondents--use of virtualization-aware security technologies is up to 21% from 17% in 2008. These technologies leverage investments you've already made, in terms of process and training, but there are pitfalls you must be aware of. First, the abstraction of security technology to a layer between the VMs and the disk or memory itself does introduce latency, and these tools can't do everything at this abstraction level, so you may need a hybrid setup to detect attacks at the host, abstraction, and hypervisor management levels.
The largest security improvement any company can make in its virtual environment, however, isn't related to technology at all. It is change management. Put processes in place to ensure segregation of duties between virtual administrators and guest VMs. Make sure all access to and from the hypervisor is monitored, and approvals are in hand before changes are made.
Hardening of the hypervisor followed by proper configuration of the virtual infrastructure itself will decrease the impact of any successful attack. Think containment; VLANs, segmentation, and port mirroring were instituted for a reason, so keep using them in the virtual world just as you do in the physical one.
Don't scrap policies specific to your environment that have worked and kept you safe; all the same features and capabilities exist in the virtual world. They may be check boxes in the hypervisor software instead of a command-line interface, but they're still important.
Start focusing on the human element, and don't expect one team--or worse, one person--to have all the expertise needed. Abstracting hardware is difficult, and implementing security properly in all areas of a virtual environment isn't easy, either. But it must happen before even more areas come under the virtual paradigm.
Michael A. Davis is the CEO of Savid Technologies, a technology and security consulting firm based in Chicago. Write to us at [email protected].