Your Cloud Was Breached. Now What?You're not happy. You just experienced a breach. Here's how to keep calm and secure your cloud.
First of all, take a deep breath. If you stay vigilant during a cloud breach -- and have a proactive security model in place -- you’ll weather the storm. The first step is to be prepared.
There are a lot of similarities between a physical breach and one that happens in the cloud. Some of the preparation mechanisms remain the same. The big difference comes in the toolset. Lots of cloud service providers (CSPs) offer very granular log aggregation, visibility into virtual networks, and even the ability to create cloud-ready audit trails. To be completely ready, here’s what you need to organize in advance:
- Documentation, electronic or physical. In many cases, you will need to create a breach protocol to follow. CSPs will probably have their own breach protocols. However, the data and settings residing on your system may still be your responsibility to document.
- Snapshotting services and physical removal tools. Even if your virtual machine lives in the cloud, at some point a physical server was compromised. You need to snapshot and isolate that server. This needs to be done immediately. Remember, snapshots can be done if a VM is on or off. Your initial step should not be to alter the state of the VM. Rather, it should be to document and snapshot the instance.
- Virtual and physical machines. There is a solid chance that you may need to transport data, snapshots, and other resources physically as well as virtually. In some cases, you’ll need to make arrangements to transfer the affected physical gear for post-breach analysis. Your CSP can help you take down impacted hardware for further testing.
The immediate response
There are three mandatory rules you must follow immediately, particularly if your workload is a VM or is residing in the cloud: Do not alter the condition of the VM or cloud instance. If it’s off, leave it off; If it’s on, leave it on. Avoid attempts to access files, and do not change settings.
These seven steps will take you from breach to remediation.
Step 1: Create snapshots of VMs, virtual appliances, and configurations. This can include screenshots, log dumps, or configuration collections. In some cases you may need to take written notes on what appears on the monitor, management screen, or any other output device. Active programs may require more extensive documentation of the virtual machine’s activity. At this point, it is imperative that you do not make any state changes to the cloud instance or VM, as it could significantly alter your research. The machine should remain active until you have the snapshotting process,
Step 2: Protect perishable data, both physical and virtual. Is there a drive attached to a server? Is DAS being used? Perishable data (both physical and virtual) should be immediately secured, documented, and/or snapshotted, and in some cases physically photographed. If an end-point using a cloud service becomes compromised, make sure to include the power supply and ensure these devices remain plugged in, even when in storage.
Step 3: Properly take down the physical resource or virtual instance. There are ways to take down a physical machine -- and ways to take down a virtual instance. Both are critical processes during a breach. Regardless, prior to changing the state of the physical or virtual instance, document and snapshot everything! Because most VMs and cloud platforms utilize shared storage, you may have some extra work here. Massive breaches can force you to take your storage platform offline temporarily for snapshots and evidence gathering. Document the LUNs, connections, and even disk aggregates that were used for that VM; create a snapshot of the assigned virtual disk(s); and make sure to document all processes during your investigation.
Step 4: Identify all incoming network lines, connections, virtual interfaces, and ports assigned to the VM or cloud instance. You will need to work with security, network, storage, and infrastructure teams to document and understand how all configurations impact the state of the breached cloud or VM instance. Collaboration during a breach is absolutely critical. Plus, your CSP should have dedicated teams to reference as well.
Step 5: Collect and label all media used during the response process. Just because your breach happened in the cloud doesn’t mean you won’t have physical documentation. Massive breaches still involve digital photographs, paper trails, and governance documentation. You will have digital and physical media that will be collected from the breached instance. Fortunately, cloud management tools can help with log aggregation and VM state identification and can even help provide historical reporting.
Step 6: Seal all collected devices, drives, and evidence in a secured area. Proper protocol will dictate that any and all evidence gathered must be locked down and secured. At this point in the process, you’ve taken your snapshots, pulled necessary physical components, and gathered as much data as possible. Now you absolutely need to lock it down for analysis and evaluation.
Step 7: Remediate and respond. You’re not happy -- you just experienced a breach. At a high-level, you understand where the breach came from. So your final task is to now to lock down ports, services, or other affected areas. But, if you followed my earlier advice, you’re also staying calm and looking at better ways to secure your cloud for the future.
Bill is an enthusiastic technologist with experience in datacenter design, management, and deployment. His architecture work includes large virtualization and cloud deployments as well as business network design and implementation. Bill enjoys writing, blogging, and educating ... View Full Bio