When I first got into Information Technology over 20 years ago, I started out in networking and data centers. When doing work, we always made sure to have another method of accessing our gear in case a configuration change didn’t work and eliminated access to the equipment or environment. Also, during my military service we would always discuss what would happen in a worst case scenario and have contingency plans in case something went wrong. I’ve carried these principles over into my security career and they have served me well. Given the high likelihood that we are all operating in some type of compromised environment, I think it is good to cover how these principles can be used to maintain operational security during incidents.
Philosophy
Hope for the best, prepare for the worst
No battle plan withstands first contact with the enemy
Train how you fight
There are a lot of things you can do to prepare for a security incident such as, running table top exercises, preparing playbooks and modifying tool sets, but there is an important area that I think is often overlooked during preparation. There is an implicit assumption that all of the fancy tools and environments you have prepared will be available when the incident kicks off. If you are preparing for your incidents with this assumption then you could find your team in trouble. One key component of a realistic table top scenario is to remove capabilities and see how the team reacts. This could be something as simple as saying the incident commander is no longer reachable or something more drastic like not being able to access your SIEM. The point is, the more worst case situations you can prepare for, the better off you will be during a real incident. Here are a few things you should consider to maintain operational security and the ability to respond.
Access To Playbooks
Playbooks are a critical tool for responding to an incident. Incidents can be highly stressful and playbooks exist to help your responders remember all of the steps, access methods, processes, tools, etc. they need to successfully respond. What would you do if an incident kicked off and you couldn’t access your playbooks? Does your team have these playbooks stored in an alternative environment? Do they sync them to an offline, physical storage device that is stored in a safe or passed between people on call?
Playbooks need to not only be accessible via alternate methods, but they also need to be protected. What if the attacker had access to your playbooks? I’m willing to bet your playbooks have a detailed list of all the things your team will do to respond and this can be a useful blueprint for an attacker to follow to avoid detection or expand their footprint. I highly recommend encrypting your playbooks and vaulting the credentials so you can monitor and control access as needed.
Access To Tooling
Tooling is also essential during the incident response process. Tools help teams investigate, gather evidence and ultimately recover from the incident. Access to tools and scripts can help with a number of activities such as, identify hashing algorithms, identifying password encryption types, scanning for open ports or scanning for vulnerabilities. Tools that are useful during incident response should be stored in an image or container that is updated regularly and kept in an offline environment. This will help the incident response team in a few ways. First, it will make sure all of the necessary tools that are needed in the playbook are available even if systems and network access isn’t accessible. Second, it avoids having to pull tools from the internet that haven’t been vetted. In a stressful situation like an incident it is possible to overlook small details that could make your situation worse, such as downloading a tool that has a trojan or backdoor in it. Lastly, it is important to control the flow of information during an incident. If your team is using online tools as part of the investigation they could be leaking information to the internet that can reveal details about the incident, which could not only derail your investigation, but could harm your company brand and reputation.
Another consideration for tooling is how you plan to preserve evidence if you are under active compromise. Having pre-staged environments that are isolated and only used for incidents can be useful. Similarly, having tooling available to image machines, quarantine networks or even encrypt files to send them to an alternate cloud / storage provider can all be useful during an incident.
Access To Communications
Communication during an incident is key. I like to set up a standing War Room where people can come and go as needed. My presence is there to make decisions and help direct resources where they can be most useful. In the chaos of an incident it is important to make sure you have primary, secondary and sometimes even tertiary communication methods available. What happens if Zoom or WebEx is down and you can’t stand up a virtual war room? Do you have access to a call sheet so you can get a hold of everyone and stand up a conference call? What would you do if your primary chat method such as slack or teams wasn’t available or was compromised? Do you have Signal or Wickr set up so you can still chat with your team?
Additionally, I highly recommend you copy legal on your incident communications so they can advise you accordingly (see my post on Legal Privilege). I also recommend you encrypt your communications, which can help protect your comms if you are operating in a compromised environment.
Access To People
Lastly, but probably the most important, is access to your people. People are essential when responding to an incident and running table tops to prepare them is essential. Cross training should also happen regularly in case an expert in a specific skill isn’t readily available. Ultimately, the more training and cross training your team has, the better prepared you will be for an incident.
Conducting table top drills and running through failure scenarios can be fun and also educational. Next time you run a table top exercise take out your top person and see how the team reacts. Put people on the spot in different roles such as investigation, evidence gathering, recovery, interfacing with engineering, etc. If someone struggles with a role during the table top you can flag that area for additional training.
Leadership is also essential during an incident and I’ve written blog posts about this in the past. Try rotating people in different leadership positions so they can get a feel for what it takes. This can also help them understand and anticipate what is needed from them during an incident.
Wrapping Up
Incidents can be both exciting and stressful at the same time. The thoroughness and frequency of your training will dictate how well your team responds during an incident. Most importantly, planning for various worst case scenarios can ensure your team is able to successfully respond, communicate and lead the rest of the business through an incident.
370 Security Blog 