Tag: #incident

What happens post incident?

One of the most exciting, stressful and true tests of a CSOs ability to lead during crisis is during a security incident. Unfortunately, it is inevitable that CSOs will experience several security incidents during their tenure. This could be something as small as a configuration error, or as large as a full on public data breach. I also think CSOs should assume they are operating in some state of compromise such as a malware infection or an attacker with complete remote access. Given this volatility of enterprise environments and the inevitability of some sort of security incident, the question becomes what happens afterwards? Just because you are no longer under active attack doesn’t mean your work is done. As a CSO, here are some things you should consider after an incident:

Retro / Post Mortem

First, I highly recommend conducting a retro or post mortem on the incident. This is a blameless session to discuss what happened, why it happened and most importantly what the team learned from the incident and what they are collectively going to do differently. During this exercise the CSO should plan to ask questions and listen a lot. I find being the note taker is exceptionally helpful because it is difficult for me to talk while taking notes. I want to hear opinions and ideas from the team on what happened and how we are going to improve. The result of this retro will likely kick of follow on activities such as increased training, requests for investment, development of new processes, creation of new detections or even additional automation. The point is it is a time to learn and rebuild.

Investment

Never let an incident go to waste. Instead of viewing the incident as a failure, I choose to look it as an opportunity for growth. It is easy to point the finger after the fact, but no environment is 100% perfect and secure. Therefore, after the retro is completed it is important to capitalize on the incident to ask for additional investment to respond to the improvement areas raised in the retro. This could mean asking for additional personnel to respond to events, it could mean additional training to fill knowledge gaps, it could mean a new tool or technology to improve detection and response or new processes to improve responses. As the CSO you need to distill down the retro suggestions into an actionable plan that focuses on the risks along with areas for improvement and investment.

Increased Regulatory Requirements

Unfortunately, there are consequences for having a security incident. For publicly traded companies this can come in the form of increased regulatory requirements such as being required to have an independent 3rd party audit your security practice on a periodic basis. It could also mean fines or reporting the incident as material in your upcoming SEC filings. You may even get inquiries from various regulatory bodies asking what happened and what you are doing to prevent it in the future.

Legal Ramifications

Similar to the regulatory requirements your company may face increased legal pressure as a result of the security incident. This could come from a number of different sources such as: lawsuits from outside entities (such as a consumer group or class action), increased contractual obligations from customers who are now concerned about your security practices, law enforcement investigations and costs for outside counsel who have expertise in your specific business area and are helping your company limit the damage.

Financial Implications

This is probably the biggest area for a company to navigate after an incident. Financial implications can manifest in a number of ways. Here are a few:

Increased Cyber Insurance Costs

As a result of the incident, your company may face an increase in cyber insurance premiums. These premiums may even become so expensive that your company can no longer afford to pay them, or your company could be deemed uninsurable.

Customer Impact

Depending on your business, this could be something simple like contracts not getting renewed and loss of new business or it could be something more material like having to pay to notify all of your consumers, paying for credit monitoring and having to compensate your loyal customers in some way to retain their business.

Market Impact

This is a broad area, but post incident your company could face a decrease in stock price. It may even be more difficult for your company to secure lines of credit because of the business risk. The severity and financial impact of the incident could potentially put your company at risk for M&A takeover or may require the business to declare bankruptcy and look for a buyer who has enough capital to weather the storm.

Budget Impact

Again, this is a broad area, but whenever I have an incident I try to keep track of the number of people involved, number of hours it took to deal with it and the opportunity cost of dealing with this versus doing something else I had planned. As an example, the opportunity cost could be something like “as a result of this incident we had to delay project x for 3 months while we dealt with the incident.” All of this will have an impact to timing, budgets and manpower.

Can You Calculate The True Cost Of An Incident?

Unless your company keeps track of everyone’s time and is able to create a time code entry for each specific incident, I find it unlikely that we will ever know the true cost of an incident. There are many direct costs, but there are so many indirect costs that are hard to estimate. The ramifications of an incident may stretch on for years and may change hands from one CSO to the next. However, I do think a CSO should have enough grounding in business principles to be able to estimate the cost of an incident. This can be useful when gaining support from your other C-Suite peers, when presenting to the board or when making investment requests to the CFO. Having context for all of the ramifications of an incident along with potential areas of growth and improvement can be a valuable story to tell.

Leadership During An Incident

At some point in your CSO career you are going to have to deal with and lead through an incident. Here are some things I have found helpful.

Know Your Role

Unless you work at a very small company, I argue your role is not to be hands on keyboard during an incident. You shouldn’t be looking up hashes, checking logs, etc. Your role is to coordinate resources, focus efforts and cognitively offload your team from key decisions. You need to lead people during this chaotic event.

Declaring An Actual Incident

This may vary depending on company size and type, but in general the CSO should not be the one to declare a security incident. The CSO (and their representatives) can certainly advise and recommend, but declaring an incident carries legal, regulatory and business ramifications that should be made by a combination of the Chief Legal Counsel and some representation of C-Suite members (CEO, CTO, etc.). Once an incident is declared, your company will most likely need to disclose it on SEC forms and customers may need to be notified. All of this could impact your company’s reputation, stock price and customer goodwill.

Use A War Room

A war room is simply a place where everyone can gather for updates, questions, etc. It is a place that is dedicated to this function. If you are physically in the office, it is a dedicated conference room that has privacy from onlookers. If you have a virtual team it is a Zoom, Teams, WebEx, etc. that gets created and shared with people that need to know.

The CSO’s role in the war room is to keep the war room active and focused. Once the war room is created and the right people join, everyone should discuss what happened, what is impacted and what the course of action should be. Document this somewhere and pin it to the appropriate channels. If people join and start asking basic questions, send them away to read the existing documentation first. If people want to have a detailed technical discussion then send them to a breakout room. The point is to keep the main room clear for making decisions and directing resources.

Bridge The Gap

Your role during an incident is two fold – 1) Communicate to other leaders within the company about what happened so you can get the appropriate support to resolve the incident and 2) Direct the appropriate resources to focus on resolving the incident quickly, while following appropriate chain of evidence, legal requirements, customer notifications, etc.

Communicating To Executive Leadership and the Board
 

Keep it short and sweet so they can respond as needed. The purpose of this email is to inform them so they can give you the support you and your team need. Make sure to invoke legal privilege and keep the audience small (I discuss this in my post about Legal Privilege).

I use the following email template when communicating about an incident.

Subject: PRIVILEGED – Security Incident In [Product/Service X]

A security incident was detected at [Date / Time] in [product x] resulting in [data breach/ransomware/etc.] At this time the cause of the incident is suspected to be [x]. Customer impact is [low/medium/high/critical].

The security team and impacted product team are actively working to resolve the incident by [doing x]. This resolution is expected [at date / time x].

For any questions please reach out to me directly or join the war room [here].

Next update to this audience in [x time period].

Communicating To Responders
 

Your job here is to get the team any resources they need, offload them from decisions and then get out of their way. It is also important that you buffer them from any distractions and protect them from burnout by enforcing handoffs and reminding people to take breaks. It is easy for your team to get caught up in the excitement and sacrifice their personal well being. Learn to recognize the signs of fatigue and have resource contingency plans in place so you can shift resources as needed to keep the overall investigation and response on track.

Designate someone to help coordinate logistics like meeting times, capturing notes, etc. Capture action items, who owns the action item and when the next update or expected completion time will be.

Have A Backup Plan An Practice Using Them

Hope for the best and prepare for the worst. Can your incident response team still function if your messaging service is down? What if your paging program doesn’t work or you can’t stand up a virtual war room? Part of your incident response playbooks should include fallback plans for out of band communications in the event of a total disruption of productivity services at your company. Practice using these during table top exercises so everyone knows the protocols for when to fall back on them if needed.

Wrapping Up

Incidents are both exciting and stressful. It is up to the CSO to lead from the front and provide guidance to their team, executive leadership and the rest of the organization. CSO’s need to buffer their teams to allow them to focus on the task at hand, while protecting them from burnout. CSO’s also need to remember the conduct and response of the organization could be recalled in court some day so following appropriate evidence collection, notification guidelines and legal best practices are a must.