Author: Lee Vorthman

I'm a U.S. Navy veteran and the Global Chief Security Officer (CSO) at a Fortune 100 cloud company where I've built a successful security program from the ground up and have partnered with the business to increase trust and reduce risk. I have over 25 years experience across a wide variety of industries such as technology, government & defense, education and oil & gas. I hold a number of professional certifications such as, EC-Council's Certified Chief Information Security Officer (C|CISO), Digital Director's Network (DDN) Board Certified Qualified Technology Expert (QTE) and ISC(2) Certified Information Systems Security Professional (CISSP). Previously I was the Chief Technology Officer (CTO) for Civilian Agencies and Cybersecurity Initiatives at NetApp U.S. Public Sector and the Chief Information Security Office for an Oil & Gas software company. I am available for consulting and speaking opportunities. Thoughts and opinions are my own and do not represent any employer past or present.

Navigating The First 90-180 Days In A New CISO Role

Late one Friday afternoon a call comes in and you find out you landed your next CISO role. All the interview prep, research, networking and public speaking has paid off! Then it dawns on you that you could be walking into a very difficult situation over the next few months. Even though the interview answered a lot of questions, you won’t know the reality of the situation until you start. How will your expectations differ from reality? What can you do to minimize risk as you come up to speed? How should you navigate these first 90-180 days in your new role?

Prior To Starting

Let’s assume you have some time to wind down your current position and you are also going to take some time off before starting the new role. During this transition period I highly advise you reach out to your peers in the new role and start asking questions to get more detail about the top challenges and risks you need to address. Start with the rest of the C-Suite, but also get time with board members and other senior business leaders to get their perspectives. Focus on building rapport, but also gather information to build on what you learned during the interview process so you can hit the ground running.

You can also use this time to reach out to your CISO peers in your network who are in the same industry, vertical or company type to get their perspective on what they did when they first joined their company. Learn from their experience and try to accelerate your journey once you start. Keep the lines of communication open so if you run into a situation you are unsure of you can ask for advice.

Once You Start

Build Relationships

First and foremost, start building relationships as quickly as possible. Target senior leadership first, such as board members, the C-Suite and other senior leaders. Work your way down by identifying key influencers and decision makers throughout the org. Play the “new person card” and ask questions about anything and everything. Gain an understanding of the “operational tempo” of the business such as when key meetings take place (like board meetings). Understand the historical reasons why certain challenges exist. Understand the political reasons why challenges persist. Understand the OKRs, KPIs and other business objectives carried by your peers. Learn the near and long term strategy for the business. Start building out a picture of what the true situation is and how you want to begin prioritizing.

Understand the historical reasons why certain challenges exist. Understand the political reasons why challenges persist.

Plan For The Worst

Don’t be surprised if you take a new role and are immediately thrown into an incident or other significant situation. You may not have had time to review playbooks or processes, but you can still fall back on your prior experience to guide the team through this event and learn from it. Most importantly, you can use this experience to identify key talent and let them lead, while you observe and take notes. You can also use your observation of the incident to take notes on things that need to be improved such as interaction with non-security groups, when to inform the board, how to communicate with customers or how to improve coordination among your team.

Act With Urgency

Your first few months in the role are extremely vulnerable periods for both you and the company. During this period you won’t have a full picture of the risks to the business and you may not have fully developed your long term plan. Despite these challenges, you still need to act with urgency to gain an understanding of the business and the risk landscape as quickly as possible. Build on the existing program (if any) to document your assumptions, discoveries, controls and risks so you can begin to litigation proof your org. Map the maturity of security controls to an industry framework to help inform your view of the current state of risk at the company. Begin building out templates for communicating your findings, asks, etc. to both the board and your peers. Most importantly, the company will benefit from your fresh perspective so be candid about your findings and initial recommendations.

Evaluate The Security Org

In addition to the recommendations above, one of the first things I like to do is evaluate the org I have inherited. I try to talk to everyone and answer a few questions:

  1. Is the current org structure best positioned to support the rest of the business?
  2. How does the rest of the business perceive the security org?
  3. Where do we have talent gaps in the org?
  4. What improvements do we need to make to culture, diversity, processes, etc. to optimize the existing talent of the org?

Answering these questions may require you to work with your HR business partner to build out new role definitions and career paths for your org. You may also need to start a diversity campaign or a culture improvement campaign within the security org. Most importantly, evaluate the people in your org to see if you have the right people in the right places with the right skillsets.

A Plan Takes Shape

As you glide past the 90 day mark and start establishing your position as a trusted business partner, you should arrive at a point where a clear vision and strategy is starting to take shape. Use the information you have gathered from your peers, your program documentation and your observations to start building a comprehensive plan and strategy. I’ve documented this process in detail here. In addition to building your program plan you can also begin to more accurately communicate the state of your security program to senior leaders and the board. Show how much the existing program addresses business risk and where additional investment is needed. I’ve documented a suggested process here. Somewhere between your 90 and 180 day mark you should have a formalized plan for where you are over invested, under invested or need to make changes to optimize existing investment. This could include restructuring your org, buying a new technology, adjusting contractual terms or purchasing short term cyber insurance. It could even include outsourcing key functions of the security org for the short term, until you can get the rest of your program up to a certain standard. Most importantly, document how you arrived at key decisions and priorities.

Take Care Of Yourself

Lastly, on a personal note, make sure to take care of yourself. Starting a new role is hectic and exciting, but it is also a time where you can quickly overwork yourself. Remember building and leading a successful security program is a marathon not a sprint. The work is never done. Get your program to a comfortable position as quickly as possible by addressing key gaps so you can avoid burning yourself out. Try to establish a routine to allow for physical and mental health and communicate your goals to your business partners so they can support you.

During this time (or the first year) you may also want to minimize external commitments like dinners, conferences and speaking engagements. When you start a new role everyone will want your time and attention, but be cautious and protective of your time. While it is nice to get a free meal, these dinners can often take up a lot of time for little value on your end (you are the product after all). Most companies have an active marketing department that will ask you to engage with customers and the industry. Build a good relationship with your marketing peers to interweave customer commitments with industry events so you are appropriately balancing your time and attending the events that will be most impactful for the company, your network and your career.

Wrapping Up

Landing your next CISO role is exciting and definitely worth celebrating. However, the first 90-180 days are critical to gain an understanding of the business, key stakeholders and how you want to start prioritizing activities. Most importantly, build relationships, act with urgency and document everything so you can minimize the window of exposure as you are coming up to speed in your new role.

Build A Proactive Security Program By Focusing On The Fundamentals

A common topic at security conferences, CISO dinners and networking events is: “how you are preparing your program for a new and upcoming regulation?” For CISOs, this conversation is a way to exchange ideas, gather information and compare programs. Unfortunately, CISOs often express feeling underprepared for the upcoming shift in the regulatory landscape causing them to scramble to meet the new requirements. I’m sure this feeling has existed since the first CISO role was created and has been continuing through SOX, PCI-DSS, HIPAA, GDPR, DORA and CMMC. If you have ever felt your program can be better prepared for new challenges or are looking to be more proactive then this post is for you. The goal is to prepare your security program so well that any new challenges are a non-event and I fundamentally believe there are lots of things CISOs can do with their security programs to achieve this goal.

What Causes Programs To Be Reactive?

Underfunding

There are several issues that can cause a security program to be reactive and understanding the problem is the first step to over coming it. One of the most common issues with any security program is underfunding. Underfunding a security program can have ripple effects on staff, technology, risk management and compliance activities. Underfunding can be a conscious choice of the business, but more often it is the result of the CISO failing to articulate or demonstrate how the security program creates value for the business. If you can’t link your security program back to business objectives and risk then your program is falling short. When a program is underfunded it can’t innovate or gain breathing room. As a result the program will be in a perpetual state of reactivity and constantly responding to the next problem that comes up.

Poor Understanding Of Risk

But wait! You say. My program is well funded. I have the staff and technology I need, but we are still reactive. This can be for a few other reasons, such as your program has a poor understanding of the risk landscape for the business. At a basic level this means documenting your program, controls, policies, exceptions and strategy so you are in lock step with what the business is trying to accomplish. The culture of the security program should be “help me say yes to your security ask”, instead of always saying no.

Thoroughly understanding the risk landscape for the business, such as where your security program effectively manages that risk and where the business can take on more risk, is critical to helping the business operate, expand and be successful. If you haven’t mapped your program to risk then your program will always be reactive because you will have to constantly evaluate the changing business conditions each time slowing down the business and pulling resources from other areas.

Shiny Thing Syndrome

One final reason your security program can be reactive is shiny thing syndrome. This is where someone in the org (it can be you, the CTO, the CEO, etc.) is constantly enamored with new technology, things they read in Harvard Business Review or whatever they think is “cool”. This means your program will constantly lurch from thing to thing without ever gaining momentum. It also means instead of following a clear and well laid out strategy and roadmap, your program will hop around and never achieve success. They best way to counter shiny thing syndrome is with a well documented program, with a clear understanding of where you are and where you are going.

Shifting To Become Proactive

So the big question is: how do you shift your program to become proactive? We can talk about a lot of ideas like automation, AI, processes, etc., but I truly believe the core of any security program should be the fundamentals and by focusing on these fundamentals you can stop being reactive.

Don’t Practice During The Game

Here is an analogy that I like to use for what a proactive security program means. Consider you are learning to play baseball. You could go out into the field look around and hope the ball doesn’t get hit to you. Worse, you could have no idea which way to face, what to do with the glove or even how to win the game. You are just standing there… waiting to react to whatever happens and hoping to figure it out. This is a security program that hasn’t mastered the fundamentals.

However hope is not a strategy and you shouldn’t practice your skills at the game. You should practice the skills you need before the game, hone them over and over until they become instinctive allowing you to proactively shift your strategy during the game. This is what a proactive security program can do. By focusing on the fundamentals like knowing what you have, where it is and what the status is, you know you won’t have to scramble to figure these things out when a new regulation comes out or a new incident hits. By thoroughly documenting your program against an industry standard framework and continually measuring compliance and risk against that framework you will eventually master the fundamentals and become proactive. Focusing on and mastering the fundamentals allows you to continually refine your program so you can anticipate where the business, industry and regulatory environment is going. In fact, any changes in the business, industry or regulatory environment should be a non-event because your program is so robust that you can help the business take on and manage whatever new risk comes up.

Wrapping Up

Next time you are faced with a challenging incident, new regulation, new compliance activity or are at odds with the business, ask yourself if your program has mastered the fundamentals. Do an honest assessment of your program, conduct a retrospective of past activities and assess where you need to improve. Find new ways to articulate the value of your program and link your program back to business risk so you can get the funding and support you need. By mastering the fundamentals you are mastering important skills when it doesn’t matter, so you can be proactive and anticipate events before they matter.

Is Agile and DevOps Holding Back Your Security Program?

There are a variety of development models you can run into as a CISO. If you are in the defense or government sector your engineering teams will probably use something like waterfall. However, if your company produces software products or services then most likely your organization uses Agile or SAFe and DevOps as the core development methodologies. Most companies have shifted to Agile over the past decade due to the ability to break work into smaller chunks, iterate and get products to market faster. They have also shifted to DevOps, where the teams that build the thing are also the team that operates and maintains it. However, there are some disadvantages to using Agile and DevOps methodologies and as a CISO you should be familiar with the negative aspects and how to deal with them.

When DevOps Isn’t

If your engineering and development teams aren’t very mature, or if there is an imbalance between engineering, product management and sales, your Agile development and DevOps models can become skewed too far towards new feature development at the expense of all else. If this happens the security organization will find it difficult to inject security priorities into the sprints. This can cause a build up of tech debt, which can heavily impact security roadmaps.

For example, if one of your main development platforms needs to be upgraded to the latest version in order to make a new security feature available, then this will need to be prioritized and planned for within the sprints. Ideally, you can do this upgrade without any down time, but that isn’t always the case. If dev teams don’t have any slack in their sprints for operational maintenance, such as upgrades, refactoring code, improving observability, improving reliability or practicing good fundamentals, then tech debt will build up over time and the security org will find it difficult to advance their roadmap and effectively manage risk.

If you are finding your dev and engineering teams aren’t allowing enough time for DevOps activities then it is worthwhile to go back to the fundamentals and develop a RACI for who is responsible for the different tasks required to operate and maintain your products and services. Set clear expectations and metrics to measure teams. Often, when the DevOps model is broken there is a weak sense of ownership and dev teams need to be reminded that they need to maintain the things they build. You may also need to spend time with your Chief Product Officer, Chief Technology Officer or your Chief Revenue (Sales) Officer to set expectations and get their support to change the behavior of the engineering teams and how they interact with product or sales. Ultimately, good reporting that can reveal patterns of behavior, backlogs, tech debt and lack of fundamentals will go a long way to articulating the problem set and enlisting the support of the rest of the C-Level to fix the broken DevOps model.

When Agile Becomes An Excuse

Similar to the example above, Agile methodologies can become an excuse for engineering teams to not prioritize security requests such as code fixes, library updates, software patches and vulnerability remediation. Your security org may start to recognize this problem if the engineering teams never insert your activities into the sprints or if they over estimate how long it will take to complete your security request (a common delaying tactic).

When Agile becomes an excuse for dev teams it can help to have security champions or product security specialists that embed with the engineering teams to champion security priorities and make sure they get included in the appropriate sprint to hit the milestone or deadline. Often, engineering teams just don’t understand the activity and having a security expert embedded in the team can help remove the FUD (fear, uncertainty and doubt) and get the teams comfortable with including and completing security priorities. Once this muscle gets exercised enough and the engineering teams are showing consistent performance, the security champions can move on to another team that needs more help.

When dev teams are using Agile as an excuse there can be a variety of reasons such as lack of maturity or over prioritization of features over all else. This is where good metrics such as sprint velocity, capacity and estimation accuracy can help. Measuring these metrics over time and discussing them at the executive level can help identify teams that need help or are consistently under performing. This can be important for a CISO who is trying to get security priorities inserted into the team. Understanding where an engineering team is on the maturity scale and using clear metrics to report on their performance can help shift priorities as needed.

One thing you absolutely should not do as an executive team is assign engineering teams percentages of work, such as “spend 20% of your time on security activities”. This is a recipe for disaster because unless the work going into the 20% is clearly agreed on, engineering teams may use any type of work they think is security work to include in the 20%. This will cause a serious disconnect between security and the engineering teams because engineering will point out they are spending 20% of their time doing security stuff, but security won’t be getting the priorities they need. Instead of assigned percentages, it is better to have dynamic sprints where work is pulled in based on their priority with clear criteria, such as risk or revenue, that can help teams fill their sprints appropriately.

When MVP Isn’t Good Enough

Lastly, Agile is a really good methodology for building products that can iterate and develop functionality over time. It is rare that these are mission critical products and instead are products that are launched with a small feature set or core service that expands in functionality over time. However, when it comes to mission critical products, like security products, Agile and the MVP (minimum viable product) just won’t cut it. The reality is unless you have a really robust MVP, you will most likely need a GA (general availability) of the product before it has the type of functionality you need for your security program. Understanding the limitations of Agile and how to negotiate when you will get the features you need is important for organizations that build their own products internally or for organizations that partner closely with external companies to build custom products or develop products in tandem.

Whenever I interface with a team that plans to build a security product internally or with a company that is developing a custom product for my group I make sure I am extremely clear about the features and functionality I need before I will adopt the product. Often, I will compare the product in development with other existing products or I will perform extensive testing of the new product based on real world scenarios. However your security team decides to verify and validate the new thing, make sure your requirements are clear and your testing is repeatable. Document every decision and for external partners consider contract language that can protect you if the full functionality of the product or service isn’t working after you buy. Most importantly, don’t buy or renew based on promises from an external partner. If it doesn’t exist when you buy, it will be tough to get them to prioritize your requests after you hand over the check.

Wrapping Up

I’ve covered a few scenarios where Agile and DevOps can go wrong and ultimately hold back your security program. If this happens it is important to recognize the behavior and develop tactics to correct the issue. This can involve setting expectations with your C-Suite counterparts, developing clear RACI models with engineering teams or clearly documenting functionality required for a security purchase. Whatever the issue, measuring and monitoring performance will help to articulate any issues you run into and build strong relationships between security and the engineering (DevOps) teams.

Should Security Be An Approver For IT and Business Requests?

Over the course of my career I have consistently seen security in the approval chain for various IT operations and business requests, such as identity, network and even customer contracts. Having security in the approval chain may seem logical at first glance, but it can actually mask or exacerbate underlying operations issues. Having a second set of eyes on requests can make sense to provide assurance, but the question is – does it make sense for security to be an approver?

Understand The Scope of Your Security Program

First and foremost, the scope of your security program will ultimately dictate how and when security should be included in an approval process. For example, if security owns networking or identity, then it will make sense to staff an operations team to support these areas and it will make sense to have security as an approver for requests related to these functions.

It may also make sense to include security in the approval chain as an evaluator of risk for functions security doesn’t own. For example, security won’t own the overall contract, finance or procurement processes, but they should be included as an approver to make sure contract terms and purchases align to security policies and are not opening up the business to unnecessary risk. They can also be included in large financial transfers as a second set of eyes to make sure the business isn’t being scammed out of money. In these examples, security is creating good friction to slow critical processes down in a healthy way to make sure they make sense and to use time as a defense mechanism.

Other Benefits Of Security As An Approver

Including security as an approver for general IT processes can have other benefits, but these need to be weighed carefully against the risks and overall function of the business. For example, security can help provide an audit trail for approving activities that may create risk for the company. This audit trail can be useful during incident investigations to determine root cause for an incident. It can also help avoid compliance gaps for things like FedRAMP, SOC, etc. where some overall business or IT changes need to be closely managed to maintain compliance. However, creating an audit trail is not unique to the security function and, if the process is properly designed, can be performed by other functions as well.

Another advantage of including security in the approval chain is separation of duties. For example, if one team owns identity, and they are requesting elevated privileges to something, it can present a conflict of interest if they approve their own access requests. Instead, security often acts as a secondary reviewer and approver to provide separation of duties to make sure requests by a team that owns the thing isn’t approving their own requests.

Where Including Security As An Approver Can Go Wrong

The biggest issue with having security in the approval chain for most things is they typically are not the owner of those things. If approval processes are not designed properly (with other approvers besides security), then the processes can confuse ownership and give a false impression of security or compliance. For example, I typically see security as an approver for identity and access requests when security doesn’t own the identity function. At first glance, this seems to make sense because identity is a critical IT function that needs to be protected. However, if security doesn’t own the identity function (or the systems that need access approved), how do they know whether the request should be approved or not. Instead, what happens is almost all requests end up being approved (unless they are egregious) and the process serves no real purpose other than creating unnecessary friction and giving a false sense of security.

Another issue I have seen with including security in the approval chain is they effectively become “human software” where they are manually performing tasks that should be automated instead. Using security personnel as “middleware” masks the true pain and inefficiency of the process for the process owner. This takes critical human capital away from their intended purpose, is a costly solution to a problem and opens up the business to additional risk.

When Does It Make Sense For Security To Be An Approver?

I’ve listed a few examples where it makes sense for security to be an approver for things it doesn’t own – like large financial transactions, some procurement activities and security specific contract terms. However, I argue security shouldn’t be included as an approver in most IT operations processes unless security actually owns that process or thing that needs a specific security approval. Instead, the business owner of the thing should be the ultimate approver and processes should be designed to provide appropriate auditing and compliance, but without needing security personnel to perform those checks manually.

One of the few areas where it will always make sense to have security as an approver is for security exceptions. First, exceptions should be truly exceptional and not used as a band-aid for broken or poorly designed process. Second, exceptions should be grounded in business risk, while documenting the evaluation criteria, decisions, associated policies and duration. This is a core security activity because exceptions are ultimately evaluating risk and deviation from policy. I’ve written other posts about how the exception process can have other benefits as well.

Wrapping Up

Don’t fall into the trap of using your security team as a reviewer and approver for IT operations requests if security doesn’t actually own the thing related to the request. This places the security team in an adversarial position, can be a costly waste of resources, masks process inefficiencies, gives a false sense of security and can open up the business to risk. Instead, be ruthlessly focused in how your security team is utilized to make sure when they are engaged it is to perform a function that is at the core of their mission – protecting the business and managing risk.

Following SnowFlake, Cloud Providers Need To Shift To Secure By Default

In May 2024, SnowFlake experienced a data breach as a result of exposed credentials that allowed a threat actor to access customer accounts that weren’t secured with MFA. The fallout from this data breach ultimately impacted large SnowFlake customers like Ticketmaster, AutoZone, Santander Bank and AT&T. Following the announcement of the breach, SnowFlake implemented refined security measures to avoid similar incidents in the future. However, the question remains why aren’t publicly accessible cloud companies secure by default?

A Pervasive Stigma Against Security

Before we can answer the question about why companies aren’t secure by default, we need to look at the underlying psychology and motivation for companies and in particular the arguments that are made against implementing security.

Startup Mentality

One of the most pervasive (and quite frankly horrible) arguments against building in security by default is the “move fast and break things” mentality that is pervasive at startups. Startup life is a tough one and a good metaphor is you are building your parachute as you are falling. Either you succeed and live, or you burn in and cease to exist. The problem with startup mentality is when you succeed and live, most startups fail to to shift from survival mode to maturity mode as the company grows and matures.

In maturity mode, companies need to resolve all of the debt they incurred just to survive. This can be operational debt, technical debt or security debt. Unfortunately, if the survival mentality persists, this debt continues to accrue and can kill the company because the cost to continue to operate exceeds the incoming revenue.

Security Is Bad For Productivity

Another argument that frequently pops up against implementing security is the perception that security is bad for productivity. I find this argument particularly ironic since employees seem willing to tolerate bad processes, bad experiences and other examples of bad friction, yet they complain the loudest about new security controls (like being required to change their password periodically). My own opinion about this perception is employees are largely indifferent to security (or in general they think it is a good thing). However, security often results in very visible changes to processes and ways of working and it is the change that employees don’t like. They associate security with change and since change is bad, security is bad.

This is similar to the argument that security increases friction and the assumption that all friction is bad. While this assumption is not only false, it also leads to the thought process that any friction in the customer experience will lead to lost customers and sales. The reality is some friction is good and acts as safeguards to steer people towards a desired (secure) outcome.

Security As An Upsell

One last reason for failing to implement security by default is when companies choose to profit from security as an upsell (I’m looking at you Microsoft). By charging extra for the most useful or best features these companies are implicitly and explicitly placing a cost on adding in security, which is perpetuating the stigma that security is bad.

The reality is some friction is good and acts as safeguards to steer people towards the desired (secure) outcome.

Changing Perception

Leading research for high performing cultures indicates teams that are able to effectively prioritize and execute on all of their demands are the highest performing teams. In particular, teams that were able to incorporate security into their processes actually went faster and performed better, than teams who struggled with or ignored security altogether. If you want to read more on this you can check out Accelerate by Nicole Forsgren, PhD.

One other thing we can do to change this negative perception of security is to stop allowing members of the security function introduce bad friction. We have all experienced bad friction in the form of time wasters, security theater and the dreaded “no”. This behavior doesn’t help the mission of security and perpetuates the stigma against our profession.

Default Opted-In

Assuming companies can overcome the startup mentality, successfully incorporate security into their development processes and overcome the stigma of security as being bad, what should they be doing to make their products and services secure by default?

The first thing companies can do is discard the notion that increased security will inhibit sales or drive customers away. Instead, companies should use security as a selling point and configure their services to be secure by default, which means customers will need to go through some sort of initial security setup when they purchase the product or service. Customers that don’t want to do this will need to explicitly opt-out or seek alternate providers, firmly placing the liability for not meeting security best practices on their shoulders.

Enforce Security Best Practices

What security functionality should companies offer by default to their customers? Here is a short list:

Multi-Factor Authentication – including the option for OTP, secure tokens and passkeys.

Encryption – all data and transport protocols should be encrypted by default with the latest versions available.

Access Control and Detection – default deny for access to resources and make customers explicitly allow access. This includes making resources non-public by default until a customer specifies otherwise. Detect changes in the state of resources and notify customer contacts of abnormalities.

Easy Button For Fundamentals – make it easy for customers to pull a comprehensive asset inventory, control their instance or tenancy with a master account and offer simple reports for ways they can improve their security posture.

Wrapping Up

There a lots of reasons why security becomes an afterthought for companies. Often, it is because they fail to shift from survival mode to maturity mode. Other times, their culture persists the notion that security has a bad stigma and inhibits the business. Some companies even upsell customers on security functionality, which limits the adoption of security controls. The reality is companies that practice secure by design and incorporate security into development cultures move faster and outpace their competition. Companies that offer publicly available software and services need to shift their mentality to make security a default setting that is turned on at the onset of the relationship, like any other core product feature. Until companies start making security default opt-in, we will continue to experience massive data breaches like the one from SnowFlake.

Navigating The CISO Job Market

I had an interesting conversation with a friend over coffee last week and we were discussing how weird the CISO job market is right now. Even though the unemployment rates are favorable, the tech sector has actually seen slightly negative employment growth rates, which is not normal. This is largely due to a hangover effect from record hiring during COVID, but there are also other issues in the market right now that is making it challenging. The following is a review of all the things I am seeing in the tech job market right now, particularly with respect to hiring for CISO positions.

Macro Tech Environment

Let’s take a step back and look at the overall economy to understand some of the higher level factors influencing the CISO job market. First, let’s look at one end of the tech market starting with large companies. Over hiring and high compensation packages from COVID have made existing employees stay in place and so natural turnover at public companies is below average. In addition to this, fears of a recession and high interest rates have made large companies cautious about hiring new employees. When the cost to borrow money is higher, it slows growth and ultimately impacts hiring. As a result, companies are trying to get back to growth through layoffs and attrition. They are trying to artificially increase attrition by withholding bonuses, pay raises and promotions, or requiring new job requirements like return to office 4 or 5 days a week.

Second, at the other end of the market, higher interest rates impact Venture Capital (VC) and Private Equity (PE), which ultimately impacts funding for startups and subsequent job creation. With the smaller end of the market being squeezed (VC / PE) and the larger end of the market also being squeezed there aren’t a lot of options for candidates to go. Compound this with record tech layoffs over the past year and an influx of new college grads to the job market and you create a highly competitive market.

Too Much Noise

The highly competitive job market is making job candidates seeking employment and existing CISOs seeking career growth (or a change) compete with each other. The competition is causing candidates to get desperate and apply to any job that sounds sounds remotely interesting, regardless of whether or not they are qualified for the role. This is also compounded by unrealistic career expectations from past promotions, boot camps and college campuses that make people think they can qualify for the top spots, despite lacking meaningful experience. Add in how easy LinkedIn and other jobs sites have made it to apply for jobs and the net effect is to create tons of noise for recruiters and drown out qualified candidates.

I spoke to a recruiter a few weeks ago who had a job posting up for 24 hours and received thousands of applicants, of which only a handful were qualified and advanced to the interview process. Due to the volume of unqualified applicants, recruiters are only pushing through the first handful of qualified candidates and are passing on the rest of the backlog. Of all these applicants the only candidates who are getting to the first round interview phase are direct referrals.

In addition to too much applicant noise, recruiters are also finding a high number of candidates that are mis-representing themselves. Recruiters and hiring managers aren’t stupid. They can read between the lines of your career history and discern what you were really doing. If you claim to be a CISO, yet have never held more than a manager level job, then you are mis-representing yourself. The reality is, recruiters want to get paid on placing the top candidates. They are unwilling to put someone forward for a top spot that can’t back up their resume. Top candidates can not only defend their experience, but have lots of direct and indirect network connections that can vouch for them as referrals, if needed. The CISO community is a small one and people know who is the real deal and who is faking it. The sad reality is, people who mis-represent themselves are only hurting themselves by artificially placing themselves in a higher, more competitive tier than they are qualified for and as a result will never land that top spot.

Companies Are Being More Strict

High interest rates, tight budgets and a noisy applicant process mean companies are being more strict with their job requirements. More top CISO positions are requiring candidates to be on site at the corporate headquarters location at least 4 days a week. Companies are also searching globally, but hiring locally by giving preference to local candidates they don’t have to relocate and also preference to internal candidates that cost less than a retained search. CISO salaries have also slowed or stagnated with only the top spots paying top salaries. The rest are paying mid-range or low balling candidates in an attempt to get a qualified applicant at a lower price. On top of this, companies are also being more strict with degree requirements (usually a Masters for CISOs), years of experience and certifications. They are also filtering out candidates with lots of job hopping and short career stints because even though you may have carried the CISO title, it is highly unlikely you accomplished anything meaningful if you were there for less than 18 months.

The only candidates who are getting to the first round interview phase are direct referrals.

Be Cautions

Lastly, there are a few other issues that are disrupting the job market. The first is fake job postings. There are more and more reports of fake job postings that entice applicants, but are really out to steal their personal information. Be cautious and use your network to validate the postings if you are interested in applying for a CISO role (this comes back to direct referrals also).

Second, companies are leaving zombie positions out there to give the impression they have open roles, when they really don’t. They are doing this for a few reasons – they want the market and their employees to think they are hiring and growing even when budgets are tight and companies are trying to cut headcount. If you see a job posting out there for more than a few days, it is highly likely it is a zombie posting.

The last issue I want to highlight is how job sites mis-represent numbers to entice companies to spend money with them, while hurting applicants. I’m specifically referring to how LinkedIn and other job sites show metrics on “number of applicants” for job postings, when in reality these are only the number of people that have viewed the posting, not applied. I mention this because I have seen a number of posts from people who have expressed interest in a role, but have been discouraged by the “number of applicants” and as a result didn’t apply.

Maximizing Your Opportunity

Now that you understand what is going on with the job market, let’s discuss what you can do to maximize the likelihood you will land that interview and get the job.

  1. Invest in yourself – take this time to get certifications, degrees, etc. that make you competitive and demonstrate constant learning and knowledge. Invest in yourself while looking for a new role.
  2. Invest in your network – do a deep dive on your network. LinkedIn makes it easy to download your list of connections and sort them my company, degree of connection, etc. Use this analysis to understand where you have connections and where you don’t. Look for people that can connect you to individuals that hire for positions you want at your targeted companies. Find ways to meet with these people. Do the same for recruiters. Build these connections before you need them because it is always better to be a live person than a random InMail on LinkedIn.
  3. Update your resume and LinkedIn – Seriously, if you don’t know how then ask someone or pay someone. First impressions matter.
  4. Practice interview questions – Write down key accomplishments and the details for how you achieved them. Think of your weaknesses and how you turn those into strengths. Ask your network for recent interview questions and develop answers. Preparation matters and will pay off during the interview process.
  5. Stop blasting your resume into the ether – If you see a role you want to apply for, poll your network to see if you know anyone at the company or if your network knows someone at the company. Get your resume directly into the hands of the recruiter or hiring manager. Direct referrals are the only reliable way to get an interview.
  6. Get focused – Have you been attending a lot of networking events lately in the hope of meeting someone who is hiring? Consider the value of all the “networking” activities you are doing. As a single person you can’t scale to attend every event that is out there so you need to be targeted. Consider the audience of who is attending and consider the value of the event. If you are attending events that are also attended by all of your competition then you probably aren’t going to land your next job there. Instead, consider all the events and networking groups in your area, which one’s have the most likelihood of putting you in front of people that hire for your role and focus on maximizing the potential of those events.
  7. Stop directly asking people for jobs – there is no faster way to end a conversation or relationship than asking someone for a job they don’t have. Instead, if you have the opportunity to make an ask of someone, ask them to connect you with someone they know may be looking for someone with your background. Take the pressure off of them, keep the connection alive and expand your network at the same time.
  8. Consider staying put – the tech sector seems to lag what the overall economy is doing by a few years. If the tech sector is contracting it will eventually expand and get back positive employment rates. This can also give you time to build your credentials, while looking for the ideal next step.

Should Companies Be Held Liable For Software Flaws?

Following the CrowdStrike event two weeks ago, there has been an interesting exchange between Delta Airlines and CrowdStrike. In particular, Delta has threatened to sue CrowdStrike to pursue compensation for the estimated $500M of losses allegedly incurred during the outage. CrowdStrike has recently hit back at Delta claiming the airline’s recovery efforts took far longer than their peers and other companies impacted by the outage. This entire exchange prompts some interesting questions about whether a technology company should be held liable for flaws in their software and where the liability should start and end.

Strategic Technology Trends

Software quality, including defects that lead to vulnerabilities, has been identified as a strategic imperative according to CISA and the Whitehouse in the 2023 National Cybersecurity Strategy. Specifically, the United States wants to “shift liability for software products and services to promote secure development practices” and it would seem the CrowdStrike event falls into this category of liability and secure software development practices.

In addition to strategic directives, I am also seeing companies prioritize speed to market over quality (and even security). In some respects it makes sense to prioritize speed, particularly when pushing updates for new detections. However, there is clearly a conflict in priorities when a company optimizes for speed over quality for a critical detection update that causes an impact larger than if the detection update had not been pushed at all. Modern cloud infrastructure and software development practices prioritize speed to market over all else. Hyperscale cloud providers have made a giant easy button that allows developers to consume storage, network and compute resources without consideration for the down stream consequences. Attempts by the rest of the business to introduce friction, gates or restrictions on these development processes are met with derision and usually follow accusations of slowing down the business or impeding sales. Security often falls in this category of “bad friction” because they are seen as the “department of no”, but as the CrowdStrike event clearly shows, there needs to be a balance between speed and quality in order to effectively manage risk to the business.

One last trend is the reliance on “the cloud” as the only BCP / DR plan. While cloud companies certainly market themselves as globally available services, they are not without their own issues. Cloud environments still need to follow IT operations best practices by completing a business impact analysis and implementing a BCP / DR plan. At the very least, cloud environments should have a rollback option in order to revert to the last known good state.

…as the CrowdStrike event clearly shows, there needs to be a balance between speed and quality in order to effectively manage risk to the business.

What Can Companies Do Differently?

Companies that push software updates, new services or new products to their customers need to adopt best practices for quality control and quality assurance. This means rigorously testing your products before they hit production to make sure they are as free of defects as possible. CrowdStrike clearly failed to properly test their update due to a claimed flaw in their testing platform. While it is nice to know why the defect made it into production, CrowdStrike still has a responsibility to make sure their products are free from defects and should have had additional testing and observability in place.

Second, for critical updates (like detections), there is an imperative by companies to push the update globally as quickly as possible. Instead, companies like CrowdStrike should prioritize customers in terms of industry risk. They should then create a phased rollout plan that stages their updates with a ramping schedule. By starting small, monitoring changes and then ramping up the rollout, CrowdStrike could have minimized the impact to a handful of customers and avoided a global event.

Lastly, companies need to implement better monitoring and BCP / DR for their business. In the case of CrowdStrike, they should have had monitoring in place that immediately detected their products going offline and they should have had the ability to roll back or revert to the last known good state. Going a step further they could even change the behavior of their software where instead of causing a kernel panic that crashes the system, the OS recovers gracefully and automatically rolls back to the last known good state. However, the reality is sophisticated logic like this costs money to develop and it is difficult for development teams to justify this investment unless the company has felt a financial penalty for their failures.

Cloud environments still need to follow IT operations best practices by completing a business impact analysis and implementing a BCP / DR plan.

Contracts & Liability

Speaking of financial penalties, the big question is whether or not CrowdStrike can be held liable for the global outage. My guess is this will depend on what it says in their contracts. Most contracts have a clause that limits liability for both sides and so CrowdStrike could certainly face damages within those limits (probably only a few million at most). It is more likely CrowdStrike will face losses for new customers and existing customers that are up for contract renewal. Some customers will terminate their contracts. Others will negotiate better terms or expect larger discounts on renewal to make up for the outage. At most this will hit CrowdStrike for the next 3 to 5 years (depending on contract length) and then the pricing and terms will bounce back. It will be difficult for customers to exit CrowdStrike en masse because it is already a sunk cost and companies wont want to spend the time or energy to deploy a new technology. Some of the largest customers may have the best terms and ability to extract concessions from CrowdStrike, but overall I don’t think this will impact them for very long and I don’t think they will be held legally liable in any material sense.

Delta Lags Industry Standard

If CrowdStrike isn’t going to be held legally liable, what happens to Delta and their claimed lost $500M? Let’s look at some facts. First, as CrowdStrike has rightfully pointed out, Delta lagged the world for recovering from this event. They took about 20 times longer to get back to normal operations than other airlines and large companies. This points to clear underinvestment in identifying critical points of failure (their crew scheduling application) and developing sufficient plans to backup and recover if critical parts of their operation failed.

Second, Delta clearly hasn’t designed their operations for ease of management or resiliency. They have also failed to perform an adequate Business Impact Analysis (BIA) or properly test their BCP / DR plans. I don’t know any specifics about their underlying IT operations, but a few recommendations come to mind such as implementing active / active instances for critical services and moving to thin clients or PXE boot for airport kiosks and terminals. Remove the need for a human to touch any of these systems physically, and instead implement processes to remotely identify, manage and recover these systems from a variety of different failure scenarios. Clearly Delta has a big gap in their IT Operations processes and their customers suffered as a result.

Wrapping Up

What the CrowdStrike event highlights is the need for companies to prioritize quality, resiliency and stability over speed to market. The National Cybersecurity Strategy has identified software defects as a strategic imperative because they lead to vulnerabilities, supply chain compromise and global outages. Companies with the size and reach of CrowdStrike can no longer afford to prioritize speed over all else and instead need to shift to a more mature and higher quality SDLC. In addition, companies that use popular software need to consider diversifying their supply chain, implementing IT operations best practices (like SRE) and implementing a mature BCP and DR plan on par with industry standards.

What the CrowdStrike event highlights is the need for companies to prioritize quality, resiliency and stability over speed to market.

When it comes to holding companies liable for global outages, like the one two weeks ago, I think it will be difficult for this to play out in the courts without resorting to a legal tit-for-tat that no one wins. Instead, the market and customers need to weigh in and hold these companies accountable through share prices, contractual negotiation or even switching to a competitor. Given the complexity of modern software, I don’t think companies should be held liable for software flaws because it is impossible to eliminate all flaws. Additionally, modern SDLCs and CI/CD pipelines are exceptionally complex and this complexity can often result in failure. This is why BCP/DR and SRE is so important, so you can recover quickly if needed. Yes, CrowdStrike could have done better, but clearly Delta wasn’t even meeting industry standards. Instead of questioning whether companies should be held liable for software flaws, a better question is: At what point does a company become so essential that they by default become critical infrastructure?

How CIOs, CTOs and the rest of the C-Suite Can Better Support CISOs

There are a variety of reporting structures for CISOs, such as reporting to the CTO, CIO, CFO or CEO. No matter who the CISO reports to, the CISO is still an integral part of the C-Suite. Yet despite this, CISOs don’t always receive full support from the rest of their C-Suite peers, which can cause friction and open up the business to risk. In this post I’ll cover how the rest of the C-Suite can better support their CISO peers and how doing so will actually help them achieve their goals as well.

Strategic Planning

First and foremost, the CISO needs to be included in strategic planning sessions about new markets, mergers and acquisitions (M&A), divestitures, new product launches and new customer types. Each of these areas will create new security risks and regulatory requirements that can have lengthy lead times for addressing. The CISO needs to be informed about product roadmaps, new features and new technology initiatives. If the CISO and security group are left out of these strategic discussions the business could be forced to delay a new business opportunity or worse enter the new opportunity without properly managing the risks.

Master The Fundamentals

Second, CTOs and CIOs need their teams to master and execute on the fundamentals. This means things like asset inventory, logging, observability, QA, QC and operations support (event notification and cost analysis). The reality is the rest of the business needs these things and these are not problems the CISO should own, yet if they are not in place they will cripple a security program. For this reason, a lot of CISOs will try to tackle these issues, but they won’t be successful without support from the C-Suite that actually owns these functions. So, one of the best ways the CTO and CIO can support the CISO is to lead the way on the heavy lifting for these fundamentals that way the CISO can draft off of these and focus on making their security program as effective as possible to manage risk.

Accountability

Speaking of mastering the fundamentals, what we are really talking about is accountability. The rest of the C-Suite needs to hold their teams accountable for completing or resolving security issues. This could be things like resolving technical debt, completing training, fixing vulnerabilities or appropriately prioritizing security requests. If accountability isn’t enforced at the C-Suite, then the rest of the business will become siloed and ignore other initiatives across the company. This can cause security issues to pile up and open up the business to risk that will be impossible for the CISO to manage. By holding your teams accountable and partnering with the CISO function you will create a partnership that can accelerate the business instead of creating unnecessary friction.

One easy way to get visibility into what your teams are doing, so you can drive accountability, is with an exceptions process. Exceptions are a common process for a security function and it allows the security team to have escalating levels of approval based on risk. It also allows for reporting and metrics about how many exceptions a team has requested, how many have been approved and how long it takes the team to resolve an exception. This can provide other C-Suite members valuable insights into how their function is performing with respect to their security commitments and it also allows the C-Suite to drive accountability into their functions by acting as the senior executive approver for critical risks in their function.

An exceptions process doesn’t have to be just for security. The entire company can benefit from an exceptions process such as for purchasing, contracts, sales, finance and engineering. Exceptions across the company can give visibility, promote good friction and drive accountability.

Support Good Friction

There are two different types of friction in a company and we have all experienced them. Good friction exists to help slow people down to consider their actions or minimize risk. These are processes like confirming large financial transactions or requiring validation of someone’s identity before using a critical resource. Bad friction wastes people’s time and is adversarial. These are processes that are inefficient, people that exercise unnecessary control over others or people that never follow through on activities. This type of friction needs to be avoided.

The rest of the C-Suite can support the creation of good friction with respect to security and how security engages with their teams. Good friction can actually accelerate the business by front loading activities where they will take less time, instead of trying to resolve issues later in the lifecycle where they are incredibly difficult and expensive to resolve. Some examples of good friction are security checks as part of the CI/CD pipeline, like SAST, automated attack simulation, or automated compliance reviews. When the rest of the C-Suite supports good friction it will actually make everyone’s job easier and less risky.

Help Advocate For Security

Another way the rest of the C-Suite can support the CISO is by helping to advocate the value of the security function beyond being an insurance policy or compliance function. While the security function may be viewed as a cost center, it can actually drive revenue and generate value. By including the CISO in the strategic planning process, CISOs can advocate product features with customers and engage with customers in a more proactive way. CISOs can also work with the go to market and finance teams to create processes for tracking customer engagements by the security team. This can shed light into the direct and indirect ways the security function is driving revenue, which can change the perspective of the security function from simply being a cost center. Having other C-Suite members advocate and support the CISO with customer engagements, building revenue tracking and involving the security team in all phases of the business can help improve the value of security and reduce overall risk.

Cultural Change

The last area the C-Suite can help the CISO with is cultural change. The Chief People Officer or Chief HR officer can work with the CISO to create and adapt comp structures for the security team that reflects the competitiveness of the market. They can also work with the CISO to create career paths, training and job specific performance metrics for the security function. The Chief People Officer and the HR function are also critical partners for the CISO to backstop security policies and enforce these policies across the company. HR can create and enforce consequences for policy violations, such as lack of eligibility for promotion, and they can also help manage the worst offenders with termination. HR can also set incentives to reward good security behavior such as giving spot bonuses, rapid promotions or even tying bonuses to completion of key security goals.

Outside of the culture of the security function, the rest of the C-Suite can set the tone for the culture with respect to how the company should view and engage with security. In particular, the C-Suite can lay the foundation for a security first culture and hold people accountable for implementing this throughout their functions. They can also shift the culture by holding business owners accountable for the things they own. Lastly, if the rest of the C-Suite carries KPIs, OKRs or other annual performance metrics as part of their annual goals this can help cross pollinate and incentivize the entire company to execute on effectively managing risk.

Wrapping Up

Close partnership with the rest of the C-Suite is essential for the CISO to be successful. The rest of the C-Suite can support the CISO and the security function by involving the CISO in strategic planning, driving accountability, mastering the fundamentals, supporting good friction, advocating for security and helping to drive cultural change. By supporting these areas, the rest of the C-Suite will set the tone from the top and work with the CISO to govern the risk of the business in a way that allows it to eliminate bad friction, accelerate growth and remain competitive.

A CISO’s Analysis Of the CrowdStrike Global Outage

Overnight from July 18 to July 19, 2024, Windows systems running CrowdStrike ceased functioning and displayed the blue screen of death (BSOD). As people woke up on the morning of July 19th they discovered a wide reaching global outage of the consumer services they rely on for their daily lives, such as healthcare, travel, fast food and even emergency services. The ramifications of this event will continue to be felt for at least the next week as businesses recover from the outage and investors react to the realization that global businesses are extremely fragile when it comes to technology and business operations.

Technical Details

An update by CrowdStrike (CS) to the C-00000291*.sys file dated 0409UTC was pushed to all customers running CS Falcon agents. This file was corrupt (reports indicate a null byte header issue) and when Windows attempted to load this file it crashed. Rebooting the impacted systems does not resolve the issue because of the way CS Falcon works. CS Falcon has access to the inner workings of the operating system (kernel) such as memory access, drivers, and registry entries that allow CS to detect malicious software and activity. The CS Falcon agent is designed to receive updates automatically in order to keep the agent up to date with the latest detections. In this case, the update file was not properly tested and somehow made it through Quality Assurance and Quality Control, before being pushed globally to all CS customers. Additionally, CrowdStrike customers are clearly running CS Falcon on production systems and do not have processes in place to stage updates to CS Falcon in order to minimize the impact of failed updates (more on this below).

Global Impact

This truly is a global outage and the list of industries is far reaching attesting to the success of CS, but also the risks that can impact your software supply chain. As of Monday, Delta airlines is still experiencing flight cancellations and delays as a result of impacts to their pilot scheduling system. The list of impacted companies can be found here, here and here, but I’ll provide a short list as follows:

Travel – United, Delta, American, major airports

Banking and Trading – VISA, stock exchanges

Emergency & Security Services – Some 911 services and ADT

Cloud Providers – AWS, Azure

Consumer – Starbucks, McDonalds, FedEx

Once the immediate global impact subsides, there will be plenty of finger pointing at CrowdStrike for failing to properly test an update, but what this event clearly shows is a lack of investment by some major global companies in site reliability engineering (SRE), business continuity planning (BCP), disaster recovery (DR), business impact analysis (BIA) and proper change control. If companies were truly investing in SRE, BCP, DR and BIA beyond a simple checkbox exercise, this failed update would have been a non-event. Businesses would have simply executed their BCP / DR plan and failed over, or immediately recovered their critical services to get back up and running (which some did). Or, if they are running proper change control along immutable infrastructure they could have immediately rolled back to the last good version with minimal impact. Clearly, more work needs to be done by all of these companies to improve their plans, processes and execution when a disruptive event occurs.

Are global companies really allowing live updates to mission critical software in production without going through proper testing? Or even better, production systems should be immutable, preventing any change to production without being updated in the CI/CD pipeline and then re-deployed. Failed updates became an issue almost two decades ago when Microsoft began patch Tuesday. Companies quickly figured out they couldn’t trust the quality of the patches and instead would test the patches in staging, which runs a duplicate environment to production. While this may have created a short window of vulnerability, it came with the advantages of stability and uninterrupted business operations.

Modern day IT Operations (called Platform Engineering or Site Reliability Engineering) now design production environments to be immutable and somewhat self healing. All changes need to be updated in code and then re-pushed through dev , test and staging environments to make sure proper QA and QC is followed. This minimizes impact from failed code pushes and will also minimize disruption from failed patches and updates like this one. SRE also closely monitors production environments for latency thresholds, availability targets and other operational metrics. If the environment exceeds a specific threshold then it throws alerts and will attempt to self heal by allocating more resources, or by rolling back to the previous known good image.

Ramifications

Materiality

Setting aside maturity of business and IT operations, there are some clear ramifications for this event. First, this had a global impact to a wide variety of businesses and services. Some of the biggest impacts were felt by publicly traded companies and as a result these companies will need to make an 8K filing with the SEC to report a material event to their business. Even though this wasn’t a cybersecurity attack, it was still an event that disrupted business operations and so companies will need to report the expected impact and loss accordingly. CrowdStrike in particular will need to make an 8K filling, not only for loss of stock value, but for expected loss of revenue through lost customers, contractual concessions and other tangible impacts to their business. When I started this post Friday of the even, CS stock was down over 10% and by Monday morning they were down almost 20%. The stock has started to recover, but that is clearly a material event to investors.

Greater Investment In BCP / DR & BIA

Recent events, such as this one and the UHC Change Healthcare ransomware attack, have clearly shown that some business are not investing properly in BCP / DR. They may have plans on paper, but plans still need to be fully tested including rapidly identifying service degradation and implementing recovery operations as quickly as possible. The reality is this should have been a non-event and any business that was impacted longer than a few hours needs to consider additional investment in their BCP / DR plan to minimize the impact of future events. CISOs need to work with the rest of the C-Suite to review existing BCP / DR plans and update them accordingly based on the risk tolerance of the business and desired RTO and RPO.

Boards Need To Step Up

During an event like this one boards need to take a step back and remember their primary purpose is to represent and protect investors. In this case, the sub-committees that govern technology, cybersecurity and risk should be asking hard questions about how to minimize the impact of future events like this and consider if the existing investment in BCP / DR technology and processes is sufficient to offset a projected loss of business. This may include more frequent reports on when the last time BCP / DR plans were properly tested and if those plans are properly accounting for all of the possible scenarios that could impact the business such as ransomware, supply chain disruption or global events like this one. The board may also push the executive staff to accelerate plans to invest in and modernize IT operations to eliminate tech debt and adopt industry best practices such as immutable infra or SRE. The board may also insist on a detailed analysis of the risks of the supply chain, including plans to minimize single points of failure, while limiting the blast radius of future events.

Negative Outcomes

Unfortunately, this event is likely to cause a negative perception of cybersecurity in the short term for a few different reasons. First, the obvious business disruption is one people will be questioning. How, is it a global cybersecurity company is able to disrupt so much with a single update? Could this same process act as an attack vector for attackers? Reports are already indicating that malicious domains have been set up to look like the fix for this event, but instead push malware. There are also malicious domains that have been created for phishing purposes and the reality is any company impacted by this event may also be vulnerable to ransomware attacks, social engineering and other follow on attacks.

Second, this event may cause a negative perception of automatic updates within the IT operations groups. I personally believe this is the wrong reaction, but the reality is some businesses will turn off the auto-updates, which will leave them more vulnerable to malware and other attacks.

The reality is this should have been a non-event and any business that was impacted longer than a few hours needs to consider additional investment in their BCP / DR plan to minimize the impact of future events.

What CISOs Should Do

With all this in mind, what should CISOs do to help the board, the C-Suite and the rest of the business navigate this event? Here are my suggestions:

First, review your contractual terms with 3rd party providers to understand contractually defined SLAs, liability, restitution and other clauses that can help protect your business due to an event caused by a third party. This should also include a risk analysis of your entire supply chain to determine single points of failure and how to protect your business appropriately.

Second, insist on increased investment in your BIA, BCP and DR plans including designing for site reliability and random events (chaos monkey) to proactively identify and recover from disruption, including review of RTO and RPO. If your BCP / DR plan is not where it needs to be, it may require investment in a multi-year technology transformation plan including resolving legacy systems and tech debt. It may also require modernizing your SDLC to shift to CI/CD including dev, test, staging and prod environments that are tightly controlled. The ultimate goal will be to move to immutable infrastructure and IT operations best practices that allow your services to operate and recover without disruption. I’ve captured my thoughts on some of the best practices here.

Third, resist the temptation to over react. The C-Suite and investors are going to ask some hard questions about your business and they will suggest a wide range of solutions such as turning off auto-patches, ripping out CS or even building your own solution. All of these suggestions have a clear tradeoff in terms of risk and operational investment. Making a poor, reactive, decision immediately after this event can harm the business more than it can help.

Finally, for mission critical services consider shifting to a heterogeneous environment that statistically minimizes the impact of any one vendor. The concept is simple, if you need an security technology to protect your systems consider purchasing multiple vendors that all have similar capabilities, but will minimize the impact of your business operations if one of them has an issue. This obviously raises the complexity and operational cost of your environment and should only be used for mission critical or highly sensitive services that need to absolutely minimize any risk to operations. However, this event does highlight the risks of consolidating to a single vendor and you should conduct a risk analysis to determine the best course of action for your business and supply chain.

Wrapping Up

For some companies this was a non-event. Once they realized there was an outage they simply executed their recovery plans and were back online relatively quickly. For other companies, this event highlighted lack of investment in IT operations fundamentals like BCP / DR or supply chain risk management. On the positive side, this wasn’t a ransomware or other cybersecurity attack and so recovery is relatively straightforward for most businesses. On the negative side, this event can have negative consequences if businesses over react and make poor decisions. As a CISO, I highly recommend you take advantage of this event to learn from your weaknesses and make plans to shore up aspects of your operations that were sub-standard.

Tips For Managing Anxiety, Stress, Burnout and Mental Health

CISOs have been in the hot seat lately, particularly related to personal liability, increasing regulatory pressure and a shifting technological landscape. Compound these macro issues with the internal demands CISOs deal with such as incidents, budget cuts or political battles for relevancy and it is no wonder CISOs are struggling with mental health issues. All of these macro and micro issues weigh on a CISO and can cause anxiety, stress or burnout. In this post I’ll discuss methods for identifying, avoiding and managing anxiety, stress and burnout over your career as a CISO.

Maturing Your Program

One of the first areas you should evaluate is the maturity of your security program and in particular how to shift from being reactive to proactive. When you are reactive all the time you are waiting for things to happen instead of taking control of your situation and removing as much uncertainty as possible. This can be as simple as establishing playbooks, documenting your program or creating better detections. Perform an honest evaluation about the maturity of your security program and work with the rest of the business to identify areas you can standardize, automate or improve so you can become more proactive. Here are a few examples to give you some ideas:

  1. Standardize responses to customer questionnaires so you don’t have to continually answer the same questions. Publish the answers in a location where sales teams can find them and provide them to customers as needed.
  2. Standardize security and privacy contractual language. By standardizing language you reduce the need to negotiate, red line and haggle back and forth about terms, which can cause a big drain on your team.
  3. Reduce false positives by improving detections or tuning rule sets. By reducing false positives you will reduce the number of times your team gets called up to help solve an issue that never should have existed in the first place. Continually responding to non-issues can rapidly result in burnout and even dull your team’s response mechanism causing poor performance during a true incident.
  4. Hire more people to provide proper coverage so people aren’t getting woken up or working after hours.
  5. Create standard playbooks so everyone is following the same process when there is an incident.
  6. Conduct regular table tops to test your ability to respond. Continually iterate and improve so a true incident is a non-event.
  7. Meticulously document your security program so compliance and audit activities are a non-event.

These are just a few examples of ways to shift from being reactive to proactive. By documenting and evaluating your program against a standard framework you can identify opportunities for maturing your program, which will help shift your program to be more proactive. By being more proactive you will naturally reduce stress, anxiety and burnout among yourself and your team members.

There are a few other ways you can reduce stress and anxiety in your security program. First, you can delegate more to your staff allowing them growth opportunities, while allowing you much needed down time. Second, build rapport with the rest of the business by documenting decisions on risk and getting the business owners to sign off on those decisions. Reduce unnecessary friction in your security program and establish credibility with your peers. By creating allies across the rest of the business you can shift from an adversarial relationship to a trusted advisor of the business and this will naturally reduce stress and anxiety.

Lastly, as a CISO, you can reduce personal stress and anxiety by thoroughly documenting your program and getting major decisions in writing. This can help protect you from personal liability by reducing the likelihood you will become a scapegoat and will provide a record of accountability for decisions in case you end up getting sued or held accountable by some other regulatory action. You should also request coverage on the D&O liability policy and make sure you have legal coverage if needed because both will give you piece of mind.

By creating allies across the rest of the business you can shift from an adversarial relationship to a trusted advisor of the business and this will naturally reduce stress and anxiety.

Nosce Te Ipsum (Know Thyself)

Once you have a plan in place to mature your security program and reduce personal liability, the next area to work on is yourself. The normal cadence of life, including our interactions with co-workers, family members and friends can all cause stress, anxiety and burnout. It is important to acknowledge that we are humans and these are normal feelings, but by performing regular self reflection we can identify the causes of these feelings and attempt to manage or eliminate them. Here are a few things to think about:

  1. Where do you have agency (ownership and control) in your job and life? Worrying about things you can’t control or trying to control things beyond your control can increase stress and anxiety, which will lead to burnout.
  2. Identify triggers. What triggers you? Does someone at work constantly set you off? Do you get really irritable when you haven’t eaten? Does waiting for an impending scenario at work (like an incident) eat away at you? Identify the things that cause you stress and anxiety and work to reduce them. This can be a simple conversation with someone to help them understand how their behavior is impacting you, or changing your role to eliminate situations that place you in stressful or anxiety causing situations. Most importantly, by identifying your triggers, you can attempt to control your feelings of stress and anxiety instead of simply reacting and ultimately burning out.

There are a few other things you can do to help reduce anxiety and stress throughout your day. First, set boundaries and disconnect. Do you work through lunch? Do you answer emails at all hours of the night? Block off personal time to allow time to process and reflect. Delegate after hours decisions to your staff so you aren’t answering emails at all hours of the night. Take a vacation and leave your work devices at home. Once you start setting boundaries you will normalize and set the example for your team so they can manage their anxiety and stress as well.

Another way to reduce personal anxiety and stress is to cognitively offload. Do you carry around a lot of decisions and thoughts in your head? Are you constantly trying to remember lists or activities you need to do? Start keeping a running notepad and to do list on your phone. Write down things you need to do or ideas you have so you don’t lose them. Stop carrying all these activities around in your head so you aren’t constantly trying to remember everything. Offload these things to your phone, give your self a break and reduce mental stress.

One of the last things you can do for your personal mental health is change your perspective. Western society is extremely negative and this can be compounded by social media, the news or daily interactions with others. It can seem natural to complain about situations or to view things in a negative light, but this way of thinking has a lensing effect that will impact our entire lives. Instead, be present, be grateful, stop judging things and stop complaining. Look at the positive side of situations and work hard to shift your perspective. This positive mental attitude can have a tremendous effect on reducing stress and anxiety in our lives.

Manage Your Health

Personal health is another area that can contribute to or cause stress and anxiety. Personal health is directly related to your work situation (creating a mature program) and knowing yourself in the ways I listed above. However, there are a lot of things you can do with your personal health that can help manage stress and reduce anxiety. Here is a list of things you can do:

  1. Get sunlight and exercise (take walking meetings, work outside, block off time to exercise). These are all natural stress reducers.
  2. Fix your sleep
  3. Understand your family history with stress and anxiety
  4. Reduce screen time
  5. Get off social media and stop reading the news
  6. Practice breathing exercises
  7. Explore a creative, non-work related hobby (such as art, music, etc.)
  8. See your doctor and discuss your health (including mental health), get your hormones tested
  9. Improve your eating habits (reduce processed foods)
  10. Reflect on the positive things that happened in your day and mentally prepare for the positive things you want to accomplish tomorrow
  11. Practice gratitude

Wrapping Up

There can be stigma around discussing mental health issues, particularly stress and anxiety. However, by not discussing it openly we are normalizing behaviors and feelings that impact our performance and personal health. There are a lot of ways you can reduce the amount of anxiety and stress you have on a daily basis, such as becoming more proactive, delegating, identifying triggers, cognitively offloading and improving your overall personal health. By taking an active role in our exposure to stress and anxiety we reduce the likelihood of burnout and more effectively manage our daily existence. Small changes to our daily lives, routines and interactions can have an exponential impact towards improving our mental health, which will ultimately make us better CISOs.