We Are Drowning In Patches (and what to do about it)

Last week I had an interesting discussion with some friends about how to prioritize patches using criticality and a risk based approach. After the discussion I starting thinking about how nice it would be if we could all just automatically patch everything and not have to worry about prioritization and the never ending backlog of patches, but unfortunately this isn’t a reality for the majority of organizations.

Whats the problem?

There are several issues that create a huge backlog of patches for organizations.

First, let’s talk about the patching landscape organizations need to deal with. This is largely spit into two different areas. The first area is operating system (OS) and service patches. These are patches that are released periodically for the operating systems used by the business to run applications or products. Common operating systems for production workloads will be either Windows or Linux and will have stability, security or new feature patches released periodically.

Second, there are patches for software libraries that are included in the software and applications developed by your business. Typically these are lumped into the category of 3rd party libraries, which means your organization didn’t write these libraries, but they are included in your software. 3rd party library security vulnerabilities have become a huge issue over the last decade (but thats a blog post for another day).

These two patch types, OS and 3rd party library patches, require different approaches to discover, manage and remediate, which is the first challenge for auto patching. When combined with the volume of new vulnerabilities being discovered, large heterogeneous environments and the need to keep business critical applications available, keeping your assets patched and up to date becomes a real challenge.

Why isn’t auto patching a thing?

Well it is, but…

There are a few challenges to overcome before you can auto-patch.

Stability and Functionality

First, both operating system and 3rd party library patches need to be tested for stability and functionality. Usually, patches fix some sort of issue or introduce new features, but this can cause issues in other areas such as stability or functionality. It can be a complex process to roll back patches and restore business critical applications to a stable version, which is why most businesses test their patches in a staging environment before rolling them out to production. Cash is king and businesses want to minimize any disruption to cash flow.

Investment and Maturity

It is possible to automate testing for stability and functionality, but this requires a level of maturity and investment that most organizations haven’t achieved. For example, assuming your staging environment is a mirror image of your production environment (it is right?), you could auto apply the patches in staging, automatically check for stability and functionality over a set period of time and then roll those updates to production with minimal interaction. However, if your environment requires reboots or you have limited resources, patching may require down time, which could impact making money.

In order to have an environment that can support multiple versions, seamless cut over, proper load balancing, caching, etc. requires significant investment. Typically this investment is useful for keeping your products functioning and making money even if something goes wrong, but this investment can also be used to buffer maintenance activities such as patching without disruption.

Software Development Lifecycle

The last section assumes a level of software development maturity such as adoption of Agile development processes and CI/CD (continuous integration / continuous delivery). However, if your engineering group uses a different development process such as Incremental or Waterfall, then patching may become even more difficult because you are now competing with additional constraints and priorities.

What are some strategies to prioritize patching and reduce volume?

If your business runs products that aren’t mission critical, or you simply can’t justify the investment to operate an environment without down time, then auto patching probably isn’t a reality for you unless you are Austin Powers and like to live dangerously. For most organizations, you will need to come up with a strategy to prioritize patching and reduce the volume down to a manageable level.

Interestingly, this problem space has had a bunch of brain power dedicated to it over the years because it resembles a knapsack problem, which is a common problem space in mathematics, computer science and economics. Knapsack problems are problems where you have a finite amount of a resource (space, time, etc.) and you want to optimize the use of that resource to maximize some sort of requirement (like value). In the case of patching, this would mean applying the largest volume of the highest severity patches in a fixed time period to realize the maximum risk reduction possible.

Critical Assets First

Staying in the knapsack problem space, one strategy is to start with your most critical assets and apply the highest severity patches until you reach your threshold for risk tolerance. This requires your organization to have an up to date asset inventory and have categorized your assets based on business criticality and risk. For example, let’s say you have two applications at your business. One is a mission critical application for customers and generates 80% of your annual revenue. The other application provides non-mission critical functionality and accounts for the other 20% of revenue. Your risk tolerance based on your company policies is to apply all critical and high patches within 72 hours of release. In this example you would apply all critical and high patches to the mission critical application as quickly as possible (assuming other requirements are met like availability, etc.).

Guard Rails and Gates

Another strategy for reducing volume is to have guard rails or gates as part of your software development lifecycle. This means your engineering teams will be required to pass through these gates at different stages before being allowed to go to production. For example, your organization may have a policy that no critical vulnerabilities are allowed in production applications. The security organization creates a gate that scans for OS and 3rd party library vulnerabilities whenever an engineering team attempts to make changes to the production environment (like pushing new features). This way the engineering team needs to satisfy any vulnerability findings and apply patches at regular intervals coinciding with changes to production.

Wrapping Up

With the proliferation of open source software, the speed of development and the maturity of researchers and attackers to find new vulnerabilities, patching has become an overwhelming problem for a lot of organizations. In fact, it is such a big problem CISA and the Executive Order On Improving The Nation’s Cybersecurity list software patches and vulnerabilities as a key national security issue. I’ve outlined a few strategies to prioritize and reduce the volume of patches if your organization can’t afford the investment to absorb downtime without disruption. However, no matter what strategy you choose, all of them require strong fundamentals in asset inventory, asset categorization and defined risk tolerance. While these investments may seem tedious at first, the more disciplined you are about enforcing the security fundamentals (and engineering maturity), the less you will drown in patches and the closer your organization will come to the reality of auto-patching.

Are Phishing Campaigns Worth It?

Phishing campaigns are often touted as a complementary exercise to security training as a way to measure training effectiveness. The thought is, if your training is effective, users will be less likely to fall for and click on phishing emails, which will correlate to a decrease in the number of phishing incidents at your company. This sounds great in theory, but phishing campaigns have a lot of downsides that need to be considered before you hit the send button.

What Is Phishing?

The Cybersecurity and Infrastructure Security Agency (CISA) defines phishing as:

“a form of social engineering where malicious actors lure victims (typically via email) to visit a malicious site or deceive them into providing login credentials.”

What does this mean for a CISO in practical terms? It means your employees will constantly receive emails that look legitimate, but are actually scams. They are trying to get your employees to click on links in the email so they can steal credentials, install malware, get access to sensitive data, or steal money. Phishing campaigns are often one of the first methods attempted in a more targeted attack that can use the phished credentials to allow the attacker to gain a foothold into your environment.

What Are The Common Defenses Against Phishing?

User Awareness Training

One of the most effective ways to counter the threat of phishing attacks is to educate your users. Regular user awareness training on how to recognize and take action against phishing emails has proven to be highly effective. Why? Phishing is trying to trick your users into performing an action they wouldn’t normally perform. This is a form of phycological or social engineering and the best way to instill the proper mindset in someone is through regular training. This training should test for understanding and the ability for users to recognize and report phishing emails. When in doubt, report it and delete it.

DMARC, SPF and DKIM

Domain-based Message Authentication, Reporting and Conformance (DMARC), Sender Policy Framework (SPF) and Domain Keys Identified Mail (DKIM) are technologies that can be implemented by businesses to verify the sender of incoming email and authenticate that incoming messages are valid (not spoofed). Technologies such as SPF allow domains to publish lists of IPs and servers that will send emails, and DKIM allows domain owners to digitally sign emails coming from their domain so recipients can cryptographically validate the messages. SPF and DKIM are forms of authentication in the email world and help prevent spammers from sending mail on domains they don’t own.

DMARC is the enforcement arm in the email world. It takes the output from SPF and DKIM and takes action. This action can be configured based on your organizational preferences, but typical actions for messages that fail SPF or DKIM checks are to deliver the message, mark as SPAM or reject the message entirely. When configured properly, all three of these technologies will help filter and reduce potential phishing emails that make their way into your user’s inboxes.

MFA

Another technology that is critically important to protect against phishing attacks is to enable Multi-Factor Authentication (MFA). This is another form of defense that will protect your user accounts if a phishing email makes it through the filters and your user clicks on the phishing link in the email.

For example, a typical phishing email may impersonate a legitimate business website that requires authentication. The formatting, graphics and appearance may all look exactly the same. The only way to tell the email is a phishing email is by looking at the sender domain or email headers to detect subtle variances in spelling or formatting. If a user falls for this phishing email, clicks on it and enters their username and password, MFA will help prevent their credentials from being fully compromised. Yes, the user will need to have their password changed, but MFA such as one time passwords, tokens or passkeys, will prevent the attackers from using the phished credentials.

What Are Phishing Campaigns?

Phishing campaigns are controlled email campaigns sent by your own organization or a contracted third party to send fake phishing emails to your users to test how many open and click on the phishing links. Phishing campaigns allow organizations to directly test how well their user awareness training is working to recognize and avoid phishing attacks. Phishing campaigns can be stand alone events or they can be tied into other security testing like penetration tests.

What Are the Downsides To Phishing Campaigns?

Phishing campaigns, while popular, have questionable morality and effectiveness for a few reasons.

  1. The primary method of business communication is email. Phishing campaigns are teaching users to mistrust and in some cases stop using email for business purposes. Security organizations should find ways to support and protect the business without unnecessarily impeding it and for this reason I believe phishing campaigns are counter to the mission of an effective security organization.
  2. The top businesses have cultures that support and encourage psychological safety. Being able to respectfully speak your mind, have support from your colleagues and feel valued are all important aspects for job satisfaction and effectiveness. Phishing campaigns go against the idea of psychological safety. They attempt to trick your users into clicking on emails with questionable tactics such as promising bonuses, legitimate business purposes or even funny cat videos.
  3. One large problem with phishing campaigns is they tend to have punitive outcomes. Anyone that falls for the phishing email gets sent to remedial training or may be given a reduced set of permissions for a period of time. These punitive actions punish users for using their primary method of communication, destroy the concept of psychological safety and discourage productivity.
  4. Speaking of productivity, I see a lot of metrics about the percentage of users that clicked on phishing campaign emails along with targets to reduce those numbers after sending people to remedial training. What I don’t see are metrics on the impact the campaign has to productivity. How much longer will it take the person on finance to do their job now that she doesn’t trust anything in her email? How much longer will it take IT support to resolve the help desk ticket when they have been scolded repeatedly for falling for phishing emails? These metrics unfortunately are overlooked or not even captured. Security programs should ground their activities against the overall business strategy and make sure their programs are generating true value for the business that is measurable in the form of reduced risk as a tradeoff to other areas of the business.

A More Effective Solution

Whenever someone asks me for my thoughts about phishing campaigns I tell them honestly that I am not a fan. I’ve been on penetration testing teams that have crafted emails as part of phishing campaigns and I’ve seen the effect it has on users. I think there is a better way.

A lot of this post has gathered inspiration from various sources, but one of the main sources is the Cybersecurity and Infrastructure Agency (CISA). In October 2023, CISA the FBI and the NSA published a joint article on guidance for stopping phishing attacks. You can read their excellent recommendations here. Their article supports my sentiments here because one thing that is not in their recommendations is conducting a phishing campaign against your own users.

What are my recommendations?

  1. Conduct proactive training that tests not only comprehension, but the ability to accurately recognize phishing emails. Give employees the skills to look at email headers and give them the latitude to report suspicious emails or delete them altogether. Accept that email will be a slower and less trusted form of communication and even prevent the use of email for critical business functions (like contracting or financial activities). This training should have hands on practicals that gives the security function and senior leaders confidence they have trained their users to the best of their abilities to minimize the risk.
  2. Put controls in place to protect your employees. DMARC, DKIM, SPF and MFA can protect your users. Endpoint protection, monitoring and logging, ingress and egress filtering, etc. can all provide defense in depth to stop phishing attacks from being successful. The point here is, a comprehensive and well executed security program is one of your best defenses against phishing attacks.
  3. Employees that fall for real world phishing emails should be given a second chance. Assume good intent here. Most employees will recognize when they have done something bad and will feel guilty about it. They will punish themselves so the organization should support them, offer them additional training and help them get back to doing their job. Having proper security controls in place can help minimize the impact of your employees clicking on phishing emails.
  4. As a last resort, I will recommend some sort of punitive action, but this should not be the default and should be used sparingly. For users that just don’t get it and are repeat offenders they should face disciplinary action such as termination or reduced job responsibilities. This ties into organizations that evaluate how well employees support and uphold the security objectives of the organization. Repeated violations of Acceptable Use Policies (AUP) should fall under an HR/Legal action that minimizes the risk of the employee to the business.

Wrapping Up

Phishing is a form of social engineering and is a real business risk. It can lead to credential compromise, malware infections or Business Email Compromise (BEC) resulting in real business loss. A well rounded and comprehensive security program will help counter the threat of phishing attacks through comprehensive security controls and processes. Most importantly, I recommend security teams remove phishing campaigns from their tool chest and instead use proactive techniques to educate users, while protecting the business with a defense in depth approach.

Using Exceptions As A Discovery Tool

Security exceptions should be used sparingly and should be truly exceptional circumstances that are granted after the business accepts a risk. In mature security programs the security exceptions process is well defined and has clear criteria for what will and will not meet the exception criteria. In mature programs exceptions should be the exception, not the norm. However, in newer security programs exceptions can be a useful tool that provides discovery as well as risk acceptance.

Maturing A Security Program

One of the first things a new CISO will need to do is understand the business and how it functions. As part of this process the CISO will need to take an inventory of the current state of things so he or she can begin to form a strategy on how to best manage risk. As a new CISO your security program may not have well defined security policies and standards. As you begin to define your program and roll out these policies, the exception process can be a valuable tool that gives the perception of a choice, while allowing the security team to uncover areas of the business that need security improvement. Over time, as the business and security program mature, the CISO can gradually deny any requests to renew or extend these exceptions.

Rolling Out A New Security Process

Another area that is useful to have an exceptions process is when rolling out a new security process. For example, if you are rolling out a new process that will require teams to perform SAST and DAST scanning of their code and fix vulnerabilities before going into production, then allowing security exceptions during the initial rollout of the process can be useful to allow teams more time to adapt their development processes to incorporate the new security process. Allowing exceptions can foster good will with the development team and allow the security function visibility into the behavior and culture of the rest of the business. This can allow the security function and development team the opportunity to collaborate together with the ultimate goal of removing any exceptions and following the process to reduce risk to the business.

Tackling Security Tech Debt or Shadow IT

A common maturity evolution for companies is the elimination of shadow IT. The security function can assist with the elimination of shadow IT by creating an exception process and allowing an amnesty period where the business is allowed to continue to operate their shadow IT as long as it is declared. In reality you are giving the business the perception that they will be granted an exception when they are really giving the security function visibility into things they wouldn’t otherwise know about. This can be a useful tool to discover and eliminate policy exceptions as long as it is used sparingly and with good intent (not punitively).

Documentation Is Key

No matter how you choose to use exceptions within your security program there are a few best practices to follow.

  1. Exceptions should be truly exceptional. If you do grant one for discovery purposes make sure there is a plan to close the exception. Exceptions shouldn’t be the rule and they shouldn’t be expected. Sometimes the rest of the business just needs someone to tell them no.
  2. Time box the exception. Don’t just grant an exception without some sort of end date. The business needs to know an exception is temporary and there should be a well defined plan to make improvements and close the exception. The security team should grant a reasonable amount of time to execute that plan, but it shouldn’t be a never ending story.
  3. Review often. Security exceptions should be reviewed often. Part of your security program should review the open exceptions, which ones are ending, if there are patterns where there are lots of similar exceptions and if there are teams who request a high volume of exceptions. Reviewing exceptions gives you insight into how well security processes and controls are working. It also gives you insight into which parts of the business need help.
  4. Require the business owner to sign off. The reality of a well run security program is the business ultimately owns the decision if they want to accept a risk or not. The CISO makes a recommendation, but they don’t own the business systems or processes. As a result, the security exception process should require the business owner to sign off on any exception. This will ensure there is documentation that they were made aware of the risk, but this can also act as a visibility tool for the business owner into their own teams. I’ve often found a business leader is not always aware of what their teams are doing at the tactical level and the exceptions process can provide them the opportunity to check their team and correct behavior before it gets to the CISO.

Wrapping Up

The exception process can be a valuable tool for discovery of hidden risk throughout the business. By offering an amnesty period and giving the perception of flexibility, the security team can foster good will with the business while gaining valuable visibility into areas that may be hidden. The exception process also is a valuable tool for the security program to document risk acceptance by the applicable business owner, but can also provide business owners visibility into how well their team is meeting security requirements. Lastly, as the security program matures, the security team can gradually require the business to close down the exceptions by improving their security posture.

Defining Your Security Front Door

A key skill for any security program is to partner with and enable the business to be successful. CISOs need to ensure their security teams are approachable, reasonable and most importantly balancing the needs of the business against potential security risks. While security teams exist to help protect the business, they don’t own the business systems or processes and as a result need to adopt an advisory or consultative role with the rest of the business to ensure success.

With that in mind, the way the rest of the business finds and engages with the security team can create a good first impression or can set the tone for a difficult interaction. Think of a house that has great curb appeal – it feels inviting and gives the impression that the owners take good care of their property. The same concept exists for the security program, which I call the Security Front Door.

The Front Door Concept

The security front door defines how the rest of the business engages with and interacts with the security team. The front door can be a confluence page, slack channel with pinned information, or some place that is easily discoverable and accessible. Your security front door should clearly lay out information and resources so the rest of the business can either self serve or easily request and receive help when needed.

What Should Be In Your Front Door?

The front door for your security program should include ways to perform the most commonly requested actions from the security team. For example, you probably want really clear ways to request the following:

  • Report an incident
  • Request vulnerability remediation help
  • Request an exception
  • Request an architectural review
  • Dashboards
  • Discover documentation for policies and processes
  • Other – a general way to request help for anything else

The front door is not just a way to make a good first impression and enable the business, but when set up correctly it can actually offload the security team and help the business move faster.

Wrapping Up

The front door is a great way to engage with the business to help them move faster, find information and request assistance from the security team. When done correctly it can allow the rest of the business to self serve and can actually offload the security team by reducing the volume of requests that come in. Setting up the security front door may require a lot of up front work, but by understanding the rest of the business, their key pain points and most commonly requested security asks, you can design a front door that will be a win-win for everyone.

Annual Planning For CISOs

The beginning of the year is a popular time for making personal resolutions, which can focus on health, finance or love. While the beginning of the year is a popular time to set resolutions, really what we are talking about is setting goals to improve ourselves. I’m a huge proponent of setting personal goals for the year because it gives focus and purpose to your actions. The beginning of the year is also a great time to review the annual goals of your security program to set your focus and establish priorities. Annual planning has several objectives that CISOs need to consider and include in their process and I’ll cover them in the rest of this post.

Strategic Planning (Strat Planning)

Strategic, or “strat” planning as it is sometimes called, looks at where the business and your organization want to be over a long term time period. Something like 18 months to 5 years is typical in strat planning. The planning session should include discussion of the one or more of the following macro level business topics:

  • Market forces and opportunities
  • Industry trends
  • Regulatory and legal landscape
  • Competition
  • Customer sentiment, goals, etc.
  • Economic and financial environment
  • Geo-political climate
  • Technology trends and latest research

This discussion could be part of a SWOT analysis (Strengths, Weaknesses, Opportunities, Threats), but the goal is to understand where your business is and where you want it to go in the long term.

Align The Security Program

Once the business has a strategic plan, the CISO should conduct a similar planning exercise for where they want the security program to be. These are sometimes called “North Stars”, but they are essentially high level objectives over the long term that merge technology trends, regulatory requirements and security goals into long term objective. These won’t be very specific, but instead should act as guidance for where your team should focus and hopefully end up over the next few years.

Examples

An example of a strategic trend and security objective are as follows:

Trend: As companies shift from the datacenter to the cloud and bring your own device (BYOD), the concept of a traditional perimeter no longer makes sense.

Strategic Security Objective: Shift to a zero trust strategy where identity becomes the perimeter.

The goal is to choose big ticket objectives that will take multiple years to achieve, but will provide guidance to your org and the rest of the business about the direction your team is taking. Your strategic plan will inform the next section, which is your operational plan.

Operational Planning (Op Planning)

Operational planning is more tactical in nature and covers a shorter time period than strategic planning. Op planning usually follows either a fiscal or calendar year that way it aligns to performance reviews and budgeting cycles. In op planning the CISO will select the high level goals they want the security organization to complete that year. Usually op planning will include discussion and planning of the following:

  • Budget creation, forecasting and changes
  • Headcount planning
  • Technology investments (if any)
  • Top risks to focus on
  • Any audits or compliance certifications needed that year
  • Development of timing and roadmap for completing specific projects and tasks
  • Discussion of security controls and services
  • Skill gaps and training requirements

The point is to create a tactical plan for the year that will inform your team’s specific goals and objectives. These goals should be clear and measurable. I typically use an iterative approach to break my goals down to my directs and then they break their goals down to their teams and so on. This ensures alignment throughout the business.

Measuring and Adjusting

One important aspect of any plan is to continually measure progress and adjust if needed. Goals and objectives aren’t useful if the business has shifted and they are no longer relevant or have become un-obtainable.

Wrapping Up

Strategic and operational planning are important activities for every CISO. These plans define the long term vision for the security organization and break down that vision into tactical objectives that are accomplished throughout the year. This post discussed a high level overview of what goes into strategic and operational planning, but aligning security plans to business risk, mapping security controls, obtaining funding and reporting progress are all complex activities that every CISO needs to master.

Career Options Post CISO

Last year was a busy year for CISOs. Increased regulation from the SEC and other entities are raising the stakes for companies and CISOs. 2023 demonstrated that regulators and law enforcement are not only going to hold companies accountable for incidents and breaches, but they will also pursue accountability against individual CISOs. The CISO role is at an inflection point created by new technologies, increases in regulation and unprecedented personal risk. Given the high stakes of the role I think we are going to see an exodus in the number of people who are willing to shoulder the burden and personal liability of this role. Which begs the question: what are the options for someone after serving in a CISO role?

Serving On A Board

Serving on a board seems like a popular choice lately and now that the SEC has mandated cybersecurity experience on the board I think companies will look to increase their board membership with former CSOs and CISOs. The challenge with serving on a board is finding one that can compensate you sufficiently. I’ve served on several boards over the past 15 years and the compensation will depend on the company size and maturity. Start ups are typically able to offer compensation in the form of equity in the company, but this may turn out to be worthless if the company doesn’t make it. Big company board positions are few and far between, but will pay the best. My advice for CISOs looking to transition into a paying board position is to serve on a board or several boards in your spare time and then transition to become a full time, paid, board member if and when the company can support it.

Advisory CISO / CISO In Residence

One way to “float” between a CISO role and a board member role is to get connected with a Private Equity (PE) or venture Capital (VC) company as an Advisory CISO or CISO In Residence. These roles help the PE and VC companies evaluate potential investments and then help guide the companies to success. If you are an Advisory CISO you can evaluate the companies and if you see one you think has real potential you can choose to be their CISO or serve on their board. Advisory CISOs are not only compensated by the PE / VC company, but they “consult” to the investment companies on a periodic basis and sometimes they are offered the opportunity to invest in the companies they are advising. Not a bad gig.

Consultant

One of the most common post C-Level career paths is to become a consultant. If you are well connected, are in a critical industry or are just great with people, this can be a viable career option. The experience you have built up over your career still has value and companies will pay you handsomely for your time to help advise them. If you work for a company that is unwilling to protect you if you are sued then this may be a way to continue in a CISO capacity, but without the personal risk. I’ve known people who have quit their current role out of frustration and when the company realizes the expertise they are about to lose they hire the person back as a consultant.

Field CISO

Field CISOs are fancy titles for people that are in sales or pre-sales. They typically have a specific region they are assigned and they use the Field CISO title to establish executive relationships with other CISOs and C-Suite members to help sell products and services. Field CISOs typically have extensive industry experience in a particular vertical and then they use that expertise to help tailor solutions to their customers.

Title Change (But Still Security)

Another option post CISO role is to get a title change, but still work in a security related role. This could be something like a Chief Trust Officer or Chief Risk Officer. These roles can offer more flexibility to have a positive impact on the business because they aren’t constrained by the same expectations as a CISO role. At the end of they day you are still a C-Level security executive and can continue to advance your career towards your goals.

Role Change (Not Security)

CISOs are one of the few roles that touch every aspect of the business. As a result, CISOs are well versed in a lot of different business disciplines and it would be easy for a CISO to transition to a CTO, CIO, engineering executive or product executive. For example, a CISO who is looking to exit the role may look to join a security focused startup as their CTO. Their deep industry experience and past credentials will provide credibility and allow them to continue working in the security space in a different capacity. Eventually, they can even hire a CISO to report to them and have oversight over the security function.

Start A Company

CISOs are also well positioned to see gaps in the industry where a solution hasn’t been developed. Lots of well known companies have been formed by former security executives who have left their role to start a company to develop a security related product or service. Starting a company doesn’t mean you have to develop a new technology. You could also start a consulting company, a training company or a staffing company. If you are sitting on a great idea then this is a viable option for you.

Double Down

Lastly, if you enjoy the CISO role, but don’t feel supported or protected by your current company, then find a new CISO role that gives you the support and protection you seek. Part of the interview process for your new role should include questions about who the role reports to, what is the expected budget and headcount, will the role be included in the D&O Policy, what happens if you are personally sued, what is the severance package and how will success be measured? These should all be table stakes for any company looking to hire or retain a CISO and satisfying these requirements will go a long way to making your CISO feel comfortable that you have their back and won’t treat them as a scapegoat.

2023 End Of Year Review & 2024 Look Ahead

At the start of 2023 I created personal and professional goals for myself to speak at conferences more, attend more professional events and capture my professional experience in a series of blog posts. In this post I’ll share what worked, what didn’t and how my results compared to my goals. At the end I’ll discuss what my goals are for 2024

Blogging (and Podcasts)

Just before 2023 kicked off I created this blog, primarily to capture my experience as a way to give back to the industry and to catalog my professional experience as a historical reference for myself. The two biggest lessons learned are: just get started and be consistent.

Just Get Started

I talked to a lot of people in 2023 about this blog and a surprising number confessed they also wanted to write a blog or create a podcast, but hadn’t started for a number of reasons, such as:

  • “I’m not a good writer”
  • “Nobody wants to hear what I have to say”
  • “I don’t have time”
  • “I need to get permission”

It is easy to hide behind these excuses and never start so my advice is to stop procrastinating and just get started. No one cares if your grammar isn’t perfect or if your content isn’t perfect. If you need permission, then track down who can approve your content and remove that barrier. Getting started will allow you to iterate, try new things and learn how to get better. The only way you will be able to get started is to set some ground rules for yourself. I personally found I work best by setting aside time on the weekend to write a post and then reward myself with some screen time such as video games or a movie. I also found that I write better when I have an idea and start writing it as soon as possible. If I wait, I often forget my thought process and then don’t write about that topic. Lastly, I found keeping a running idea log on my phone worked well. Whenever I have an idea I write it down in the idea log with as much context as possible, often creating a rough outline on the spot. Then when I get in front my my computer I can fill in the rest of the post.

Be Consistent

My goal in 2023 was to write one blog post or LinkedIn post a week. This felt sustainable without being a massive time commitment every day that would distract from my day job. My secondary goal was to increase my number of readers, followers and connections both virtually and in person. Being consistent is the best way to accomplish all of these goals. I was most consistent when I set aside time on the weekend to write for a few hours before rewarding myself with another activity. I found writing a post on the weekend or during the week allowed me time to refine it before posting it. I also relied heavily on scheduling posts, which allowed me to write several posts when I felt inspired and then post them when I was ready. This also allowed me some wiggle room if I was sick or traveling. By being consistent you continually pop up in people’s feeds and the social media algorithms will begin to recommend your content to people, which will increase your follower base.

So How Did I Do In 2023?

At the start of 2023 I set a goal to write one post per week. On this blog I achieved 38 posts and a combined 94 LinkedIn posts (which includes the blog posts). I started 2023 with 2010 followers and 1938 LinkedIn connections. I now have 2392 followers and 2091 connections. In general I connect with anyone on LinkedIn, but I do prune the connections if people try to sell me stuff or abuse the connection. Overall, I saw a 20% increase in followers and an 8% increase in connections and I’m very happy with these results.

Networking Events

I attended dozens of networking and industry events in 2023 and these spawned tons of additional follow on meet ups. If I meet someone new at an event, I try to connect with them on LinkedIn and then meet up for coffee or drinks to get to know them better. The top events I attended in 2023 were:

  • Gartner Evanta CIO / CISO Summits
  • HMG Strategy Denver Summit
  • Colorado=Security CISO Dinner Series

Public Speaking

One of my 2023 goals was to get back on the public speaking circuit. There are a few security related conferences in Denver with the top one being the Rocky Mountain Information Security Conference. In 2023 I gave a talk at RMISC about “A CISO Primer On Legal Privilege,” which gave a high level overview of legal privilege and had great audience discussion around the topic. I also spoke at a smaller, more intimate conference put on by BrainGu called RS2. This conference wasn’t security specific, but it did have a wide variety of speakers and thought leaders. The smaller conference setting allowed for great networking opportunities and I met a lot of great people there. Lastly, in July 2023 I met Hunter Muller of HMG Strategy as part of being nominated as a 2024 CISO of the Year. At the HMG Denver Summit I spoke on a panel with 3 other CISOs about “Innovation in Cybersecurity”. The HMG conference was a fantastic opportunity to meet other technology executives and hear their lessons learned.

Other Activities

In 2023 I explored joining a few new advisory boards. I’ve been on the CIO/CISO Advisory Board for the Denver Gartner Evanta community for the past few years, but at the end of 2023 I was also asked to join the HMG Denver Advisory Board. I also joined the advisory board of Phoenix Security and the STAR network for 1011 Venture Capital. My goal in all of this is to expand my network, keep up to date with industry trends and give back to the security community.

In addition to advisory boards I also explored doing video blogs and podcasts with other leaders. Most notably, Milan Patel and I have been doing a series of video blogs about the intersection of security and compliance. This has been great to cross pollenate our ideas and also draw from a different pool of followers.

2024 Look Ahead

I had a busy year in 2023 and am very happy with my results. So what’s in store for 2024?

  1. Continue the blog with a focus on being more timely with hot topics. My most popular posts were the ones that discussed my thoughts on topics that had immediate relevance in the news or industry.
  2. Do more blogs, podcasts or webinars with other industry leaders.
  3. Submit to speak at conferences. I’ll plan to continue to submit to speak at RMISC, a Gartner Evanta event and an HGM Strategy Summit. If another opportunity pops up I’ll definitely write a post about it.
  4. Explore joining additional advisory boards. I am enjoying advising various companies and industry groups on how to navigate the complex cybersecurity market. My experience as a CISO, CTO and lifelong technologist provides perspective so I can help guide these groups to be successful.

Opsec During Incidents

When I first got into Information Technology over 20 years ago, I started out in networking and data centers. When doing work, we always made sure to have another method of accessing our gear in case a configuration change didn’t work and eliminated access to the equipment or environment. Also, during my military service we would always discuss what would happen in a worst case scenario and have contingency plans in case something went wrong. I’ve carried these principles over into my security career and they have served me well. Given the high likelihood that we are all operating in some type of compromised environment, I think it is good to cover how these principles can be used to maintain operational security during incidents.

Philosophy

Hope for the best, prepare for the worst

No battle plan withstands first contact with the enemy

Train how you fight

There are a lot of things you can do to prepare for a security incident such as, running table top exercises, preparing playbooks and modifying tool sets, but there is an important area that I think is often overlooked during preparation. There is an implicit assumption that all of the fancy tools and environments you have prepared will be available when the incident kicks off. If you are preparing for your incidents with this assumption then you could find your team in trouble. One key component of a realistic table top scenario is to remove capabilities and see how the team reacts. This could be something as simple as saying the incident commander is no longer reachable or something more drastic like not being able to access your SIEM. The point is, the more worst case situations you can prepare for, the better off you will be during a real incident. Here are a few things you should consider to maintain operational security and the ability to respond.

Access To Playbooks

Playbooks are a critical tool for responding to an incident. Incidents can be highly stressful and playbooks exist to help your responders remember all of the steps, access methods, processes, tools, etc. they need to successfully respond. What would you do if an incident kicked off and you couldn’t access your playbooks? Does your team have these playbooks stored in an alternative environment? Do they sync them to an offline, physical storage device that is stored in a safe or passed between people on call?

Playbooks need to not only be accessible via alternate methods, but they also need to be protected. What if the attacker had access to your playbooks? I’m willing to bet your playbooks have a detailed list of all the things your team will do to respond and this can be a useful blueprint for an attacker to follow to avoid detection or expand their footprint. I highly recommend encrypting your playbooks and vaulting the credentials so you can monitor and control access as needed.

Access To Tooling

Tooling is also essential during the incident response process. Tools help teams investigate, gather evidence and ultimately recover from the incident. Access to tools and scripts can help with a number of activities such as, identify hashing algorithms, identifying password encryption types, scanning for open ports or scanning for vulnerabilities. Tools that are useful during incident response should be stored in an image or container that is updated regularly and kept in an offline environment. This will help the incident response team in a few ways. First, it will make sure all of the necessary tools that are needed in the playbook are available even if systems and network access isn’t accessible. Second, it avoids having to pull tools from the internet that haven’t been vetted. In a stressful situation like an incident it is possible to overlook small details that could make your situation worse, such as downloading a tool that has a trojan or backdoor in it. Lastly, it is important to control the flow of information during an incident. If your team is using online tools as part of the investigation they could be leaking information to the internet that can reveal details about the incident, which could not only derail your investigation, but could harm your company brand and reputation.

Another consideration for tooling is how you plan to preserve evidence if you are under active compromise. Having pre-staged environments that are isolated and only used for incidents can be useful. Similarly, having tooling available to image machines, quarantine networks or even encrypt files to send them to an alternate cloud / storage provider can all be useful during an incident.

Access To Communications

Communication during an incident is key. I like to set up a standing War Room where people can come and go as needed. My presence is there to make decisions and help direct resources where they can be most useful. In the chaos of an incident it is important to make sure you have primary, secondary and sometimes even tertiary communication methods available. What happens if Zoom or WebEx is down and you can’t stand up a virtual war room? Do you have access to a call sheet so you can get a hold of everyone and stand up a conference call? What would you do if your primary chat method such as slack or teams wasn’t available or was compromised? Do you have Signal or Wickr set up so you can still chat with your team?

Additionally, I highly recommend you copy legal on your incident communications so they can advise you accordingly (see my post on Legal Privilege). I also recommend you encrypt your communications, which can help protect your comms if you are operating in a compromised environment.

Access To People

Lastly, but probably the most important, is access to your people. People are essential when responding to an incident and running table tops to prepare them is essential. Cross training should also happen regularly in case an expert in a specific skill isn’t readily available. Ultimately, the more training and cross training your team has, the better prepared you will be for an incident.

Conducting table top drills and running through failure scenarios can be fun and also educational. Next time you run a table top exercise take out your top person and see how the team reacts. Put people on the spot in different roles such as investigation, evidence gathering, recovery, interfacing with engineering, etc. If someone struggles with a role during the table top you can flag that area for additional training.

Leadership is also essential during an incident and I’ve written blog posts about this in the past. Try rotating people in different leadership positions so they can get a feel for what it takes. This can also help them understand and anticipate what is needed from them during an incident.

Wrapping Up

Incidents can be both exciting and stressful at the same time. The thoroughness and frequency of your training will dictate how well your team responds during an incident. Most importantly, planning for various worst case scenarios can ensure your team is able to successfully respond, communicate and lead the rest of the business through an incident.

Will CVSS 4.0 Help Companies Manage Vulnerabilities Better?

About two weeks ago FIRST published version 4.0 of the Common Vulnerability Scoring Standard (CVSS), largely in response to feedback from the industry on the shortcomings of CVSS 3.1 and previous versions. The main complaint from industry with version 3.1 was that it didn’t offer any way to add additional context in a way that could help determine and prioritize risk. This led to companies to come up with their own processes to add context. In a previous blog about The Problem With Vulnerability Scanners I specifically highlighted how CVSS scores weren’t very useful and needed additional business context to make a risk prioritization decision. With that in mind, CVSS 4.0 attempts to address these shortcomings. Let’s take a look at what they changed and if it will help.

What’s New?

Both CVSS 3.1 and CVSS 4.0 include ways to evaluate vulnerabilities using the intrinsic characteristics of the vulnerability (Based), how the vulnerability changes over time (Temporal v3 or Threat v4) and how the vulnerability specifically applies to your environment (Environment). New for v4 is a Supplemental section which doesn’t impact the CVSS score, but allows you to add additional context for the vulnerability.

Additionally, CVSS 4.0 promises the ability to add real time threat context by allowing teams to use Threat Intelligence as an input to the CVSS score for a vulnerability. This additional context can be provided in new sections such as Attack Complexity, Attack Requirements, Vulnerable System and Subsequent System. CVSS 4.0 attempts to acknowledge unique environments by allowing additional fields for things like safety, ICS systems, etc. You can read about the full CVSS 4.0 specification here.

Finally! A Way To Prioritize Vulnerabilities!

CVSS 4.0 definitely seems like a huge step towards allowing teams to provide additional context to a vulnerability with the ultimate goal of influencing the score for better risk prioritization. The most common complaint I hear from engineering teams is there are too many vulnerabilities with the same criticality and they are unsure where to start. This was also feedback provided by industry to FIRST because it seemed like vulnerabilities were clustered more towards the critical and high range after the changes from v2 to v3.

CVSS 4.0 definitely answers some of the previous shortcomings and allows teams to add additional context to help make better decisions about which vulnerabilities should be prioritized for remediation over others. I know it is fairly common for the top priority to be given to external, publicly facing systems. The problem was CVSS 3.0 didn’t really provide a way to delineate between internal and external systems very well. So overall, the changes introduced in v4 are very welcome and should help teams really focus on what matters.

Is More Choice A Good Thing?

While it may seem like a good thing to be able to adjust the CVSS score for a vulnerability I do see this causing issues, particularly with external reporting. Security teams will need to have a robust process documented for how they are adjusting the score of a vulnerability and I can see situations in the future where companies are accused of subjectively adjusting their vulnerability scores down to paint a better picture than the reality.

Additionally, more choice comes with less transparency. Over the past year I have seen the volume and complexity of security questionnaires increase. The top questions focus around vulnerability remediation SLAs, incident response times and software supply chain security. Adding additional complexity into the CVSS scoring process, that allows companies to subjectively adjust the score up or down, will be extremely difficult for customers and regulators to navigate. Think back to Log4j and the reaction from your customers if you said you had Log4j vulnerabilities, but weren’t prioritizing remediation because they were on internal systems only. This may be a reasonable risk response for the business, but the perception from your customers will be difficult to manage.

Time Will Tell

Overall, it seems like CVSS 4.0 is attempting to become more of an overall risk score, rather than just a severity score. It is certainly welcome to be able to add additional context and take additional input to adjust the CVSS score as it applies to your environment and business. However, the new standard adds additional complexity and subjectivity that will make it difficult for customers and regulators to assess the true risk of a vulnerability to the business in a common way across the industry. Security teams will need to be particularly diligent in documenting a robust process for how they are adjusting the CVSS score to avoid being accused of arbitrarily adjusting the CVSS score down to make their company look better.

Chief Incident Scapegoat Officer (CISO)?

Last week the SEC filed a complaint in the Southern District of New York charging SolarWinds and specifically its CISO, Timothy Brown, with fraud. According to the compliant, the SEC alleges the company and Brown made false statements about its security posture to investors. Along with the Uber CISO, Joseph Sullivan, this is the second CISO in the past year to be specifically charged for failing to do their job. In my opinion, these court cases are going to negatively impact the CISO role and make security less transparent to investors. Let’s dive in.

What About The Other C-Levels?

Both cases are unique, however the first thing that stands out to me is only the CISOs are being named and charged. I find this odd because in an ideal organization the CISO still has to partner closely with the other C-Level execs to achieve security objectives. Things like external messaging to customers, SEC filings, etc. all require the coordination and knowledge of other C-Level execs like the CFO, Legal, Marketing and even the CEO. Why aren’t these individuals being named and charged for also contributing to the fraud?

In the worst case scenario, a CISO is poorly supported and struggles to get any of their security objectives funded or implemented. Is the CISO to blame in this scenario? What about the CEO and CFO who withheld funding? How about the Engineering leader who failed to prioritize the security recommendations of the CISO? The point is, I have never found a situation where a CISO is able to operate in a vacuum and so the other C-Level execs also have a responsibility to make sure the company is making true statements and not perpetrating fraud. They should all be held equally accountable.

Responsibility Without Authority

The CISO role has had a lot of press and a surge in visibility over the past few years, but the role still has a long way to go to be on par with other C-Level roles. It is common for the CISO role to report to the CTO, CIO or Chief Legal Counsel. It is uncommon for the CISO role to have a direct reporting line to the CEO. We can discuss who the CISO should report to, but in my opinion, the CISO role still needs to mature compared to the other legacy C-Level roles. The position is currently not on the same level as a CTO or CIO role and this impacts the scope and authority of the role.

Additionally, most CISOs don’t actually own the things they are trying to improve the security posture of. There is always a business or engineering owner that is actually responsible for building and operating the systems that make the company money. As a result, the CISO role typically ends of with all of the responsibility for security, but none of the authority. If the CISO makes a recommendation to fix something and the engineering leader rejects it, who is held accountable for that decision?

Chilling Effect On Open Discussion

My biggest concern with the SEC complaint is the reference to emails that are pointing out the known security issues with the Orion system. Matt Levine wrote a great article in Bloomberg questioning the SEC’s logic and I agree with his assessments. I have never read an SEC filing or investment statement expecting the company to highlight their massive security investments. In fact, I would question if a company should disclose that in a filing at all (unless it is material) because you may inadvertently provide information to attackers that could be used to hack the company.

Additionally, most security teams openly discuss security issues via chat or email. I find these discussions are almost always expressing frustration with current situations with the goal of gaining support for investment to remedy the issue. However, discussions via chat and email also happen to be legally discoverable forms of communication. This means every single email about how much your security sucks will be taken out of context by lawyers and used against you. The obvious solution is to never put your current security failings in writing, which means you can never create a presentation to convince the company to invest in improving security. Or alternatively, if you do place things in writing you frame them in a way that they are asking for legal advice so they can be protected by legal privilege.

Predictions For the CISO Role

I wrote a blog post after the Uber verdict, but both the Uber and SolarWinds cases have caused significant anxiety within the CISO community, which I think will impact the CISO in the following ways going forward:

  1. New CISOs hiring into a role will require companies to list them on their Directors and Officers (D&O) Liability Policy. Also, based on this Bloomberg Law Article about FTX, I recommend making sure the D&O policy specifies how much you will get if all the executives are trying to use the policy at the same time for legal fees.
  2. It will become standard for companies to cover the costs for legal counsel specified by the CISO, should they be individually named in a lawsuit.
  3. As these cases become more common, CISOs will demand higher compensation and protect themselves contractually to minimize their personal risk.
  4. Companies will (hopefully) prioritize security investments to minimize the risk of lawsuits, regulatory actions or security incidents.
  5. Costs for companies to employ and retain a CISO will go up over time.
  6. In extreme cases, the CISO role may shift from a salaried employee to a consultant (I-9) to offload the accountability for security to the company and protect themselves.

Final Thoughts

I can’t recall the last time I saw a CTO or CIO charged with investor fraud for making false statements about their products or enterprise environment. Yet, the CISO role has been getting a lot of scrutiny from regulators recently. I’m all for holding people accountable, but the CISO role doesn’t seem to carry the same weight as the CTO or CIO. The role still struggles with gaining support and funding to place security first. If a company culture is weak or the other executives minimize security, then the CISO will fail to make any meaningful progress. In my opinion, if the CISO of the company is named, then all the officers should be named to drive home the message that they are all accountable for the security of the company.