Category: #strategy

We Are Drowning In Patches (and what to do about it)

Last week I had an interesting discussion with some friends about how to prioritize patches using criticality and a risk based approach. After the discussion I starting thinking about how nice it would be if we could all just automatically patch everything and not have to worry about prioritization and the never ending backlog of patches, but unfortunately this isn’t a reality for the majority of organizations.

Whats the problem?

There are several issues that create a huge backlog of patches for organizations.

First, let’s talk about the patching landscape organizations need to deal with. This is largely spit into two different areas. The first area is operating system (OS) and service patches. These are patches that are released periodically for the operating systems used by the business to run applications or products. Common operating systems for production workloads will be either Windows or Linux and will have stability, security or new feature patches released periodically.

Second, there are patches for software libraries that are included in the software and applications developed by your business. Typically these are lumped into the category of 3rd party libraries, which means your organization didn’t write these libraries, but they are included in your software. 3rd party library security vulnerabilities have become a huge issue over the last decade (but thats a blog post for another day).

These two patch types, OS and 3rd party library patches, require different approaches to discover, manage and remediate, which is the first challenge for auto patching. When combined with the volume of new vulnerabilities being discovered, large heterogeneous environments and the need to keep business critical applications available, keeping your assets patched and up to date becomes a real challenge.

Why isn’t auto patching a thing?

Well it is, but…

There are a few challenges to overcome before you can auto-patch.

Stability and Functionality

First, both operating system and 3rd party library patches need to be tested for stability and functionality. Usually, patches fix some sort of issue or introduce new features, but this can cause issues in other areas such as stability or functionality. It can be a complex process to roll back patches and restore business critical applications to a stable version, which is why most businesses test their patches in a staging environment before rolling them out to production. Cash is king and businesses want to minimize any disruption to cash flow.

Investment and Maturity

It is possible to automate testing for stability and functionality, but this requires a level of maturity and investment that most organizations haven’t achieved. For example, assuming your staging environment is a mirror image of your production environment (it is right?), you could auto apply the patches in staging, automatically check for stability and functionality over a set period of time and then roll those updates to production with minimal interaction. However, if your environment requires reboots or you have limited resources, patching may require down time, which could impact making money.

In order to have an environment that can support multiple versions, seamless cut over, proper load balancing, caching, etc. requires significant investment. Typically this investment is useful for keeping your products functioning and making money even if something goes wrong, but this investment can also be used to buffer maintenance activities such as patching without disruption.

Software Development Lifecycle

The last section assumes a level of software development maturity such as adoption of Agile development processes and CI/CD (continuous integration / continuous delivery). However, if your engineering group uses a different development process such as Incremental or Waterfall, then patching may become even more difficult because you are now competing with additional constraints and priorities.

What are some strategies to prioritize patching and reduce volume?

If your business runs products that aren’t mission critical, or you simply can’t justify the investment to operate an environment without down time, then auto patching probably isn’t a reality for you unless you are Austin Powers and like to live dangerously. For most organizations, you will need to come up with a strategy to prioritize patching and reduce the volume down to a manageable level.

Interestingly, this problem space has had a bunch of brain power dedicated to it over the years because it resembles a knapsack problem, which is a common problem space in mathematics, computer science and economics. Knapsack problems are problems where you have a finite amount of a resource (space, time, etc.) and you want to optimize the use of that resource to maximize some sort of requirement (like value). In the case of patching, this would mean applying the largest volume of the highest severity patches in a fixed time period to realize the maximum risk reduction possible.

Critical Assets First

Staying in the knapsack problem space, one strategy is to start with your most critical assets and apply the highest severity patches until you reach your threshold for risk tolerance. This requires your organization to have an up to date asset inventory and have categorized your assets based on business criticality and risk. For example, let’s say you have two applications at your business. One is a mission critical application for customers and generates 80% of your annual revenue. The other application provides non-mission critical functionality and accounts for the other 20% of revenue. Your risk tolerance based on your company policies is to apply all critical and high patches within 72 hours of release. In this example you would apply all critical and high patches to the mission critical application as quickly as possible (assuming other requirements are met like availability, etc.).

Guard Rails and Gates

Another strategy for reducing volume is to have guard rails or gates as part of your software development lifecycle. This means your engineering teams will be required to pass through these gates at different stages before being allowed to go to production. For example, your organization may have a policy that no critical vulnerabilities are allowed in production applications. The security organization creates a gate that scans for OS and 3rd party library vulnerabilities whenever an engineering team attempts to make changes to the production environment (like pushing new features). This way the engineering team needs to satisfy any vulnerability findings and apply patches at regular intervals coinciding with changes to production.

Wrapping Up

With the proliferation of open source software, the speed of development and the maturity of researchers and attackers to find new vulnerabilities, patching has become an overwhelming problem for a lot of organizations. In fact, it is such a big problem CISA and the Executive Order On Improving The Nation’s Cybersecurity list software patches and vulnerabilities as a key national security issue. I’ve outlined a few strategies to prioritize and reduce the volume of patches if your organization can’t afford the investment to absorb downtime without disruption. However, no matter what strategy you choose, all of them require strong fundamentals in asset inventory, asset categorization and defined risk tolerance. While these investments may seem tedious at first, the more disciplined you are about enforcing the security fundamentals (and engineering maturity), the less you will drown in patches and the closer your organization will come to the reality of auto-patching.

Annual Planning For CISOs

The beginning of the year is a popular time for making personal resolutions, which can focus on health, finance or love. While the beginning of the year is a popular time to set resolutions, really what we are talking about is setting goals to improve ourselves. I’m a huge proponent of setting personal goals for the year because it gives focus and purpose to your actions. The beginning of the year is also a great time to review the annual goals of your security program to set your focus and establish priorities. Annual planning has several objectives that CISOs need to consider and include in their process and I’ll cover them in the rest of this post.

Strategic Planning (Strat Planning)

Strategic, or “strat” planning as it is sometimes called, looks at where the business and your organization want to be over a long term time period. Something like 18 months to 5 years is typical in strat planning. The planning session should include discussion of the one or more of the following macro level business topics:

  • Market forces and opportunities
  • Industry trends
  • Regulatory and legal landscape
  • Competition
  • Customer sentiment, goals, etc.
  • Economic and financial environment
  • Geo-political climate
  • Technology trends and latest research

This discussion could be part of a SWOT analysis (Strengths, Weaknesses, Opportunities, Threats), but the goal is to understand where your business is and where you want it to go in the long term.

Align The Security Program

Once the business has a strategic plan, the CISO should conduct a similar planning exercise for where they want the security program to be. These are sometimes called “North Stars”, but they are essentially high level objectives over the long term that merge technology trends, regulatory requirements and security goals into long term objective. These won’t be very specific, but instead should act as guidance for where your team should focus and hopefully end up over the next few years.

Examples

An example of a strategic trend and security objective are as follows:

Trend: As companies shift from the datacenter to the cloud and bring your own device (BYOD), the concept of a traditional perimeter no longer makes sense.

Strategic Security Objective: Shift to a zero trust strategy where identity becomes the perimeter.

The goal is to choose big ticket objectives that will take multiple years to achieve, but will provide guidance to your org and the rest of the business about the direction your team is taking. Your strategic plan will inform the next section, which is your operational plan.

Operational Planning (Op Planning)

Operational planning is more tactical in nature and covers a shorter time period than strategic planning. Op planning usually follows either a fiscal or calendar year that way it aligns to performance reviews and budgeting cycles. In op planning the CISO will select the high level goals they want the security organization to complete that year. Usually op planning will include discussion and planning of the following:

  • Budget creation, forecasting and changes
  • Headcount planning
  • Technology investments (if any)
  • Top risks to focus on
  • Any audits or compliance certifications needed that year
  • Development of timing and roadmap for completing specific projects and tasks
  • Discussion of security controls and services
  • Skill gaps and training requirements

The point is to create a tactical plan for the year that will inform your team’s specific goals and objectives. These goals should be clear and measurable. I typically use an iterative approach to break my goals down to my directs and then they break their goals down to their teams and so on. This ensures alignment throughout the business.

Measuring and Adjusting

One important aspect of any plan is to continually measure progress and adjust if needed. Goals and objectives aren’t useful if the business has shifted and they are no longer relevant or have become un-obtainable.

Wrapping Up

Strategic and operational planning are important activities for every CISO. These plans define the long term vision for the security organization and break down that vision into tactical objectives that are accomplished throughout the year. This post discussed a high level overview of what goes into strategic and operational planning, but aligning security plans to business risk, mapping security controls, obtaining funding and reporting progress are all complex activities that every CISO needs to master.