The Dichotomy Of Security

If you have ever read Extreme Ownership or The Dichotomy of Leadership by Jocko Willink, then you will be familiar with the concept of dichotomy and how opposing forces of a skill set can compliment each other. Mastering both sides can allow flexibility and increase the effectiveness of that skill set when dynamically applied to a given situation. This is true in the security space, where fundamental opposing forces need to be balanced in order to manage risk and achieve success. Let’s take a look at a few examples.

Security Extremes

The easiest example of the dichotomy of security is to look at the extremes. Security professionals jokingly say the most secure company is one that is not connected to the internet. While this may be true, it will also prevent the company from conducting business effectively and so the company will cease to exist and security will no longer be needed.

On the other end of the spectrum there is the extreme of a business that has zero security and so there are no impediments to conducting business. While this may sound great to some, the reality is the company will be unable to effectively conduct business because of the real threats that exist on the internet. In the situation the company will also cease to exist because they will be hacked into oblivion.

It is obvious there is a dichotomy between no security and no connectivity and these forces need to be appropriately balanced for a security program to be effective, while allowing the business to operate.

Manual vs Automated Security

Another example of dichotomy is between manual security tasks and automation. While every CISO I know is striving to increase automation of security tasks, the reality is humans are still going to be needed in any security program for the foreseeable future.

Manual tasks are ideal for situations where humans need to demonstrate creativity, intuition or make complex decisions based on subtle context. Security functions like penetration testing, threat hunting, red teaming and offensive security require high amounts of skill and experience that automation, like AI, hasn’t been able to replicate. Additionally, soft skills such as reporting to the board, shifting culture, building alliances and making prioritization decisions are all extremely complex and unlikely candidates for automation. However, while manual activities benefit activities that require a high degree of creativity, they are inherently slow and can impede the normal flow of business.

Recently, the advances in automation and artificial intelligence have exponentially increased their usefulness. Automation is extremely useful for offloading repeatable tasks that lend themselves to being programmatically defined. For example, attack simulation products have made huge strides in offloading repetitive tasks of reconnaissance, enumeration, vulnerability assessment and remedial exploitation. We are seeing additional advances in automation related to incident response where events can be correlated and specific activities in an IR playbook can be completed to offload analysts and help focus their attention. AI has also helped to offload lower level operational activities like call centers and help desk inquiries.

While automation may accelerate parts of the business and offload humans from repeatable tasks, it does introduce complexity, which can be difficult to troubleshoot or can cause outright failures. Automation is also rigid because it is only as good as the parameters of the process it is following. This means it can’t think outside of the box or demonstrate creativity. There is also the risk of introducing bias into your processes if your underlying model is flawed.

As you can see manual security processes and automated security processes are opposing forces that need to be balanced based on the skill of your security team and the needs of the business.

The Human Problem

The last dichotomy I want to discuss is the human problem in security. Humans are necessary because of their creativity, diversity and capacity for adapting to an infinite number of situations. However, the flexibility in human nature also presents one of the fundamental security problems – how to you protect against human nature?

The reality is humans are flawed, but in a good way. Threat actors can try to take advantage of these flaws, whether they are logical (like firewall rules) or physical (like human psychology). Humans are essential to every aspect of a business and so we have to figure out how to protect them. The most difficult balance in security is developing a program that is comprehensive enough to protect against human nature without stifling it.

The Security Ideal

The ideal security program will recognize the dichotomy of the security challenges it faces and balance them accordingly. The ideal security program balances security with flexibility. We are seeing this balance manifest in mature security programs via concepts like security guard rails and the paved path. The paved path and guard rails attempt to allow a certain amount of latitude for acceptable behavior, while being rigid enough to protect users and the business accordingly.

Application In Other Domains

The concept of dichotomy is universal across any domain. In fact, this is an area of extensive research in disciplines like mathematics, computer science, military strategy, and economics. Specifically, in the space of network and graph theory there is a concept call max flow, min cut. These are counter principles that are opposite, yet complimentary. If you think of any network (road, supply chain, computer network, etc.) the point of maximum flow across that network is also the point where maximum disruption (minimum cut) can occur. From a military or security stand point you will want to protect the max flow/min cut, but from an attacker stand point, the max flow / min cut, is the area that will require the least amount of effort for maximum damage. Pretty neat!

Wrapping Up

An effective security program will balance the needs of security with the needs business with the ultimate goal of effectively managing risk. A critical skill for any security practitioner is to be flexible and adaptive. Specifically, by recognizing that security issues have two sides to them, security practitioners can demonstrate empathy towards the business and find an appropriate balance that can protect without impeding the business.

We Are Drowning In Patches (and what to do about it)

Last week I had an interesting discussion with some friends about how to prioritize patches using criticality and a risk based approach. After the discussion I starting thinking about how nice it would be if we could all just automatically patch everything and not have to worry about prioritization and the never ending backlog of patches, but unfortunately this isn’t a reality for the majority of organizations.

Whats the problem?

There are several issues that create a huge backlog of patches for organizations.

First, let’s talk about the patching landscape organizations need to deal with. This is largely spit into two different areas. The first area is operating system (OS) and service patches. These are patches that are released periodically for the operating systems used by the business to run applications or products. Common operating systems for production workloads will be either Windows or Linux and will have stability, security or new feature patches released periodically.

Second, there are patches for software libraries that are included in the software and applications developed by your business. Typically these are lumped into the category of 3rd party libraries, which means your organization didn’t write these libraries, but they are included in your software. 3rd party library security vulnerabilities have become a huge issue over the last decade (but thats a blog post for another day).

These two patch types, OS and 3rd party library patches, require different approaches to discover, manage and remediate, which is the first challenge for auto patching. When combined with the volume of new vulnerabilities being discovered, large heterogeneous environments and the need to keep business critical applications available, keeping your assets patched and up to date becomes a real challenge.

Why isn’t auto patching a thing?

Well it is, but…

There are a few challenges to overcome before you can auto-patch.

Stability and Functionality

First, both operating system and 3rd party library patches need to be tested for stability and functionality. Usually, patches fix some sort of issue or introduce new features, but this can cause issues in other areas such as stability or functionality. It can be a complex process to roll back patches and restore business critical applications to a stable version, which is why most businesses test their patches in a staging environment before rolling them out to production. Cash is king and businesses want to minimize any disruption to cash flow.

Investment and Maturity

It is possible to automate testing for stability and functionality, but this requires a level of maturity and investment that most organizations haven’t achieved. For example, assuming your staging environment is a mirror image of your production environment (it is right?), you could auto apply the patches in staging, automatically check for stability and functionality over a set period of time and then roll those updates to production with minimal interaction. However, if your environment requires reboots or you have limited resources, patching may require down time, which could impact making money.

In order to have an environment that can support multiple versions, seamless cut over, proper load balancing, caching, etc. requires significant investment. Typically this investment is useful for keeping your products functioning and making money even if something goes wrong, but this investment can also be used to buffer maintenance activities such as patching without disruption.

Software Development Lifecycle

The last section assumes a level of software development maturity such as adoption of Agile development processes and CI/CD (continuous integration / continuous delivery). However, if your engineering group uses a different development process such as Incremental or Waterfall, then patching may become even more difficult because you are now competing with additional constraints and priorities.

What are some strategies to prioritize patching and reduce volume?

If your business runs products that aren’t mission critical, or you simply can’t justify the investment to operate an environment without down time, then auto patching probably isn’t a reality for you unless you are Austin Powers and like to live dangerously. For most organizations, you will need to come up with a strategy to prioritize patching and reduce the volume down to a manageable level.

Interestingly, this problem space has had a bunch of brain power dedicated to it over the years because it resembles a knapsack problem, which is a common problem space in mathematics, computer science and economics. Knapsack problems are problems where you have a finite amount of a resource (space, time, etc.) and you want to optimize the use of that resource to maximize some sort of requirement (like value). In the case of patching, this would mean applying the largest volume of the highest severity patches in a fixed time period to realize the maximum risk reduction possible.

Critical Assets First

Staying in the knapsack problem space, one strategy is to start with your most critical assets and apply the highest severity patches until you reach your threshold for risk tolerance. This requires your organization to have an up to date asset inventory and have categorized your assets based on business criticality and risk. For example, let’s say you have two applications at your business. One is a mission critical application for customers and generates 80% of your annual revenue. The other application provides non-mission critical functionality and accounts for the other 20% of revenue. Your risk tolerance based on your company policies is to apply all critical and high patches within 72 hours of release. In this example you would apply all critical and high patches to the mission critical application as quickly as possible (assuming other requirements are met like availability, etc.).

Guard Rails and Gates

Another strategy for reducing volume is to have guard rails or gates as part of your software development lifecycle. This means your engineering teams will be required to pass through these gates at different stages before being allowed to go to production. For example, your organization may have a policy that no critical vulnerabilities are allowed in production applications. The security organization creates a gate that scans for OS and 3rd party library vulnerabilities whenever an engineering team attempts to make changes to the production environment (like pushing new features). This way the engineering team needs to satisfy any vulnerability findings and apply patches at regular intervals coinciding with changes to production.

Wrapping Up

With the proliferation of open source software, the speed of development and the maturity of researchers and attackers to find new vulnerabilities, patching has become an overwhelming problem for a lot of organizations. In fact, it is such a big problem CISA and the Executive Order On Improving The Nation’s Cybersecurity list software patches and vulnerabilities as a key national security issue. I’ve outlined a few strategies to prioritize and reduce the volume of patches if your organization can’t afford the investment to absorb downtime without disruption. However, no matter what strategy you choose, all of them require strong fundamentals in asset inventory, asset categorization and defined risk tolerance. While these investments may seem tedious at first, the more disciplined you are about enforcing the security fundamentals (and engineering maturity), the less you will drown in patches and the closer your organization will come to the reality of auto-patching.