Month: February 2024

We Are Drowning In Patches (and what to do about it)

Last week I had an interesting discussion with some friends about how to prioritize patches using criticality and a risk based approach. After the discussion I starting thinking about how nice it would be if we could all just automatically patch everything and not have to worry about prioritization and the never ending backlog of patches, but unfortunately this isn’t a reality for the majority of organizations.

Whats the problem?

There are several issues that create a huge backlog of patches for organizations.

First, let’s talk about the patching landscape organizations need to deal with. This is largely spit into two different areas. The first area is operating system (OS) and service patches. These are patches that are released periodically for the operating systems used by the business to run applications or products. Common operating systems for production workloads will be either Windows or Linux and will have stability, security or new feature patches released periodically.

Second, there are patches for software libraries that are included in the software and applications developed by your business. Typically these are lumped into the category of 3rd party libraries, which means your organization didn’t write these libraries, but they are included in your software. 3rd party library security vulnerabilities have become a huge issue over the last decade (but thats a blog post for another day).

These two patch types, OS and 3rd party library patches, require different approaches to discover, manage and remediate, which is the first challenge for auto patching. When combined with the volume of new vulnerabilities being discovered, large heterogeneous environments and the need to keep business critical applications available, keeping your assets patched and up to date becomes a real challenge.

Why isn’t auto patching a thing?

Well it is, but…

There are a few challenges to overcome before you can auto-patch.

Stability and Functionality

First, both operating system and 3rd party library patches need to be tested for stability and functionality. Usually, patches fix some sort of issue or introduce new features, but this can cause issues in other areas such as stability or functionality. It can be a complex process to roll back patches and restore business critical applications to a stable version, which is why most businesses test their patches in a staging environment before rolling them out to production. Cash is king and businesses want to minimize any disruption to cash flow.

Investment and Maturity

It is possible to automate testing for stability and functionality, but this requires a level of maturity and investment that most organizations haven’t achieved. For example, assuming your staging environment is a mirror image of your production environment (it is right?), you could auto apply the patches in staging, automatically check for stability and functionality over a set period of time and then roll those updates to production with minimal interaction. However, if your environment requires reboots or you have limited resources, patching may require down time, which could impact making money.

In order to have an environment that can support multiple versions, seamless cut over, proper load balancing, caching, etc. requires significant investment. Typically this investment is useful for keeping your products functioning and making money even if something goes wrong, but this investment can also be used to buffer maintenance activities such as patching without disruption.

Software Development Lifecycle

The last section assumes a level of software development maturity such as adoption of Agile development processes and CI/CD (continuous integration / continuous delivery). However, if your engineering group uses a different development process such as Incremental or Waterfall, then patching may become even more difficult because you are now competing with additional constraints and priorities.

What are some strategies to prioritize patching and reduce volume?

If your business runs products that aren’t mission critical, or you simply can’t justify the investment to operate an environment without down time, then auto patching probably isn’t a reality for you unless you are Austin Powers and like to live dangerously. For most organizations, you will need to come up with a strategy to prioritize patching and reduce the volume down to a manageable level.

Interestingly, this problem space has had a bunch of brain power dedicated to it over the years because it resembles a knapsack problem, which is a common problem space in mathematics, computer science and economics. Knapsack problems are problems where you have a finite amount of a resource (space, time, etc.) and you want to optimize the use of that resource to maximize some sort of requirement (like value). In the case of patching, this would mean applying the largest volume of the highest severity patches in a fixed time period to realize the maximum risk reduction possible.

Critical Assets First

Staying in the knapsack problem space, one strategy is to start with your most critical assets and apply the highest severity patches until you reach your threshold for risk tolerance. This requires your organization to have an up to date asset inventory and have categorized your assets based on business criticality and risk. For example, let’s say you have two applications at your business. One is a mission critical application for customers and generates 80% of your annual revenue. The other application provides non-mission critical functionality and accounts for the other 20% of revenue. Your risk tolerance based on your company policies is to apply all critical and high patches within 72 hours of release. In this example you would apply all critical and high patches to the mission critical application as quickly as possible (assuming other requirements are met like availability, etc.).

Guard Rails and Gates

Another strategy for reducing volume is to have guard rails or gates as part of your software development lifecycle. This means your engineering teams will be required to pass through these gates at different stages before being allowed to go to production. For example, your organization may have a policy that no critical vulnerabilities are allowed in production applications. The security organization creates a gate that scans for OS and 3rd party library vulnerabilities whenever an engineering team attempts to make changes to the production environment (like pushing new features). This way the engineering team needs to satisfy any vulnerability findings and apply patches at regular intervals coinciding with changes to production.

Wrapping Up

With the proliferation of open source software, the speed of development and the maturity of researchers and attackers to find new vulnerabilities, patching has become an overwhelming problem for a lot of organizations. In fact, it is such a big problem CISA and the Executive Order On Improving The Nation’s Cybersecurity list software patches and vulnerabilities as a key national security issue. I’ve outlined a few strategies to prioritize and reduce the volume of patches if your organization can’t afford the investment to absorb downtime without disruption. However, no matter what strategy you choose, all of them require strong fundamentals in asset inventory, asset categorization and defined risk tolerance. While these investments may seem tedious at first, the more disciplined you are about enforcing the security fundamentals (and engineering maturity), the less you will drown in patches and the closer your organization will come to the reality of auto-patching.

Are Phishing Campaigns Worth It?

Phishing campaigns are often touted as a complementary exercise to security training as a way to measure training effectiveness. The thought is, if your training is effective, users will be less likely to fall for and click on phishing emails, which will correlate to a decrease in the number of phishing incidents at your company. This sounds great in theory, but phishing campaigns have a lot of downsides that need to be considered before you hit the send button.

What Is Phishing?

The Cybersecurity and Infrastructure Security Agency (CISA) defines phishing as:

“a form of social engineering where malicious actors lure victims (typically via email) to visit a malicious site or deceive them into providing login credentials.”

What does this mean for a CISO in practical terms? It means your employees will constantly receive emails that look legitimate, but are actually scams. They are trying to get your employees to click on links in the email so they can steal credentials, install malware, get access to sensitive data, or steal money. Phishing campaigns are often one of the first methods attempted in a more targeted attack that can use the phished credentials to allow the attacker to gain a foothold into your environment.

What Are The Common Defenses Against Phishing?

User Awareness Training

One of the most effective ways to counter the threat of phishing attacks is to educate your users. Regular user awareness training on how to recognize and take action against phishing emails has proven to be highly effective. Why? Phishing is trying to trick your users into performing an action they wouldn’t normally perform. This is a form of phycological or social engineering and the best way to instill the proper mindset in someone is through regular training. This training should test for understanding and the ability for users to recognize and report phishing emails. When in doubt, report it and delete it.

DMARC, SPF and DKIM

Domain-based Message Authentication, Reporting and Conformance (DMARC), Sender Policy Framework (SPF) and Domain Keys Identified Mail (DKIM) are technologies that can be implemented by businesses to verify the sender of incoming email and authenticate that incoming messages are valid (not spoofed). Technologies such as SPF allow domains to publish lists of IPs and servers that will send emails, and DKIM allows domain owners to digitally sign emails coming from their domain so recipients can cryptographically validate the messages. SPF and DKIM are forms of authentication in the email world and help prevent spammers from sending mail on domains they don’t own.

DMARC is the enforcement arm in the email world. It takes the output from SPF and DKIM and takes action. This action can be configured based on your organizational preferences, but typical actions for messages that fail SPF or DKIM checks are to deliver the message, mark as SPAM or reject the message entirely. When configured properly, all three of these technologies will help filter and reduce potential phishing emails that make their way into your user’s inboxes.

MFA

Another technology that is critically important to protect against phishing attacks is to enable Multi-Factor Authentication (MFA). This is another form of defense that will protect your user accounts if a phishing email makes it through the filters and your user clicks on the phishing link in the email.

For example, a typical phishing email may impersonate a legitimate business website that requires authentication. The formatting, graphics and appearance may all look exactly the same. The only way to tell the email is a phishing email is by looking at the sender domain or email headers to detect subtle variances in spelling or formatting. If a user falls for this phishing email, clicks on it and enters their username and password, MFA will help prevent their credentials from being fully compromised. Yes, the user will need to have their password changed, but MFA such as one time passwords, tokens or passkeys, will prevent the attackers from using the phished credentials.

What Are Phishing Campaigns?

Phishing campaigns are controlled email campaigns sent by your own organization or a contracted third party to send fake phishing emails to your users to test how many open and click on the phishing links. Phishing campaigns allow organizations to directly test how well their user awareness training is working to recognize and avoid phishing attacks. Phishing campaigns can be stand alone events or they can be tied into other security testing like penetration tests.

What Are the Downsides To Phishing Campaigns?

Phishing campaigns, while popular, have questionable morality and effectiveness for a few reasons.

  1. The primary method of business communication is email. Phishing campaigns are teaching users to mistrust and in some cases stop using email for business purposes. Security organizations should find ways to support and protect the business without unnecessarily impeding it and for this reason I believe phishing campaigns are counter to the mission of an effective security organization.
  2. The top businesses have cultures that support and encourage psychological safety. Being able to respectfully speak your mind, have support from your colleagues and feel valued are all important aspects for job satisfaction and effectiveness. Phishing campaigns go against the idea of psychological safety. They attempt to trick your users into clicking on emails with questionable tactics such as promising bonuses, legitimate business purposes or even funny cat videos.
  3. One large problem with phishing campaigns is they tend to have punitive outcomes. Anyone that falls for the phishing email gets sent to remedial training or may be given a reduced set of permissions for a period of time. These punitive actions punish users for using their primary method of communication, destroy the concept of psychological safety and discourage productivity.
  4. Speaking of productivity, I see a lot of metrics about the percentage of users that clicked on phishing campaign emails along with targets to reduce those numbers after sending people to remedial training. What I don’t see are metrics on the impact the campaign has to productivity. How much longer will it take the person on finance to do their job now that she doesn’t trust anything in her email? How much longer will it take IT support to resolve the help desk ticket when they have been scolded repeatedly for falling for phishing emails? These metrics unfortunately are overlooked or not even captured. Security programs should ground their activities against the overall business strategy and make sure their programs are generating true value for the business that is measurable in the form of reduced risk as a tradeoff to other areas of the business.

A More Effective Solution

Whenever someone asks me for my thoughts about phishing campaigns I tell them honestly that I am not a fan. I’ve been on penetration testing teams that have crafted emails as part of phishing campaigns and I’ve seen the effect it has on users. I think there is a better way.

A lot of this post has gathered inspiration from various sources, but one of the main sources is the Cybersecurity and Infrastructure Agency (CISA). In October 2023, CISA the FBI and the NSA published a joint article on guidance for stopping phishing attacks. You can read their excellent recommendations here. Their article supports my sentiments here because one thing that is not in their recommendations is conducting a phishing campaign against your own users.

What are my recommendations?

  1. Conduct proactive training that tests not only comprehension, but the ability to accurately recognize phishing emails. Give employees the skills to look at email headers and give them the latitude to report suspicious emails or delete them altogether. Accept that email will be a slower and less trusted form of communication and even prevent the use of email for critical business functions (like contracting or financial activities). This training should have hands on practicals that gives the security function and senior leaders confidence they have trained their users to the best of their abilities to minimize the risk.
  2. Put controls in place to protect your employees. DMARC, DKIM, SPF and MFA can protect your users. Endpoint protection, monitoring and logging, ingress and egress filtering, etc. can all provide defense in depth to stop phishing attacks from being successful. The point here is, a comprehensive and well executed security program is one of your best defenses against phishing attacks.
  3. Employees that fall for real world phishing emails should be given a second chance. Assume good intent here. Most employees will recognize when they have done something bad and will feel guilty about it. They will punish themselves so the organization should support them, offer them additional training and help them get back to doing their job. Having proper security controls in place can help minimize the impact of your employees clicking on phishing emails.
  4. As a last resort, I will recommend some sort of punitive action, but this should not be the default and should be used sparingly. For users that just don’t get it and are repeat offenders they should face disciplinary action such as termination or reduced job responsibilities. This ties into organizations that evaluate how well employees support and uphold the security objectives of the organization. Repeated violations of Acceptable Use Policies (AUP) should fall under an HR/Legal action that minimizes the risk of the employee to the business.

Wrapping Up

Phishing is a form of social engineering and is a real business risk. It can lead to credential compromise, malware infections or Business Email Compromise (BEC) resulting in real business loss. A well rounded and comprehensive security program will help counter the threat of phishing attacks through comprehensive security controls and processes. Most importantly, I recommend security teams remove phishing campaigns from their tool chest and instead use proactive techniques to educate users, while protecting the business with a defense in depth approach.

Using Exceptions As A Discovery Tool

Security exceptions should be used sparingly and should be truly exceptional circumstances that are granted after the business accepts a risk. In mature security programs the security exceptions process is well defined and has clear criteria for what will and will not meet the exception criteria. In mature programs exceptions should be the exception, not the norm. However, in newer security programs exceptions can be a useful tool that provides discovery as well as risk acceptance.

Maturing A Security Program

One of the first things a new CISO will need to do is understand the business and how it functions. As part of this process the CISO will need to take an inventory of the current state of things so he or she can begin to form a strategy on how to best manage risk. As a new CISO your security program may not have well defined security policies and standards. As you begin to define your program and roll out these policies, the exception process can be a valuable tool that gives the perception of a choice, while allowing the security team to uncover areas of the business that need security improvement. Over time, as the business and security program mature, the CISO can gradually deny any requests to renew or extend these exceptions.

Rolling Out A New Security Process

Another area that is useful to have an exceptions process is when rolling out a new security process. For example, if you are rolling out a new process that will require teams to perform SAST and DAST scanning of their code and fix vulnerabilities before going into production, then allowing security exceptions during the initial rollout of the process can be useful to allow teams more time to adapt their development processes to incorporate the new security process. Allowing exceptions can foster good will with the development team and allow the security function visibility into the behavior and culture of the rest of the business. This can allow the security function and development team the opportunity to collaborate together with the ultimate goal of removing any exceptions and following the process to reduce risk to the business.

Tackling Security Tech Debt or Shadow IT

A common maturity evolution for companies is the elimination of shadow IT. The security function can assist with the elimination of shadow IT by creating an exception process and allowing an amnesty period where the business is allowed to continue to operate their shadow IT as long as it is declared. In reality you are giving the business the perception that they will be granted an exception when they are really giving the security function visibility into things they wouldn’t otherwise know about. This can be a useful tool to discover and eliminate policy exceptions as long as it is used sparingly and with good intent (not punitively).

Documentation Is Key

No matter how you choose to use exceptions within your security program there are a few best practices to follow.

  1. Exceptions should be truly exceptional. If you do grant one for discovery purposes make sure there is a plan to close the exception. Exceptions shouldn’t be the rule and they shouldn’t be expected. Sometimes the rest of the business just needs someone to tell them no.
  2. Time box the exception. Don’t just grant an exception without some sort of end date. The business needs to know an exception is temporary and there should be a well defined plan to make improvements and close the exception. The security team should grant a reasonable amount of time to execute that plan, but it shouldn’t be a never ending story.
  3. Review often. Security exceptions should be reviewed often. Part of your security program should review the open exceptions, which ones are ending, if there are patterns where there are lots of similar exceptions and if there are teams who request a high volume of exceptions. Reviewing exceptions gives you insight into how well security processes and controls are working. It also gives you insight into which parts of the business need help.
  4. Require the business owner to sign off. The reality of a well run security program is the business ultimately owns the decision if they want to accept a risk or not. The CISO makes a recommendation, but they don’t own the business systems or processes. As a result, the security exception process should require the business owner to sign off on any exception. This will ensure there is documentation that they were made aware of the risk, but this can also act as a visibility tool for the business owner into their own teams. I’ve often found a business leader is not always aware of what their teams are doing at the tactical level and the exceptions process can provide them the opportunity to check their team and correct behavior before it gets to the CISO.

Wrapping Up

The exception process can be a valuable tool for discovery of hidden risk throughout the business. By offering an amnesty period and giving the perception of flexibility, the security team can foster good will with the business while gaining valuable visibility into areas that may be hidden. The exception process also is a valuable tool for the security program to document risk acceptance by the applicable business owner, but can also provide business owners visibility into how well their team is meeting security requirements. Lastly, as the security program matures, the security team can gradually require the business to close down the exceptions by improving their security posture.

Defining Your Security Front Door

A key skill for any security program is to partner with and enable the business to be successful. CISOs need to ensure their security teams are approachable, reasonable and most importantly balancing the needs of the business against potential security risks. While security teams exist to help protect the business, they don’t own the business systems or processes and as a result need to adopt an advisory or consultative role with the rest of the business to ensure success.

With that in mind, the way the rest of the business finds and engages with the security team can create a good first impression or can set the tone for a difficult interaction. Think of a house that has great curb appeal – it feels inviting and gives the impression that the owners take good care of their property. The same concept exists for the security program, which I call the Security Front Door.

The Front Door Concept

The security front door defines how the rest of the business engages with and interacts with the security team. The front door can be a confluence page, slack channel with pinned information, or some place that is easily discoverable and accessible. Your security front door should clearly lay out information and resources so the rest of the business can either self serve or easily request and receive help when needed.

What Should Be In Your Front Door?

The front door for your security program should include ways to perform the most commonly requested actions from the security team. For example, you probably want really clear ways to request the following:

  • Report an incident
  • Request vulnerability remediation help
  • Request an exception
  • Request an architectural review
  • Dashboards
  • Discover documentation for policies and processes
  • Other – a general way to request help for anything else

The front door is not just a way to make a good first impression and enable the business, but when set up correctly it can actually offload the security team and help the business move faster.

Wrapping Up

The front door is a great way to engage with the business to help them move faster, find information and request assistance from the security team. When done correctly it can allow the rest of the business to self serve and can actually offload the security team by reducing the volume of requests that come in. Setting up the security front door may require a lot of up front work, but by understanding the rest of the business, their key pain points and most commonly requested security asks, you can design a front door that will be a win-win for everyone.