Security Controls – 370 Security Blog

What’s Better – Complete Coverage With Multiple Tools Or Partial Coverage With One Tool?

The debate between complete coverage with multiple tools versus imperfect coverage with one tool regularly pops up in discussions between security professionals. What we are really talking about is attempting to choose between maximum functionality and simplicity. Having pursued both extremes over the course of my security career I offer this post to share my perspective on how CISOs can think about navigating this classic tradeoff.

In Support Of All The Things

Let’s start with why you may want to pursue complete coverage by using multiple technologies and tools.

Heavily Regulated And High Risk Industries

First, heavily regulated and high risk businesses may be required to demonstrate complete coverage of security requirements. These are industries like the financial sector or government and defense. (I would normally say healthcare here, but despite regulations like HIPAA the entire industry has lobbied against stronger security regulations and this has proven disastrous via major incidents like the Change Healthcare Ransomware Attack). The intent behind any regulation is to establish a minimum set of required security controls businesses need to meet in order to operate in that sector. It may not be possible to meet all of these regulatory requirements with a single technology and therefore, CISOs may need to evaluate and select multiple technologies to meet the requirements.

Defense In Depth

Another reason for selecting multiple tools is to provide defense in depth. The thought process is: multiple tools will provide overlap and small variances in how they meet various security controls. These minor differences can offer defenders an advantage because if one piece of technology is vulnerable to an exploit, another piece of technology may not be vulnerable. By layering these technologies throughout your organization you reduce the chances an attacker will be successful.

An example of this would be if your business is protected from the internet by a firewall made by Palo Alto. Behind this PA firewall is a DMZ and the DMZ is separated from your internal network by a firewall from Cisco. This layered defense will make it more difficult for attackers to get through the external firewall, DMZ, internal firewall and into the LAN. (See image below for a very simplistic visual)

Downside Of All The Things

All the things may sound great, but unless you are required to meet that level of security there can be a lot of downsides.

First, multiple technologies introduce complexity into an environment. This can make it more difficult to troubleshoot or detect issues (including security events). It can also make it more difficult to operationally support these technologies because they may have different interfaces, APIs, protocols, configurations, etc. It may not be possible to centrally manage these technologies, or it may require the introduction of an additional technology to manage everything.

Second, all of these technologies can increase the number of people required to support them. People time can really add up as a hidden cost and shouldn’t be thrown away lightly. People time starts the second you begin discussing the requirements for a new technology and can include the following:

Proof of Concepts (PoCs)
Tradeoff & Gap Analysis
Requests for Information (RFI)
Requests for Proposal (RFP)
Requests for Quotes (RFQ)
Contract Negotiation
Installation
Integration
Operation & Support

Finally, multiple technologies can cause performance impacts, increased costs and waste. Performance impacts can happen due to differences in technologies, complexity, configuration errors or over consumption of resources (such as agent sprawl). Waste can happen due to overlap and duplicated functionality because not all of the functionality may not get used despite the fact you are paying for it.

Advantages and Disadvantages Of A Single Tool

A single tool that covers the majority, but not all, of your requirements offers one advantage – simplicity. This may not sound like much, but after years of chasing perfection, technology simplicity can have benefits that may not be immediately obvious.

First, seeking out a single tool that meets the majority of requirements will force your security team to optimize their approach for the one that best manages risk while supporting the objectives of the business. Second, a single tool is easier to install, integrate, operate and support. There is also less demand on the rest of the business in terms of procurement, contract negotiation and vendor management. Lastly, a single tool requires less people to manage it and therefore you can run a smaller and more efficient organization.

The biggest disadvantage of a single tool is it doesn’t provide defense in depth. One other disadvantage is it won’t meet all of your security requirements and so the requirements that aren’t met should fall within the risk tolerance of the business or somehow get satisfied with other compensating controls.

A single tool that covers the majority, but not all, of your requirements offers one advantage – simplicity.

Wrapping Up

There are a lot of advantages to meeting all of your requirements with multiple tools, but these advantages come with a tradeoff in terms of complexity, operational overhead, duplicated functionality and increased personnel requirements. If you operate a security program in a highly regulated or highly secure environment you may not have a choice so it is important to be aware of these hidden costs. A single tool reduces complexity, operational overhead and personnel demands, but can leave additional risk unmet and fails to provide defense in depth. Generally, I favor simplicity where possible, but you should always balance the security controls against the risk tolerance and needs of the business.

If Data Is Our Most Valuable Asset, Why Aren’t We Treating It That Way?

There have been several high profile data breaches and ransomware attacks in the news lately and the common theme between all of them has been the disclosure (or threat of disclosure) of customer data. The after effects of a data breach or ransomware attack are far reaching and typically include loss of customer trust, refunds or credits to customer accounts, class action lawsuits, increased cyber insurance premiums, loss of cyber insurance coverage, increased regulatory oversight and fines. The total cost of these after effects far outweigh the cost of implementing proactive security controls like proper business continuity planning, disaster recovery (BCP/DR) and data governance, which begs the question – if data is our most valuable asset, why aren’t we treating it that way?

The Landscape Has Shifted

Over two decades ago, the rise of free consumer cloud services, like the ones provided by Google and Microsoft, ushered in the era of mass data collection in exchange for free services. Fast forward to today, the volume of data growth and the value of that data has skyrocketed as companies have shifted to become digital first or mine that data for advertising purposes and other business insights. The proliferation of AI has also ushered in a new data gold rush as companies strive to train their LLMs on bigger and bigger data sets. While the value of data has increased for companies, it has also become a lucrative attack vector for threat actors in the form of data breaches or ransomware attacks.

The biggest problem with business models that monetize data is: security controls and data governance haven’t kept pace with the value of the data. If your company has been around for more than a few years chances are you have a lot of data, but data governance and data security has been an afterthought. The biggest problem with bolting on security controls and data governance after the fact is it is hard to reign in pandoras box. This is also compounded by the fact that it is hard to put a quantitative value on data, and re-architecting data flows is seen as a sunk cost to the business. The rest of the business may find it difficult to understand the need to rearchitect their entire business IT operations since there isn’t an immediate and tangible business benefit.

Finally, increased global regulation is changing how data can be collected and governed. Data collection is shifting from requiring consumers to opt-out to requiring them to explicitly opt-in. This means consumers and users (an their associated data) will no longer be the presumptive product of these free services without their explicit consent. Typically, increased regulation also comes with specific requirements for data security, data governance and even data sovereignty. Companies that don’t have robust data security and data governance are already behind the curve.

False Sense Of Security

In addition to increased regulation and a shifting business landscape, the technology for protecting data really hasn’t changed in the past three decades. However, few companies implement effective security controls on their data (as we continue to see in data breach notifications and ransomware attacks). A common technology used to protect data is encryption at rest and encryption in transit (TLS), but these technologies are insufficient to protect data from anything except physical theft and network snooping (MITM). Both provide a false sense of security related to data protection.

Furthermore, common regulatory compliance audits don’t sufficiently specify protection of data throughout the data lifecycle beyond encryption at rest, encryption in transit and access controls. Passing these compliance audits can give a company a false sense of security that they are sufficiently protecting their data, when the opposite is true.

Just because you passed your compliance audit, doesn’t mean you are good to go from a data security and governance perspective.

Embrace Best Practices

Businesses can get ahead of this problem to make data breaches and ransomware attacks a non-event by implementing effective data security controls and data governance, including BCP/DR. Here are some of my recommendations for protecting your most valuable asset:

Stop Storing and Working On Plain Text Data

Sounds simple, but this will require significant changes to business processes and technology. The premise is the second data hits your control it should be encrypted and never, ever, unencrypted. This means data will be protected even if an attacker accesses the data store, but it also will mean the business will need to figure out how to modify their operations to work on encrypted data. Recent technologies such as homomorphic encryption have been introduced to solve these challenges, but even simpler activities like tokenizing the data can be an effective solution. Businesses can go one step further and create a unique cryptographic key for every “unique” customer. This would allow for simpler data governance, such as deletion of data.

Be Ruthless With Data Governance

Storage is cheap and it is easy to collect data. As a result companies are becoming digital data hoarders. However, to truly protect your business you need to ruthlessly govern your data. Data governance policies need to be established and technically implemented before any production data touches the business. These policies need to be reviewed regularly and data should be purged the second it is no longer needed. A comprehensive data inventory should be a fundamental part of your security and privacy program so you know where the data is, who owns it and where the data is in the data lifecycle.

The biggest problem with business models that monetize data is: security controls and data governance haven’t kept pace with the value of the data.

Ruthlessly governing data can have a number of benefits to the business. First, it will help control data storage costs. Second, it will minimize the impact of a data breach or ransomware attack to the explicit time period you have kept data. Lastly, it can protect the business from liability and lawsuits by demonstrating the data is properly protected, governed and/or deleted. (You can’t disclose what doesn’t exist).

Implement An Effective BCP/DR and BIA Program

Conducting a proper Business Impact Analysis (BIA) of your data should be table stakes for every business. Your BIA should include what data you have, where it is and most importantly, what would happen if this data wasn’t available? Building on top of the BIA should be a comprehensive BCP/DR plan that appropriately tiers and backs up data to support your uptime objectives. However, it seems like companies are still relying on untested BCP/DR plans or worse solely relying on single cloud regions for data availability.

Every BCP/DR plan should include a write once, read many (WORM) backup of critical data that is encrypted at the object or data layer. Create WORM backups to support your RTO and RPO and manage the backups according to your data governance plan. Having a WORM backup will prevent ransomware attacks from being able to encrypt the data and if there is a data breach it will be meaningless because the data is encrypted. BCP / DR plans should be regularly tested (up to full business failover) and security teams need to be involved in the creation of BCP/DR plans to make sure the data will have the confidentiality, integrity and availability when needed.

Don’t Rely On Regulatory Compliance Activities As Your Sole Benchmark

My last recommendation for any business is – just because you passed your compliance audit, doesn’t mean you are good to go from a data security and governance perspective. Compliance audits exist as standards for specific industries to establish a minimum bar for security. Compliance standards can be watered down due to industry feedback, lobbying or legal challenges and a well designed security program should be more comprehensive than any compliance audit. Furthermore, compliance audits are typically tailored to specific products and services, have specific scopes and limited time frames. If you design your security program to properly manage the risks to the business, including data security and data governance, you should have no issues passing a compliance audit that assesses these aspects.

Wrapping Up

Every business needs to have proper data security and data governance as part of a comprehensive security program. Data should never be stored in plain text and it should be ruthlessly governed so it is deleted the second it is no longer needed. BCP/DR plans should be regularly tested to simulate data loss, ransomware attacks or other impacts to data and, while compliance audits are necessary, they should not be the sole benchmark for how you measure the effectiveness of your security program. Proper data protection and governance will make ransomware and data breaches a thing of the past, but this will only happen if businesses stop treating data as a commodity and start treating it as their most valuable asset.

Security Theater Is The Worst

We have all been there…we’ve had moments in our life where we have had to “comply” or “just do it” to meet a security requirement that doesn’t make sense. We see this throughout our lives when we travel, in our communities and in our every day jobs. While some people may think security theater has merit because it “checks a box” or provides a deterrent, in my opinion security theater does more harm than good and should be eradicated from security programs.

What Is Security Theater?

Security theater was first coined by Bruce Schneier and refers to the practice of implementing security measures in the form of people, processes or technologies that give the illusion of improved security. In practical terms, this means there is something happening, but what that something is and how it actually provides any protection is questionable at best.

Examples Of Security Theater

Real life examples of security theater can be seen all over the place, particularly when we travel. The biggest travel security theater is related to liquids. TSA has a requirement that you can’t bring liquids through security unless they are 3 ounces or smaller. However, you can bring a bottle of water through if it is fully frozen…what? Why does being frozen matter? What happens if I bring 100, 3 ounce shampoo bottles through security? I still end up with the same volume of liquid and security has done nothing to prevent me from bringing the liquid through. As for water, the only thing that makes sense for why they haven’t relaxed this requirements is to prop up the businesses in the terminal that want to sell overpriced bottles of water to passengers. Complete theater.

“Security theater is the practice of implementing security measures that give the illusion of improved security.”

Corporate security programs also have examples of security theater. This can come up if you have an auditor that is evaluating your security program against an audit requirement and they don’t understand the purpose of the requirement. For example, and auditor may insist you install antivirus on your systems to prevent viruses and malware, when your business model is to provide Software as a Service (SaaS). With SaaS your users are consuming software in a way that nothing is installed on their end user workstations and so there is little to no risk of malware spreading from your SaaS product to their workstations. Complete theater.

Another example of security theater is asking for attestation a team is meeting a security requirement instead of designing a process or security control that actually achieves the desired outcome. In this example, the attestation is nothing more than a facade designed to pass accountability from the security team, that should be designing and implementing effective controls, to the business team. It is masking ineffective process and technologies. Complete theater.

Lastly, a classic example of security theater is security by obscurity. Otherwise known as hiding in plain sight. If your security program is relying on the hope that attackers won’t find something in your environment then prepare to be disappointed. Reconnaissance tools are highly effective and with enough time threat actors will find anything you are trying to hide. Hope is not a strategy. Complete theater.

What Is The Impact Of Security Theater?

Tangible And Intangible Costs

Everything we do in life has a cost and this is certainly true with security theater. In the examples above there is a real cost in terms of time and money. People who travel are advised to get to the airport at least two hours early. This cost results in lost productivity, lost time with family and decreased self care.

In addition to tangible costs like those above, there are also intangible costs. If people don’t understand the “why” for your security control, they won’t be philosophically aligned to support it. The end result is security theater will erode confidence and trust in your organization, which will undermine your authority. This is never a place you want to be as a CISO.

Some people may argue that security theater is a deterrent because the show of doing “security things” will deter bad people from doing bad things. This sounds more like a hope than reality. People are smart. They understand when things make sense and if you are implementing controls that don’t make sense they will find ways around them or worse, ignore you when something important comes up.

With any effective security program the cost of a security control should never outweigh the cost of the risk, but security theater does exactly that.

Real Risks

The biggest problem with security theater is it can give a false sense of security to the organization that implements it. The mere act of doing “all the things” can make the security team think they are mitigating a risk when in reality they are creating the perfect scenario for a false negative.

How To Avoid Security Theater?

The easiest way to avoid security theater is to have security controls that are grounded in sound requirements and establish metrics to evaluate their effectiveness. Part of your evaluation should evaluate the cost of the control versus the cost of the risk. If your control costs more than the risk then it doesn’t make sense and you shouldn’t do it.

The other way to avoid security theater is to exercise integrity. Don’t just “check the box” and don’t ask the business you support to check the box either. Take the time to understand requirements from laws, regulations and auditors to determine what the real risk is. Figure out what an effective control will be to manage that risk and document your reasoning and decision.

The biggest way to avoid security theater is to explain the “why” behind a particular security control. If you can’t link it back to a risk or business objective and explain it in a way people will understand then it is security theater.

Can we stop with all the theater?

What’s The Relationship Between Security Governance and Organizational Maturity?

Organizational and security governance is touted as a key component of any successful security program. However, I’ve been thinking about governance lately and how it relates to the overall maturity of an organization. This has prompted some questions such as: what happens if you have too much governance? and What’s the relationship between security governance and organizational maturity?

What Is Governance?

First, let’s talk about what governance is.

Governance is the process by which an organization defines, implements and controls the business.

Let’s unpack what this means for a security organization. The process of defining security for the business is done through policies, standards and guidelines. Security policies are requirements the business must meet based on laws, regulations or best practices adopted by the business. These policies align to business objectives. Implementation is done through security controls that are put in place to meet a specific policy or to manage a risk. Lastly, controlling the business is done via audits and compliance checks. The security org follows up on how well the business is following policies, implementing controls and managing risk. Control can also include enforcement, which can involve gating processes, such as requiring approval for business critical and high risk activities, or recommending additional security requirements for the business to manage a risk.

Why Do We Need Governance At All?

In an ideal world we wouldn’t. Imagine a business that is created entirely of clones of yourself. There would be implicit and explicit trust between you and your other selves to do what is best for the business. Communication would be simple and you would already be aligned. In this case you don’t need a lot (or any) governance because you can trust yourself to do the things. However, unless you are Michael Keaton in Multiplicity, this just isn’t a reality.

Governance achieves a few things for a business. First, it communicates what is required of its employees and aligns those employees to common objectives. Second, it helps employees prioritize activities. None of this would be needed if human’s weren’t so complex with diverse backgrounds, experiences, perspectives, education, etc. In an ideal world we wouldn’t need any governance at all. The reality is, we do need governance, but it needs to be balanced so it doesn’t unnecessarily impede the business.

How Does This Relate To Organizational Maturity?

Organizational maturity refers to how your employees are able to execute their tasks to achieve the objectives of the business. This relates to things like the quality of code, how quickly teams resolve operational issues or how efficiently they perform a series of tasks. It can be loosely thought of as efficiency, but I actually think it combines efficiency with professionalism and integrity. Maturity is knowing what good is and being able to execute efficiently to get there. There is a fantastic book about this topic called Accelerate: The Science of Lean Software and DevOps: Building High Performing Technology Organizations by Nicole Forsgren PhD.

Which brings us to the relationship of governance and maturity…

There is an inverse relationship between organizational maturity and organizational governance. In simple terms:

The less mature an organization, the more governance is needed.

For example, if your organization struggles to apply patches in a timely manner, continually introduces new code vulnerabilities into production or repeatedly demonstrates behavior that places the business at risk, then your organizational maturity is low. When organizational maturity is low, the business needs to put processes and controls in place to align employees and direct behavior to achieve the desired outcomes. In the examples above, increased governance is an attempt to manage risk because your employees are behaving in a way that lacks maturity and is placing the business at risk.

What causes low organizational maturity?

Organizational maturity is a reflection of employee behavior, skillset, knowledge, education and alignment. In other words, organizational maturity is a reflection of your organizational culture. In practical terms your employees may simply not know how to do something. They may not have experience with working for your type of business or in the industry you operate in. Perhaps they had a really bad boss at a past job and learned bad behavior. Whatever the reason, low organizational maturity is linked to lots of sub-optimal outcomes in business.

How To Improve Organizational Maturity?

If governance and maturity are inversely linked, the question becomes how can we increase organizational maturity so we need less governance? There are a lot of ways to increase organizational maturity. One that is fairly obvious is to start with a mature organization and maintain it over time. However, this is easier said than done and is why some organizations are fanatical about culture. This relates to everything from hiring to talent management and requires strong leadership at all levels of the company.

“The less mature an organization,
the more governance is needed.”

Other ways to improve organizational maturity are through training and education. This is why security awareness and training programs are so critical to a successful security program. Security awareness and training programs are literally attempting to improve organizational maturity through education.

One last way to improve maturity is via process. The security organization can establish a new process that all teams must follow. As teams go through this process you can educate them and reward teams that exhibit the ideal behavior by relaxing the process for them. You can also help teams educate themselves by publishing the requirements and making the process transparent. The challenge with imposing a new process is having the discipline to modify or remove the process when needed, which comes back to governance.

What’s the right level of governance?

The optimal level of governance is going to be based on your organizational maturity and desired business outcomes. In order to determine if you have too much or too little governance you need to measure organizational maturity and the effectiveness of existing organizational governance. There are industry standard processes for measuring organizational maturity, like the Capability Maturity Model Integration (CMMI) and Six Sigma, or you can create your own metrics. Some ways to measure governance effectiveness are:

Ask For Feedback On Security Processes – Are the processes effective? Do teams view them as an impediment or are they viewed favorably? Are the processes easy to navigate and objective or are they opaque and subjective?
Measure Effectiveness Of Security Controls – Are your security controls effective? If you ask a team to do work to implement a security control you should have clear metrics that determine if that control is effective. If you implement a control, but that control hasn’t changed the outcome, then the control is ineffective. This can indicate your governance is ineffective or your organizational maturity needs to improve.
Assess and Update Policy – Security policies should be living documents. They shouldn’t be set in stone. Security policies need to map back to laws and regulations they support and the business requirements needed to be successful. Laws, regulations and business requirements all change over time and so should your security policies. By having up to date and relevant security policies you can ensure your organizational governance matches the maturity of the business.

What Are Typical Scenarios For Governance And Maturity?

There are four scenarios related to governance and maturity:

A mature organization with too much governance – your organization is mature, but you are overly controlling with process and requirements. The net effect will be to slow down and impede the business unnecessarily. You are effectively lowering the organizational maturity due to too much governance.

An immature organization with too little governance – this is a recipe for disaster. If your organization is immature and you fail to govern the organization you will open the business up to unnecessary risk. You will get out maneuvered by your competitors, you will miss opportunities, you will fail to comply with laws and regulations and generally will have a lot of activity without any result. Your employees will lack coordination and as a result your business will suffer.

A mature organization with too little governance – This isn’t a bad scenario to be in. A mature organization implies they are doing the right things and don’t need a lot of guidance. A laissez faire attitude may be the right thing to allow employees flexibility and freedom, but it does come with inherent risk of not being compliant with laws and regulations. It may also mean there is duplication of effort or multiple ways of doing things, which could be optimized.

Governance and maturity are balanced – obviously this is the ideal scenario where your organizational governance is balanced to the level of maturity of the organization. Easy to think about in practice, difficult to achieve in reality.

Wrapping Up

Organizational governance and maturity are inversely related and need to be balanced in order for the business to operate effectively. There are ways to measure organizational maturity and governance effectiveness and by having a continual feedback loop you can optimally align both for success.

We Are Drowning In Patches (and what to do about it)

Last week I had an interesting discussion with some friends about how to prioritize patches using criticality and a risk based approach. After the discussion I starting thinking about how nice it would be if we could all just automatically patch everything and not have to worry about prioritization and the never ending backlog of patches, but unfortunately this isn’t a reality for the majority of organizations.

Whats the problem?

There are several issues that create a huge backlog of patches for organizations.

First, let’s talk about the patching landscape organizations need to deal with. This is largely spit into two different areas. The first area is operating system (OS) and service patches. These are patches that are released periodically for the operating systems used by the business to run applications or products. Common operating systems for production workloads will be either Windows or Linux and will have stability, security or new feature patches released periodically.

Second, there are patches for software libraries that are included in the software and applications developed by your business. Typically these are lumped into the category of 3rd party libraries, which means your organization didn’t write these libraries, but they are included in your software. 3rd party library security vulnerabilities have become a huge issue over the last decade (but thats a blog post for another day).

These two patch types, OS and 3rd party library patches, require different approaches to discover, manage and remediate, which is the first challenge for auto patching. When combined with the volume of new vulnerabilities being discovered, large heterogeneous environments and the need to keep business critical applications available, keeping your assets patched and up to date becomes a real challenge.

Why isn’t auto patching a thing?

Well it is, but…

There are a few challenges to overcome before you can auto-patch.

Stability and Functionality

First, both operating system and 3rd party library patches need to be tested for stability and functionality. Usually, patches fix some sort of issue or introduce new features, but this can cause issues in other areas such as stability or functionality. It can be a complex process to roll back patches and restore business critical applications to a stable version, which is why most businesses test their patches in a staging environment before rolling them out to production. Cash is king and businesses want to minimize any disruption to cash flow.

Investment and Maturity

It is possible to automate testing for stability and functionality, but this requires a level of maturity and investment that most organizations haven’t achieved. For example, assuming your staging environment is a mirror image of your production environment (it is right?), you could auto apply the patches in staging, automatically check for stability and functionality over a set period of time and then roll those updates to production with minimal interaction. However, if your environment requires reboots or you have limited resources, patching may require down time, which could impact making money.

In order to have an environment that can support multiple versions, seamless cut over, proper load balancing, caching, etc. requires significant investment. Typically this investment is useful for keeping your products functioning and making money even if something goes wrong, but this investment can also be used to buffer maintenance activities such as patching without disruption.

Software Development Lifecycle

The last section assumes a level of software development maturity such as adoption of Agile development processes and CI/CD (continuous integration / continuous delivery). However, if your engineering group uses a different development process such as Incremental or Waterfall, then patching may become even more difficult because you are now competing with additional constraints and priorities.

What are some strategies to prioritize patching and reduce volume?

If your business runs products that aren’t mission critical, or you simply can’t justify the investment to operate an environment without down time, then auto patching probably isn’t a reality for you unless you are Austin Powers and like to live dangerously. For most organizations, you will need to come up with a strategy to prioritize patching and reduce the volume down to a manageable level.

Interestingly, this problem space has had a bunch of brain power dedicated to it over the years because it resembles a knapsack problem, which is a common problem space in mathematics, computer science and economics. Knapsack problems are problems where you have a finite amount of a resource (space, time, etc.) and you want to optimize the use of that resource to maximize some sort of requirement (like value). In the case of patching, this would mean applying the largest volume of the highest severity patches in a fixed time period to realize the maximum risk reduction possible.

Critical Assets First

Staying in the knapsack problem space, one strategy is to start with your most critical assets and apply the highest severity patches until you reach your threshold for risk tolerance. This requires your organization to have an up to date asset inventory and have categorized your assets based on business criticality and risk. For example, let’s say you have two applications at your business. One is a mission critical application for customers and generates 80% of your annual revenue. The other application provides non-mission critical functionality and accounts for the other 20% of revenue. Your risk tolerance based on your company policies is to apply all critical and high patches within 72 hours of release. In this example you would apply all critical and high patches to the mission critical application as quickly as possible (assuming other requirements are met like availability, etc.).

Guard Rails and Gates

Another strategy for reducing volume is to have guard rails or gates as part of your software development lifecycle. This means your engineering teams will be required to pass through these gates at different stages before being allowed to go to production. For example, your organization may have a policy that no critical vulnerabilities are allowed in production applications. The security organization creates a gate that scans for OS and 3rd party library vulnerabilities whenever an engineering team attempts to make changes to the production environment (like pushing new features). This way the engineering team needs to satisfy any vulnerability findings and apply patches at regular intervals coinciding with changes to production.

Wrapping Up

With the proliferation of open source software, the speed of development and the maturity of researchers and attackers to find new vulnerabilities, patching has become an overwhelming problem for a lot of organizations. In fact, it is such a big problem CISA and the Executive Order On Improving The Nation’s Cybersecurity list software patches and vulnerabilities as a key national security issue. I’ve outlined a few strategies to prioritize and reduce the volume of patches if your organization can’t afford the investment to absorb downtime without disruption. However, no matter what strategy you choose, all of them require strong fundamentals in asset inventory, asset categorization and defined risk tolerance. While these investments may seem tedious at first, the more disciplined you are about enforcing the security fundamentals (and engineering maturity), the less you will drown in patches and the closer your organization will come to the reality of auto-patching.

Are Phishing Campaigns Worth It?

Phishing campaigns are often touted as a complementary exercise to security training as a way to measure training effectiveness. The thought is, if your training is effective, users will be less likely to fall for and click on phishing emails, which will correlate to a decrease in the number of phishing incidents at your company. This sounds great in theory, but phishing campaigns have a lot of downsides that need to be considered before you hit the send button.

What Is Phishing?

The Cybersecurity and Infrastructure Security Agency (CISA) defines phishing as:

“a form of social engineering where malicious actors lure victims (typically via email) to visit a malicious site or deceive them into providing login credentials.”

What does this mean for a CISO in practical terms? It means your employees will constantly receive emails that look legitimate, but are actually scams. They are trying to get your employees to click on links in the email so they can steal credentials, install malware, get access to sensitive data, or steal money. Phishing campaigns are often one of the first methods attempted in a more targeted attack that can use the phished credentials to allow the attacker to gain a foothold into your environment.

What Are The Common Defenses Against Phishing?

User Awareness Training

One of the most effective ways to counter the threat of phishing attacks is to educate your users. Regular user awareness training on how to recognize and take action against phishing emails has proven to be highly effective. Why? Phishing is trying to trick your users into performing an action they wouldn’t normally perform. This is a form of phycological or social engineering and the best way to instill the proper mindset in someone is through regular training. This training should test for understanding and the ability for users to recognize and report phishing emails. When in doubt, report it and delete it.

DMARC, SPF and DKIM

Domain-based Message Authentication, Reporting and Conformance (DMARC), Sender Policy Framework (SPF) and Domain Keys Identified Mail (DKIM) are technologies that can be implemented by businesses to verify the sender of incoming email and authenticate that incoming messages are valid (not spoofed). Technologies such as SPF allow domains to publish lists of IPs and servers that will send emails, and DKIM allows domain owners to digitally sign emails coming from their domain so recipients can cryptographically validate the messages. SPF and DKIM are forms of authentication in the email world and help prevent spammers from sending mail on domains they don’t own.

DMARC is the enforcement arm in the email world. It takes the output from SPF and DKIM and takes action. This action can be configured based on your organizational preferences, but typical actions for messages that fail SPF or DKIM checks are to deliver the message, mark as SPAM or reject the message entirely. When configured properly, all three of these technologies will help filter and reduce potential phishing emails that make their way into your user’s inboxes.

MFA

Another technology that is critically important to protect against phishing attacks is to enable Multi-Factor Authentication (MFA). This is another form of defense that will protect your user accounts if a phishing email makes it through the filters and your user clicks on the phishing link in the email.

For example, a typical phishing email may impersonate a legitimate business website that requires authentication. The formatting, graphics and appearance may all look exactly the same. The only way to tell the email is a phishing email is by looking at the sender domain or email headers to detect subtle variances in spelling or formatting. If a user falls for this phishing email, clicks on it and enters their username and password, MFA will help prevent their credentials from being fully compromised. Yes, the user will need to have their password changed, but MFA such as one time passwords, tokens or passkeys, will prevent the attackers from using the phished credentials.

What Are Phishing Campaigns?

Phishing campaigns are controlled email campaigns sent by your own organization or a contracted third party to send fake phishing emails to your users to test how many open and click on the phishing links. Phishing campaigns allow organizations to directly test how well their user awareness training is working to recognize and avoid phishing attacks. Phishing campaigns can be stand alone events or they can be tied into other security testing like penetration tests.

What Are the Downsides To Phishing Campaigns?

Phishing campaigns, while popular, have questionable morality and effectiveness for a few reasons.

The primary method of business communication is email. Phishing campaigns are teaching users to mistrust and in some cases stop using email for business purposes. Security organizations should find ways to support and protect the business without unnecessarily impeding it and for this reason I believe phishing campaigns are counter to the mission of an effective security organization.
The top businesses have cultures that support and encourage psychological safety. Being able to respectfully speak your mind, have support from your colleagues and feel valued are all important aspects for job satisfaction and effectiveness. Phishing campaigns go against the idea of psychological safety. They attempt to trick your users into clicking on emails with questionable tactics such as promising bonuses, legitimate business purposes or even funny cat videos.
One large problem with phishing campaigns is they tend to have punitive outcomes. Anyone that falls for the phishing email gets sent to remedial training or may be given a reduced set of permissions for a period of time. These punitive actions punish users for using their primary method of communication, destroy the concept of psychological safety and discourage productivity.
Speaking of productivity, I see a lot of metrics about the percentage of users that clicked on phishing campaign emails along with targets to reduce those numbers after sending people to remedial training. What I don’t see are metrics on the impact the campaign has to productivity. How much longer will it take the person on finance to do their job now that she doesn’t trust anything in her email? How much longer will it take IT support to resolve the help desk ticket when they have been scolded repeatedly for falling for phishing emails? These metrics unfortunately are overlooked or not even captured. Security programs should ground their activities against the overall business strategy and make sure their programs are generating true value for the business that is measurable in the form of reduced risk as a tradeoff to other areas of the business.

A More Effective Solution

Whenever someone asks me for my thoughts about phishing campaigns I tell them honestly that I am not a fan. I’ve been on penetration testing teams that have crafted emails as part of phishing campaigns and I’ve seen the effect it has on users. I think there is a better way.

A lot of this post has gathered inspiration from various sources, but one of the main sources is the Cybersecurity and Infrastructure Agency (CISA). In October 2023, CISA the FBI and the NSA published a joint article on guidance for stopping phishing attacks. You can read their excellent recommendations here. Their article supports my sentiments here because one thing that is not in their recommendations is conducting a phishing campaign against your own users.

What are my recommendations?

Conduct proactive training that tests not only comprehension, but the ability to accurately recognize phishing emails. Give employees the skills to look at email headers and give them the latitude to report suspicious emails or delete them altogether. Accept that email will be a slower and less trusted form of communication and even prevent the use of email for critical business functions (like contracting or financial activities). This training should have hands on practicals that gives the security function and senior leaders confidence they have trained their users to the best of their abilities to minimize the risk.
Put controls in place to protect your employees. DMARC, DKIM, SPF and MFA can protect your users. Endpoint protection, monitoring and logging, ingress and egress filtering, etc. can all provide defense in depth to stop phishing attacks from being successful. The point here is, a comprehensive and well executed security program is one of your best defenses against phishing attacks.
Employees that fall for real world phishing emails should be given a second chance. Assume good intent here. Most employees will recognize when they have done something bad and will feel guilty about it. They will punish themselves so the organization should support them, offer them additional training and help them get back to doing their job. Having proper security controls in place can help minimize the impact of your employees clicking on phishing emails.
As a last resort, I will recommend some sort of punitive action, but this should not be the default and should be used sparingly. For users that just don’t get it and are repeat offenders they should face disciplinary action such as termination or reduced job responsibilities. This ties into organizations that evaluate how well employees support and uphold the security objectives of the organization. Repeated violations of Acceptable Use Policies (AUP) should fall under an HR/Legal action that minimizes the risk of the employee to the business.

Wrapping Up

Phishing is a form of social engineering and is a real business risk. It can lead to credential compromise, malware infections or Business Email Compromise (BEC) resulting in real business loss. A well rounded and comprehensive security program will help counter the threat of phishing attacks through comprehensive security controls and processes. Most importantly, I recommend security teams remove phishing campaigns from their tool chest and instead use proactive techniques to educate users, while protecting the business with a defense in depth approach.