Month: July 2023

The Problem With Vulnerability Scanners

Vulnerability scanners are table stakes for any security program and they help security teams proactively identify and report on the security posture of assets, but unless you tune them properly they can lead to more problems than they fix. Here are a few things you need to take into consideration when selecting and using a vulnerability scanner as part of your security tooling.

Scanning Technique

There are two primary scanning techniques used by vulnerability scanners. The first is an unauthenticated scan, which is essentially an external scan that attempts to enumerate the ports and services running on the system or device and then match those up to known vulnerabilities. The advantage of an unauthenticated scan is it is usually easier to implement because you don’t have to load any agents onto systems. You simply turn on the service at a centralized location, tell it the IP ranges or subnet to scan and then have it dump the output somewhere you can review. However, this convenience comes with a tradeoff in terms of accuracy and coverage. By coverage I mean the ability for the scanner to fully scan everything that could be potentially be vulnerable on the system it is scanning.

The second type of scan is an authenticated scan. Authenticated scans typically require an agent to be installed on the system or for the scanner to somehow log into the device so it can scan the running services and applications for known vulnerabilities. An authenticated scan is much more accurate because the scanning agents are running on the same OS as the services and so it can eliminate false positives and provide a more comprehensive scan. However, authenticated scans also come with a tradeoff, which is you are getting much higher accuracy and coverage, but that increase in volume doesn’t factor in where that system is in your environment (i.e. internally vs. externally facing). As a result your reporting may not accurately measure the true risk to the business and you could end up having engineering teams spend time fixing vulnerabilities that may not really be a risk.

How Good Is Your Inventory?

No matter which type of scanning you choose, the scanners are only as good as your asset inventory. You can’t report on and fix what you don’t know about and so the ability to identify and scan assets is critically important. This is where an authenticated scan via agents can present a false picture to a security team. It is easy for the team to assume they are scanning the full environment, but that may not be the case if the agents aren’t installed on everything or if the scanner isn’t scanning all of your devices. Vulnerability scanners shouldn’t just scan your operating systems (compute), but should also scan your network, storage, IoT devices and anything else with a network address to present a complete picture of your enterprise.

Agents

Agents are great for increasing your scan accuracy and coverage, but they present their own challenges. First, you need to deploy all those agents onto your systems and make sure they don’t cause performance issues. This can be a time consuming process to tune the compute, storage and memory for your workloads. Second, it is easy to run into agent creep and have systems with dozens of agents on them that each do something different. These agents can conflict with each other for resources and can be difficult to manage for operational support teams.

Scan Remediation Gap

Vulnerability scanners have an inherent problem when identifying new vulnerabilities. It is often the case that a vulnerability scanner can get updated with a new vulnerability, but a fix or patch may not yet exist to remediate the vulnerability. This can present a problem for businesses that are trying to react to a critical issue quickly, but have to wait for a vendor or developer to provide a fix or implement a more complex compensating control instead. A good question for your vulnerability scan vendor is how quickly they can update their scanner and do they provide additional information about whether a fix is available or not.

CVSS Scores Aren’t Very Useful

CVSS scores have their value, but they aren’t particularly useful to prioritize risk. A truly effective vulnerability management program will take into consideration a lot of other data along with the vulnerability score to make a true determination of risk and prioritize remediation efforts. Some things I recommend adding to your vulnerability reporting to help you prioritize are the following:

  • Business criticality of the system (can it go down or is it revenue generating?)
  • How much revenue does this system make?
  • Is the system publicly facing or is it only accessible internally?
  • Is an exploit available?
  • Is a patch available?
  • What is the complexity of the exploit?
  • Are there other compensating controls already in place (like a WAF) or can you put them in place quickly?

Wrapping Up

Vulnerability scanners are an essential tool in any security program, but they can give security teams false confidence or worse, create a lot of noise for engineering teams. Understanding what type of scan you are using, the tradeoffs for that type of scan and linking the results of your scan to business risk can help any security team accurately identify and prioritize vulnerability remediation.

The Different States Of A Security Program

It may be obvious, but every company that has a security program is in a different state of maturity. As a CSO, it is important to recognize and understand what these different states mean in terms of where your energy will be applied. If you are interviewing or hiring into a company, it is critically important to understand what state the security program is in so you can determine if the opportunity is right for you and to ultimately maximize your impact in the role.

The Different States

In general, a security program can be in one of three different states:

  • New / Building
  • Existing / Incremental
  • Shrinking / Decline

New / Building

A security program that is new typically comes along with new companies, startups or possibly new business units that are acquired via acquisition. However, a company may also be establishing a new program if they are found deficient during an audit or if they suffered a security breach. In this state the CSO (or security leader) needs to establish a program from scratch, which will include mapping risks, developing a budget and establishing funding, recommending tools, evangelizing security best practices and hiring a team. There will be a lot of focus on foundational aspects of security like asset inventory, reporting and initial risk baselines for the organization. Your team will also go after initial program certifications like ISO27001, SOC or other compliance activities. You may even need to establish new processes and ways of working.

Here are some good questions to ask to determine if a program is in the new / building state:

  • Who is performing the function of security today?
  • What goals does the organization have in the first year and three years from now?
  • What is the expected annual budget?
  • How many headcount do you expect for the security team in the first year?
  • Where does your company operate and do you expect to have security resources in those geographic regions?
  • What security tooling is in place today (if any)?
  • Does the company have any existing compliance certifications (like SOC, ISO, etc.)?
  • Why is the company focusing on hiring a security leader and building a security program? Did this come about due to a security incident or other security event like a failed audit?
  • What industries does the company do business in? E.g. finance, government, healthcare, etc.

In my experience, establishing a new security program from scratch is a rare opportunity, but if you get the chance it is truly exciting and offers the opportunity for giant leaps forward in terms of security maturity for the company.

Existing / Incremental

The next state of maturity is existing or incremental and most companies will be in this state. In this state a security program has already been established and has the foundations in place in terms of people, processes and technology. Tooling has already been purchased and implemented, an annual budget has been established and a team exists with different functions like security engineering, security operations and security compliance.

An existing security program usually has smaller goals or incremental annual objectives designed to address some specific area of risk that has been outstanding, or to address a new risk area based on business growth. For example, perhaps the organization has an existing Identity and Access Management (IAM) program, but needs to roll out 2-Factor Authentication (2FA) to further secure access. Or, maybe the business is expanding into the financial industry and needs to become PCI-DSS compliant. These are incremental improvements to the security program and will require increases or reallocation of people and budgets.

A CSO or security leader in charge of an existing security program will generally keep things running smoothly, make sure the company doesn’t regress with respect to security maturity and will continually be evaluating the business for new or existing risks that need to be managed.

Here are some questions you can ask if you are interviewing for a new role that will lead an existing security program:

  • What is the annual budget for the security program?
  • What security tools are in place?
  • How is the team structured?
  • What are the security objectives for this year? For three years?
  • What security compliance certifications does the company maintain (e.g. SOC, ISO, etc.)?
  • How many people are in the security team?
  • What functions does the security team perform? (I.e. security engineering, compliance, risk, product security, security architecture, security operations and incident response, etc.)
  • Why are you looking for hire for this role or who am I replacing if I am hired?
  • How do you expect the business to perform over the next year?

Shrinking / Decline

It is an unfortunate reality that not all programs are in the building or existing states. Sometimes security programs shrink or slip into decline. This can be for a number of reasons such as poor leadership or a declining business. A shrinking security program can also be a temporary state that matches normal expansion / contraction of a mature business and the economy. Whatever the reason, leading a declining security program has significant challenges. First, the security leader will need to over communicate the existing risks to the business and make sure budget and headcount reductions match the reduction of risk as the business shrinks. A CSO can run into real trouble if the reductions are arbitrary and leave the business exposed.

Second, you can expect to have to do more with less. As the business contracts your team will still need to perform, but there may not be additional perks such as training, travel, new tooling, etc. You may also need to consider shrinking budgets and reductions in license counts or other tooling.

Another reason for a shrinking / declining security program is during mergers and acquisitions. Depending on how the deal is structured and the capabilities of the acquiring business, your security team may be redundant or parts of your team may no longer be needed.

A shrinking / declining security program isn’t the end of the world, but it does require careful leadership to make sure the risks are managed appropriately and morale doesn’t completely decline and impact the performance of the remaining team.

Not Everyone Is Good In All States

Not everyone will admit it, but the reality is not everyone is good in all states. This shouldn’t be surprising. Startup founders routinely find they can’t scale a company past a certain point and require additional help. Similarly, I have personally experienced that security programs require different leadership depending on the state of the program and the skills of the individual. Some people just can’t scale a program past the building phase and into the incremental phase. Some people don’t know how to handle decline. Leadership skills aside, some people just have a specific preference for what they like to do.

No matter where you are in your professional career or whatever state your security program is in, I hope this post will help you identify and navigate the type of security program you enjoy leading or are looking to lead one day.

Should Compensation Be Tied To Security Performance?

Lately, I’ve been thinking about how to incentivize security performance across an organization that struggles with the discipline for good security. When done correctly, security can actually help accelerate development lifecycles and is strongly tied to increased organization performance. However, for organizations that struggle, I wonder if they can reward good security behavior with some type of compensation?

CISO compensation is already tied to the security performance of the organization. The success of the security organization to deliver on security goals are already tied to annual KPIs or other performance metrics that tie back to how the CISO is reviewed and ultimately compensated. However, these goals become more risky and less achievable when the CISO is held accountable for security goals across the entire org. The reason for this is the CISO typically doesn’t own the products, systems, applications, etc. that run the underlying business. Instead, the CISO needs to manage the risk for these things and it may often be the case that the CISO or the business will need to make tradeoffs that could be sub-optimal. This could result in the CISO failing to achieve security goals across the org if the rest of the org isn’t held equally accountable.

In an ideal scenario, the rest of the C-Suite will also carry some sort of annual security goal(s) as part of their KPIs. This will effectively tie the performance and compensation of these leaders (CEO, CTO, CFO, CIO, etc.) to how well they deliver on the security goals that are set in agreement with the CISO. If the organization uses cascading goals or KPIs this means the entire org will have some part of their performance compensation tied to how well they execute their security objectives. I can guarantee an engineering team will never skip a security patch again if they are told they won’t get their annual bonus because they missed their annual security goal by shipping a product with a critical security vulnerability.

I also think organizations can gamify and incentivize compensation for security performance even further than just annual performance and compensation. Establishing an internal bug bounty program that rewards employees who find legitimate security issues or rewards teams who never deploy with a critical vulnerability can really accelerate a security program. The challenge for this is it costs money and needs to be balanced with normal business operations. However, I argue paying the people in your org to accelerate security performance will ultimately cost less than the cost of a security breach.

I personally would like to see an organization take security serious enough where they hold the other C-Suite executives accountable for security by tying their compensation to the security performance of their orgs. By bringing this issue to the forefront people will immediately see the real effects of security performance in their paychecks and they won’t be able to ignore the conversation any longer.

Here are the things I think should be part of an organization wide security performance program:

  • Meeting established security Service Level Objectives (SLOs) for patching
  • Meeting incident recovery or remediation SLOs
  • Deploying any type of infrastructure (OS, network, storage, etc.) without critical or high vulnerabilities
  • Deploying or shipping products and applications without critical or high vulnerabilities
  • Meeting SLOs for resolution of critical security findings from security researchers or external bug bounty programs
  • Resolving security risks discovered and documented during mergers and acquisitions within a set time frame (e.g. 1 year or less)
  • Requiring other C-Suite executives to carry a security performance goal for their organization that is tied to their compensation (same with their org)
  • Establishing and compensating employees via an internal bug bounty / security issue disclosure program
  • Closing security exceptions on time or before the due date
  • Achieving all security audit requirements (e.g. FedRAMP, SOC, ISO, etc.)
  • Having the entire organization go a set time frame without a phishing incident or BEC

This isn’t an exhaustive list, but I think you get the idea. Organizations should start structuring performance and compensation goals to help the security org and ultimately hold the rest of the business accountable for the security performance of the things they own. This can help remove the adversarial relationship that often develops between security and other groups and push security into the forefront of the decision making process for the rest of the business.