Tag: #cso

Compliance Corner Series: Compliance Landscape 2023

This blog post is part of the Compliance Corner Series developed in partnership with Milan Patel. This series will include a variety of discussion topics around the intersection of security and compliance. The series will include blog posts, live web streams (with Q&A) and video blogs.

“How has globalization, increasing regulatory requirements from governments and industries change how Security and Compliance must operate internally”

  1. Growing Regulatory Requirements – Impact To The Whole Company vs Single Business Unit

Lee – From a security perspective, it is difficult to standardize security controls and baselines across the whole company vs. a single business. Business lines have different requirements, customers and industries, which are unique to their product lines and customer base. Mandating the entire company meet a global regulatory requirement (like PCI-DSS, FedRAMP or HIPAA) can be costly for those lines of businesses that don’t have customers in those industries where these regulations apply. While the security program may be effectively managing risk for their particular line of business, the difference in regulatory requirements can lead to gaps in customer expectations across product lines or lines of business. It can also be costly to maintain compliance programs since you typically have to audit and produce evidence for each product. This puts a burden on development teams and can take time away from improving the security of their products, vs. generating sufficient evidence to pass audits. For smaller lines of businesses this can be difficult to justify the cost to keep up with global regulations, certify audits and maintain levels on par with the rest of the company.

It is possible to centralize compliance activities across the company, but it also requires every part of the company to standardize their security controls, risk management practices and reporting principles. This may work well for small to medium sized companies that have grown organically or who have done a good job of homogenizing their organization. For larger organizations that have grown via acquisition and have dozens of lines of businesses and product lines, centralizing compliance and security activities may be impossible.

Milan – For compliance, the issue is that many of these new requirements have no “corporate” baseline expectations.  They have not been translated (and sometimes agreed to), by individual lines of business.  Without a standard “bar” to execute against, it’s very difficult for individual compliance teams to attest that it’s being met.  In many cases, these items have no tests, or have not even been implemented.  It becomes a real issue at scale, as these are yet another new compliance item that engineering will have to meet, which will continue to compete with their feature development time requirements.

My experience is that it’s challenging for many of the central compliance/legal teams (that try to set these requirements for engineering teams), from both a positional authority, as well as a credibility perspective.  There are few engineers on these central compliance teams that are experienced with translating these regulations to actual engineering practices.  Also, there is a lack of organizational authority to enforce it uniformly across all business groups.  This just increases the challenge as it adds more time to negotiate and drive priority, especially with the speed of these incoming regulations.

2. Security and Compliance Practices – What Has Changed?

Lee – For security, there has been an increasing trend in transparency as well as a reduction in time to fix security vulnerabilities and report security incidents. Often these regulations or new compliance practices have good intent, but will be so costly to implement that they are impractical or are technologically impossible. These problems are further compounded by modern software development practices like DevOps that move incredibly quickly and can amplify deployment of internet scale products across cloud providers. Software supply chain security and reporting for compliance reasons is particularly challenging due to the complexity of modern software libraries that have multiple interlinking dependencies. Third party libraries and open source software is even more difficult because you don’t always know if you can trust the source code of that library. Providing an accurate software bill of materials (SBOM) can be a significant burden on the business to develop and maintain for compliance reporting purposes. Additionally, the priority of the business can sometimes take precedence over other activities like solving tech debt or modernizing technology stacks. Technical debt can not only hold back the business from optimizing and becoming more efficient, but can also make security and compliance activities challenging.

Milan – I’ve been doing this for a long time, on the engineering side with a large multi-billion dollar software product, and more recently on the compliance side across many products.  What I can say is that while engineering has changed… DevSecOps, updating features/software monthly, instead of yearly… or even every 5 years when hardware went “end of life” (yes, I’m totally aging myself here), compliance practices basically have not changed internally.  

Even major companies are still trying to “throw money” at the problem.  The internal compliance teams know that the SLT has to pay for compliance… but I believe the days of the compliance “bat” and “blank checkbook” are gone.  It doesn’t scale.  I will never get more bodies to manually collect and review evidence for large amounts of controls and services (and not regulatory requirements) that we will have to test and account for. Spreadsheets and email won’t work anymore.  Compliance needs to change, and be run more like a business. Even the third party auditors didn’t have incentive to be more efficient… remember, they mostly bill hours… So the more inefficient and longer it takes for a company to do an audit, the more money they make.  We’ve changed that quite a bit at Oracle, but it’s still early in the journey.

3. Looking Ahead: How Must Security and Compliance Practices Change?

Lee – For security, teams need to shift to a continuous compliance model. Instead of periodically assessing the business once a year (or some other interval) teams need to implement security controls and reporting mechanisms that can automate and continuously evaluate how the security controls meet the various regulatory requirements required for the business. By moving to a continuous compliance model it will force the business to modernize their Software Development Life Cycle (SDLC), modernize their technology stacks and resolve technical debt, which will ultimately benefit security programs. A continuous compliance program will allow security teams to tackle security at multiple points across the technology stack. First, it will evaluate, report and control how software is written. Second, it will evaluate, report and control how infrastructure is deployed and third, it will evaluate, report and control how security posture changes over time so teams can react to security drift.

Security and compliance practices must also change by holding the rest of the business accountable for their decisions. It is impossible for a security or compliance team to be successful without support from the rest of the business because the security and compliance teams don’t “own” the infrastructure and products the business uses to make money. Therefore, there needs to be support at the highest levels of executive leadership to hold the lines of business accountable and tie performance and incentives to how these businesses satisfy their security and compliance requirements.

Milan – Well, I think it must happen on two levels.  First level is central oversight and governance.  

Without a strong central compliance function that can translate, and negotiate requirements, enforce and govern equally with the engineering leaders, it won’t scale effectively.  The main challenge now is that the “fox is guarding the hen house” so to speak.  When the engineering team compliance is solely under engineering leaders in the product group, there is the ability (or even just the appearance) that decisions are made without compliance “equality” of decision making.  If a compliance item isn’t resolved, who questions whether that amount of risk is acceptable to the senior product leader?  Even if accepting the risk is “OK”, any external party can question whether the right level of compliance leadership made that decision, and not just an engineering leader who may prioritize compliance lower than say… a feature request.  

This mostly only becomes an issue after the fact.  A compliance issue happens, then the questions of “Who made that decision… why was that compliance issue not escalated… etc?  Escalate to who?  While it’s a great idea that a compliance leader (buried usually multiple levels down in an organization), will be “politically unencumbered” and rise up and strongly disagree with an issue with their senior leader (who by the way signs their checks). My experience is that it can be difficult, as being seen as an antagonist is not great for one’s career in this space.   I’ve noticed that compliance positions that have high turnover, this has been one of the main indicators of how effective the compliance leader is in the organization feels they are.  If they feel they cannot speak up, or they do and are dismissed “for higher priorities”, they feel they are not effective, or will become the scapegoat if there is an issue.  

The second level is that compliance needs to change their success metrics, and run like a business.

Most senior leaders I know, including myself, put everything into three buckets…  You’re making money… saving money… or costing money…  Compliance has always been seen as “overhead” and costing money.  Even today, I’ve had to battle that, until we made some major changes in how we operate, and what metrics we track for success.  So the question is, if I’m always seen as “costing money”, how do I change that, and what do I change it to?  I realized that while compliance does “make money” (new frameworks open new markets), I don’t really get to claim that.  I’m not out there hustling deals… So I realized years ago I could “save money”, but not in the traditional sense.  I had to change my compliance principles and metrics.  First, everyone accepts that compliance is important, but why did they hate it so much?  As an engineer doing this on a large project, the compliance team never understood that my metric was “completing the audit” or “collecting XXX number of evidence pieces”.

My engineering metric of success was TIME.

As a development operations, release manager, and development compliance owner for the large project, I only had so many engineering hours, so I needed to minimize time I spent on each engineering function, including compliance.

So in compliance, I changed the success metric to First Time Resolution (FTR) for audit evidence. Everyone knows you have to collect at least one, but we had so many “kickbacks” for many reasons.. Either the engineering team didn’t provide what was asked for, or the evidence instructions were incomplete or unclear (remember what I said earlier for “billable hours” from the auditors)? I put at least 30 min per kickback.  I used to own 1000 pieces of evidence, sometimes I could resolve them in 5 min, sometimes it took days.  By raising the FTR up to a goal of 95% (makes it through the first time), I can show, on paper… how much money I’m saving the engineering team (in time).  I was able to show this year, saving millions of dollars in audit costs (by going with an efficient auditor, that didn’t bill me for all those hours), and saving Tens of Thousands of engineering hours by eliminating the “unplanned work” of kickbacks.  

By changing the metrics to what the engineers care about, it built trust and transparency, and increased the overall value of compliance, plus their willingness to look for more efficiencies. 

Needless to say there are a lot more details to talk about for the journey to get here, but the engineering leaders saw, with hard metrics, that compliance “saved money” (via time) for them to use on other features, bug fixing, etc.  And more importantly, it built trust with them that the compliance team was a good steward of their currency, which is time.

4. Where do we go from here?

Lee – Milan, it’s been great discussing how we evolved to this state, and some great ideas and positions on what has to change in our respective security and compliance areas.  There is so much to discuss here I’m looking forward to continuing this series to dive into topics around the intersection of compliance and security.
Milan – Yes Lee, it’s been great discussing these topics.  For me, I feel like this is the beginning of another Industrial Revolution for Compliance, we have seen the change in the engineering and globalization, and we’re on the verge of revolutionizing the Security and Compliance space.  We didn’t even really get into the overall governance and continuous security/compliance versus point in time.  Maybe that’s the next discussion!

Software Supply Chain Security Considerations

Over the past five years there has been increased scrutiny on the security of supply chains and in particular software supply chains. The Solar Winds attack in 2020 brought this issue to the foreground as yet another requirement for a well rounded security program and also has been codified into several security guidelines such as, the Biden Administration Executive Order in 2021 and CISA software supply chain best practices. As businesses shift their software development practices to DevOps and Cloud, CSOs need to make sure software security is one of the components that is measured, monitored and controlled as part of a well rounded security program.

How Did We Get Here?

The use of open source software has increased over the past two decades largely because it is cheaper and faster to use an existing piece of software rather than spend time developing it yourself. This allows software development teams quicker time to market because they don’t have to re-invent software to perform certain functions and can instead focus on developing intellectual property that creates distinct value for their company.

There is also allegedly an implicit advantage to using open source software. The idea being open sourced software has more eyes on it and therefore is less prone to having malicious functions built into it and less prone to security vulnerabilities within the software. This concept may work well for large open source software projects that have a large number of contributors, but the concept falls short for smaller projects that have fewer contributors and resources. However, until the Solar Winds hack in 2020 the general attitude that open source software is more secure was generally applied to all projects regardless of size and funding. As we have learned, this flawed reasoning does not hold up and has allowed for attackers to target companies through their software supply chain.

The challenge with open source software is it is supposed to be a community led project. This means people use the software, but are also supposed to contribute back to that project. However, as companies have embraced open source software the two way street is more biased towards taking and using the software than contributing back. If corporations contributed back in ways that were proportionate to their use of open source software, the maturity, security and quality of the open source software would be drastically improved.

What Are the Risks?

There are several inherent risks involved in using open source software and they are as follows:

Can You Really Trust The Source?

How do you know the software you are pulling down from the internet doesn’t have a backdoor built into it? How do you know it is free from vulnerabilities? Is this piece of software developed and supported by an experienced group of contributors or is it maintained by one person in their basement during their spare time?

The point is, it is fairly easy to masquerade as a legitimate and well supported open source software project. Yet, it is difficult to actually validate the true source of the software.

What Is The Cadence For Vulnerability Fixes And Software Updates?

The size and scope of the open source software project dictates how well it is supported. As we saw during Log4j some projects were able to push updates very quickly, but other smaller projects took time to resolve the full scope of the issue. Any company using open source software should consider how often a project is updated and set limits on the use of software that doesn’t have regular and timely updates.

May Actually Require An Increase In Resources

There are ways for companies to manage the risk of using open source software. Assuming you can trust the source you can always pull down the source code and compile the software yourself. You can even fork the build to include fixes or other improvements to suit your specific application. However, this takes resources. It may not take the full number of resources that would be be required if you wrote the software from scratch, but maintaining the builds, version control and the general Software Development Life Cycle will need to be properly resourced and supported.

Can Complicate External Stakeholder Management

Another issue with using open source software in your supply chain is external stakeholder management. The Biden EO in 2021 requires companies to provide a software bill of materials (SBOM) for software sold to the U.S. Government. This trend has also trickled down into 3rd party partner management between companies, where contractual terms are starting to ask for software bill of materials, vulnerability disclosure timelines and other security practices related to software.

One major issue with this is: it is possible for software to be listed as vulnerable even though there may be no way to exploit it. For example, a piece of software developed by a company may include an open source library that is vulnerable, but there is no way to actually exploit that vulnerability. This can cause an issue with external stakeholders, regulators, auditors, etc. when they see a vulnerability listed. These external stakeholders may request the vulnerability be resolved, which could draw resources away from other priorities that are higher risk.

Standardization Is Hard

Finally, standardizing and controlling the use of open source software as part of the SDLC is advantageous from a security perspective, but exceptionally difficult from a practicality perspective. Unwinding the use of high risk or unapproved open source software can take a long time depending on how critical the software is to the application. If developers have to recreate and maintain the software internally that takes development time away from new features or product updates. Similarly, getting teams to adopt standard practices such as only allowing software to be a certain number of revisions out of date, only allowing software from certain sources and preventing vulnerable software from being used all takes time, but will pay dividends in the long run. Especially, with external stakeholder management or creation of SBOMs.

Wrapping Up

Using open source software has distinct advantages such as efficiency and time to market, but carries real risks that can be used as an attack vector into a company or their customers. CSOs need to include software supply chain security as one of the pillars of their security programs to identify, monitor and manage the risk of using open source software. Most importantly, a well robust software security supply chain program should consider the most effective ways to manage risk, while balancing external stakeholder expectations and without inadvertently causing an increase in resources.

Can Risk Truly Be Measured?

As a CSO, everything you do needs to be evaluated in terms of risk to the business. When you build a security program, prioritize objectives or respond to incidents all of your decisions will take into consideration risk and how to effectively manage it. Which begs the question: Can Risk Truly Be Measured?

The Problem With Risk

The problem with risk is that it is subjective. Everyone views risk, and the behaviors that contribute to risk, differently. Trying to measure a subjective concept can lead to inconsistencies, fluctuations or even conscious and subconscious influencing of the outcome. Even worse, as you engage in behavior that involves risk, your risk tolerance will adapt over time to accept the risk as normal. For the security space, this is the equivalent of failing to implement a specific security control and yet avoiding a breach or incident. As a result the business may try to rationalize that you don’t need to implement the control because you haven’t suffered the consequences of a breach. How do you consistently and objectively measure risk for yourself, your team or your business if it is subjective?

Objectivity Leads to Consistency

Risk can be measured, but it requires you to apply an objective and consistent process to gain consensus and get consistent results. There are several methods for obtaining consistent results when dealing with a subjective concept. In Operations Research there is a process known as the Analytical Hierarchy Process (AHP), which attempts to achieve a mathematical decision for a subjective concept (like deciding which house to buy). Similarly, when attempting to decide between different subjective concepts you can use simple mechanisms like pairwise comparisons or a triangle distribution to express the decision mathematically.

Here are two examples:

Pairwise comparison: This comparison is useful for comparing two things to determine which one is more important. Example: Patching our publicly facing systems with critical vulnerabilities is ten times more important than patching our systems with critical vulnerabilities that are only internally accessible.

Triangle Distribution: This distribution is useful when attempting to distill a decision down to a value when you may only know two data points, such as the extremes. Or, if two people offer different opinions for the value of something you can use the triangle distribution to find a compromise in the middle. Example: Manager 1 – The impact of this event is a 2 out of 5 (5 being highest). Manager 2 – The impact of this event is a 4 out of 5. Using a triangle distribution we decide the impact of this event is a 3.

Either of these methods can be used to determine the likelihood and impact for a specific security control in an industry standard security framework like NIST or CIS.

There are other objective methods for assessing and assigning risk. I’ve discussed the CIS Risk Assessment Method (RAM) in detail in my blog post about Building A Strategic Plan. The advantage of this method is you can apply a consistent process to assign and measure risk for every single CIS control and sub-control. This allows you to measure how you well you are managing risk over the lifetime of your security program in a repeatable way.

When Things Go Wrong

Following an objective process to measure risk requires you to take the time to methodically apply the process to each decision or control you need to asses. This can be time consuming at first, but once the initial baseline is set it is easy to update and maintain. However, it is easy to mess this process up. Here are a few things I don’t recommend:

  1. Don’t make up your own security control framework – There are industry standard security frameworks for a reason. The standard frameworks are comprehensive and have clear criteria for how to apply and asses the effectiveness of a control. The worst “frameworks” I have seen are typically made up by someone in the security team that things they know better. These “frameworks” are subjective, inconsistent, contradictory, redundant or unclear about how to apply specific controls. On top of that these “frameworks” require someone to update them, maintain them and communicate them to the rest of the org. Why duplicate work when you don’t have to?
  2. Failing to apply a consistent process – Failing to apply a consistent process for measuring and assessing risk means you will fail to get a consistent and objective result. This means you will have an inaccurate measurement of risk or the risk will fluctuate with each measurement, which will render it useless. Also in this category is allowing different teams within your company to use different processes for assessing risk. This means you will not be able to consistently compare the risk of different security controls across the business, which can lead to gaps or overconfidence.
  3. Failing to validate the result – This may be obvious, but each of your measurements should be backed up by evidence. Don’t assume your inventory is correct. Don’t assume your IDS is working and don’t assuming your DDoS protection will work properly. Test these things, audit them and spot check them periodically. Risk doesn’t always have to move in one direction. Controls can regress, postures can change and threats are constantly evolving. Validate your results on a regular basis to make sure you are getting the most accurate picture possible.

So Can You Really Measure Risk?

I believe you can measure risk and develop an accurate risk profile for your company. Despite the subjective nature of risk, you can reach an objective result by using simply mathematical principles and by using an consistent objective process. When applying these methods to an industry standard control framework, you can obtain an objective result that will accurately measure your risk for a single decision, or an entire control framework. Once you have an accurate risk profile you can use the risk to make prioritized decisions to manage risk for your business.

Security Vendor Questionnaires: Too Much or Not Enough?

Over the past few years there has been an increasing trend for customers and partners to request security teams to fill out lengthy security questionnaires seeking specific details about the state of their security program. These requests often come as part of routine audits, regulatory requirements or contract negotiations. As someone who has both sent out questionnaires and been a recipient of questionnaires I’m wondering if the industry has gone too far down this trend or if it hasn’t gone far enough? Let me explain…

As a CSO, I want to discover and manage as much risk as possible. This includes conducting business with partners, customers and other companies. I want to understand my supply chain and limit my exposure to any of their security weaknesses that could be used to attack my company. However, I also want to limit the amount of information I disclose about my security program because once I disclose it, I no longer control that information and it could eventually make its way to an adversary if the company I disclosed it to has a breach.

How do we balance these differing requirements and are security questionnaires really the best mechanism for understanding your supply chain?

How Did We Get Here?

Let’s take a step back and consider how we collectively arrived at the need for security questionnaires. There have been several high profile breaches that have set us down this path. The first was the Target breach in 2013 where Target had their point of sale systems compromised as a result of a third party HVAC vendor. The magnitude of this breach along with the realization that Target was compromised via a third party placed a spotlight on supply chain security for the entire industry.

The second high profile breach was the Solar Winds attack in 2020. This attack infiltrated the software supply chain of Solar Winds and placed a backdoor in the product. Given that Solar Winds was used by a huge number of companies this effectively compromised the software supply chain of those companies as well. This attack increased the scrutiny on the supply chain with additional emphasis on software supply chain and even leading to some sectors (like the government) requiring disclosure of a Software Bill of Materials (SBOM).

Increased Regulatory Pressure

These notable attacks (among others) have lead to an increase in regulations that force companies to disclose details around security breaches, but also to invest appropriately in security programs. Despite these investments and disclosures, companies can still face steep fines and costly lawsuits for security breaches. New regulations such as the SEC Cyber Risk Management rules and recent White House Executive Orders on Improving Cybersecurity and establishing a National Cybersecurity Strategy have elevated awareness and focus on supply chain security to the national stage.

Cyber Insurance Isn’t Helping

As Cybersecurity insurance premiums become more and more expensive, companies will continue to look for ways to decrease the cost, while still maintaining coverage. One of the most effective ways to do this is to establish and document a mature security program that you review in detail with your insurer to explain your risks and how you are mitigating them with appropriate controls. Questionnaires are one part of a security program that can demonstrate how you are evaluating and managing supply chain risk and hopefully drive down your premiums (for now). The problem is this creates an incentive where it is everyone for themselves in an attempt to lower their own premiums.

Transparency Is Lacking

The biggest issue security questionnaires are attempting to address is lack of transparency into the details of the security programs. In general, large publicly traded companies (particular cloud companies) and security product companies tend to be more transparent about security because it is built into their brand as a selling point to attract customers. However, details about technologies, program structure, response times, etc. generally lack specificity (for good reason) and the security questionnaire is an attempt to uncover those details to understand what risks exist when entering into a relationship with another company.

You might argue that companies should simply be more transparent with details about their security program, but this is not the solution. Companies should cover high level details with some specificity to demonstrate they have a security program and how it is structured. However, giving specific details about processes, response times, technologies, etc. will reveal details that can be used by an adversary for an attack. Additionally, do we really know what is happening with all this data from security questionnaires? It may be protected under Non-Disclosure Agreements (NDAs) and confidentiality agreements, but that doesn’t prevent the data from being leaked via an unprotected S3 bucket. It is extremely difficult to change a security program quickly and so it may be in the best interest of the responding company to refuse to answer the questionnaire and instead have an undocumented conversation (depending on your level of paranoia).

Yet More Audit and Regulatory Pressure

On top of all these issues, there are still very real requirements to respond to audits, questions from regulators or to provide these responses to your customers and partners that operate in heavily regulated industries (finance, healthcare, government, etc.). Responding to the questionnaires still takes time and places a burden on your security team and still comes with the risk that the information could be involuntarily disclosed to an adversary.

What’s Really Going On Here?

Responding to regulatory and audit requirements aren’t new requirements for our industry and so answering security questionnaires has been the norm for quite some time. However, I think the use of the security questionnaire has been hijacked by the industry as a catch-all way to accomplish a few things with respect to security:

  1. Assert your security requirements over another company (may work for small companies if they can fund it, but generally doesn’t work for large companies with mature programs).
  2. Minimize risk of doing business with and potentially pass on liability to the recipient company. I.e. “We asked them if they did this thing, but we got breached because of them so they clearly didn’t do that thing so it is their fault.”
  3. Create negotiating points as part of contract negotiations for concessions.

The problem with these is they are attempting to impose a solution or liability on a program they don’t control. As a CSO I can’t agree to these things because being contractually obligated to a specific security solution or SLAs removes my decision making power for how to best manage that risk within my security program. Security programs change and are constantly adapting to stay ahead of threats and risks to the business. Being boxed into a solution contractually can actually create a risk, where there wouldn’t normally be one.

Fatigue Is Real

I genuinely struggle with this problem because security questionnaires have their uses, but are causing real fatigue across the industry. The questions fall into one of two categories: either they are largely the same across customers or they are completely bonkers and don’t justify a response. As both a sender and recipient of questionnaires I can definitely understand both sides of the issue. I want as much information about my supply chain customers, but want to minimize the specifics that I share outside of my control. I want lower cyber insurance premiums and I want to pass all my audits and regulatory inquiries. However, I think the industry has deviated from the original intent of the security questionnaire due to the real fear of being held liable for a failure in their security program, which includes their supply chain.

The Problem With Vulnerability Scanners

Vulnerability scanners are table stakes for any security program and they help security teams proactively identify and report on the security posture of assets, but unless you tune them properly they can lead to more problems than they fix. Here are a few things you need to take into consideration when selecting and using a vulnerability scanner as part of your security tooling.

Scanning Technique

There are two primary scanning techniques used by vulnerability scanners. The first is an unauthenticated scan, which is essentially an external scan that attempts to enumerate the ports and services running on the system or device and then match those up to known vulnerabilities. The advantage of an unauthenticated scan is it is usually easier to implement because you don’t have to load any agents onto systems. You simply turn on the service at a centralized location, tell it the IP ranges or subnet to scan and then have it dump the output somewhere you can review. However, this convenience comes with a tradeoff in terms of accuracy and coverage. By coverage I mean the ability for the scanner to fully scan everything that could be potentially be vulnerable on the system it is scanning.

The second type of scan is an authenticated scan. Authenticated scans typically require an agent to be installed on the system or for the scanner to somehow log into the device so it can scan the running services and applications for known vulnerabilities. An authenticated scan is much more accurate because the scanning agents are running on the same OS as the services and so it can eliminate false positives and provide a more comprehensive scan. However, authenticated scans also come with a tradeoff, which is you are getting much higher accuracy and coverage, but that increase in volume doesn’t factor in where that system is in your environment (i.e. internally vs. externally facing). As a result your reporting may not accurately measure the true risk to the business and you could end up having engineering teams spend time fixing vulnerabilities that may not really be a risk.

How Good Is Your Inventory?

No matter which type of scanning you choose, the scanners are only as good as your asset inventory. You can’t report on and fix what you don’t know about and so the ability to identify and scan assets is critically important. This is where an authenticated scan via agents can present a false picture to a security team. It is easy for the team to assume they are scanning the full environment, but that may not be the case if the agents aren’t installed on everything or if the scanner isn’t scanning all of your devices. Vulnerability scanners shouldn’t just scan your operating systems (compute), but should also scan your network, storage, IoT devices and anything else with a network address to present a complete picture of your enterprise.

Agents

Agents are great for increasing your scan accuracy and coverage, but they present their own challenges. First, you need to deploy all those agents onto your systems and make sure they don’t cause performance issues. This can be a time consuming process to tune the compute, storage and memory for your workloads. Second, it is easy to run into agent creep and have systems with dozens of agents on them that each do something different. These agents can conflict with each other for resources and can be difficult to manage for operational support teams.

Scan Remediation Gap

Vulnerability scanners have an inherent problem when identifying new vulnerabilities. It is often the case that a vulnerability scanner can get updated with a new vulnerability, but a fix or patch may not yet exist to remediate the vulnerability. This can present a problem for businesses that are trying to react to a critical issue quickly, but have to wait for a vendor or developer to provide a fix or implement a more complex compensating control instead. A good question for your vulnerability scan vendor is how quickly they can update their scanner and do they provide additional information about whether a fix is available or not.

CVSS Scores Aren’t Very Useful

CVSS scores have their value, but they aren’t particularly useful to prioritize risk. A truly effective vulnerability management program will take into consideration a lot of other data along with the vulnerability score to make a true determination of risk and prioritize remediation efforts. Some things I recommend adding to your vulnerability reporting to help you prioritize are the following:

  • Business criticality of the system (can it go down or is it revenue generating?)
  • How much revenue does this system make?
  • Is the system publicly facing or is it only accessible internally?
  • Is an exploit available?
  • Is a patch available?
  • What is the complexity of the exploit?
  • Are there other compensating controls already in place (like a WAF) or can you put them in place quickly?

Wrapping Up

Vulnerability scanners are an essential tool in any security program, but they can give security teams false confidence or worse, create a lot of noise for engineering teams. Understanding what type of scan you are using, the tradeoffs for that type of scan and linking the results of your scan to business risk can help any security team accurately identify and prioritize vulnerability remediation.

The Different States Of A Security Program

It may be obvious, but every company that has a security program is in a different state of maturity. As a CSO, it is important to recognize and understand what these different states mean in terms of where your energy will be applied. If you are interviewing or hiring into a company, it is critically important to understand what state the security program is in so you can determine if the opportunity is right for you and to ultimately maximize your impact in the role.

The Different States

In general, a security program can be in one of three different states:

  • New / Building
  • Existing / Incremental
  • Shrinking / Decline

New / Building

A security program that is new typically comes along with new companies, startups or possibly new business units that are acquired via acquisition. However, a company may also be establishing a new program if they are found deficient during an audit or if they suffered a security breach. In this state the CSO (or security leader) needs to establish a program from scratch, which will include mapping risks, developing a budget and establishing funding, recommending tools, evangelizing security best practices and hiring a team. There will be a lot of focus on foundational aspects of security like asset inventory, reporting and initial risk baselines for the organization. Your team will also go after initial program certifications like ISO27001, SOC or other compliance activities. You may even need to establish new processes and ways of working.

Here are some good questions to ask to determine if a program is in the new / building state:

  • Who is performing the function of security today?
  • What goals does the organization have in the first year and three years from now?
  • What is the expected annual budget?
  • How many headcount do you expect for the security team in the first year?
  • Where does your company operate and do you expect to have security resources in those geographic regions?
  • What security tooling is in place today (if any)?
  • Does the company have any existing compliance certifications (like SOC, ISO, etc.)?
  • Why is the company focusing on hiring a security leader and building a security program? Did this come about due to a security incident or other security event like a failed audit?
  • What industries does the company do business in? E.g. finance, government, healthcare, etc.

In my experience, establishing a new security program from scratch is a rare opportunity, but if you get the chance it is truly exciting and offers the opportunity for giant leaps forward in terms of security maturity for the company.

Existing / Incremental

The next state of maturity is existing or incremental and most companies will be in this state. In this state a security program has already been established and has the foundations in place in terms of people, processes and technology. Tooling has already been purchased and implemented, an annual budget has been established and a team exists with different functions like security engineering, security operations and security compliance.

An existing security program usually has smaller goals or incremental annual objectives designed to address some specific area of risk that has been outstanding, or to address a new risk area based on business growth. For example, perhaps the organization has an existing Identity and Access Management (IAM) program, but needs to roll out 2-Factor Authentication (2FA) to further secure access. Or, maybe the business is expanding into the financial industry and needs to become PCI-DSS compliant. These are incremental improvements to the security program and will require increases or reallocation of people and budgets.

A CSO or security leader in charge of an existing security program will generally keep things running smoothly, make sure the company doesn’t regress with respect to security maturity and will continually be evaluating the business for new or existing risks that need to be managed.

Here are some questions you can ask if you are interviewing for a new role that will lead an existing security program:

  • What is the annual budget for the security program?
  • What security tools are in place?
  • How is the team structured?
  • What are the security objectives for this year? For three years?
  • What security compliance certifications does the company maintain (e.g. SOC, ISO, etc.)?
  • How many people are in the security team?
  • What functions does the security team perform? (I.e. security engineering, compliance, risk, product security, security architecture, security operations and incident response, etc.)
  • Why are you looking for hire for this role or who am I replacing if I am hired?
  • How do you expect the business to perform over the next year?

Shrinking / Decline

It is an unfortunate reality that not all programs are in the building or existing states. Sometimes security programs shrink or slip into decline. This can be for a number of reasons such as poor leadership or a declining business. A shrinking security program can also be a temporary state that matches normal expansion / contraction of a mature business and the economy. Whatever the reason, leading a declining security program has significant challenges. First, the security leader will need to over communicate the existing risks to the business and make sure budget and headcount reductions match the reduction of risk as the business shrinks. A CSO can run into real trouble if the reductions are arbitrary and leave the business exposed.

Second, you can expect to have to do more with less. As the business contracts your team will still need to perform, but there may not be additional perks such as training, travel, new tooling, etc. You may also need to consider shrinking budgets and reductions in license counts or other tooling.

Another reason for a shrinking / declining security program is during mergers and acquisitions. Depending on how the deal is structured and the capabilities of the acquiring business, your security team may be redundant or parts of your team may no longer be needed.

A shrinking / declining security program isn’t the end of the world, but it does require careful leadership to make sure the risks are managed appropriately and morale doesn’t completely decline and impact the performance of the remaining team.

Not Everyone Is Good In All States

Not everyone will admit it, but the reality is not everyone is good in all states. This shouldn’t be surprising. Startup founders routinely find they can’t scale a company past a certain point and require additional help. Similarly, I have personally experienced that security programs require different leadership depending on the state of the program and the skills of the individual. Some people just can’t scale a program past the building phase and into the incremental phase. Some people don’t know how to handle decline. Leadership skills aside, some people just have a specific preference for what they like to do.

No matter where you are in your professional career or whatever state your security program is in, I hope this post will help you identify and navigate the type of security program you enjoy leading or are looking to lead one day.

Should Compensation Be Tied To Security Performance?

Lately, I’ve been thinking about how to incentivize security performance across an organization that struggles with the discipline for good security. When done correctly, security can actually help accelerate development lifecycles and is strongly tied to increased organization performance. However, for organizations that struggle, I wonder if they can reward good security behavior with some type of compensation?

CISO compensation is already tied to the security performance of the organization. The success of the security organization to deliver on security goals are already tied to annual KPIs or other performance metrics that tie back to how the CISO is reviewed and ultimately compensated. However, these goals become more risky and less achievable when the CISO is held accountable for security goals across the entire org. The reason for this is the CISO typically doesn’t own the products, systems, applications, etc. that run the underlying business. Instead, the CISO needs to manage the risk for these things and it may often be the case that the CISO or the business will need to make tradeoffs that could be sub-optimal. This could result in the CISO failing to achieve security goals across the org if the rest of the org isn’t held equally accountable.

In an ideal scenario, the rest of the C-Suite will also carry some sort of annual security goal(s) as part of their KPIs. This will effectively tie the performance and compensation of these leaders (CEO, CTO, CFO, CIO, etc.) to how well they deliver on the security goals that are set in agreement with the CISO. If the organization uses cascading goals or KPIs this means the entire org will have some part of their performance compensation tied to how well they execute their security objectives. I can guarantee an engineering team will never skip a security patch again if they are told they won’t get their annual bonus because they missed their annual security goal by shipping a product with a critical security vulnerability.

I also think organizations can gamify and incentivize compensation for security performance even further than just annual performance and compensation. Establishing an internal bug bounty program that rewards employees who find legitimate security issues or rewards teams who never deploy with a critical vulnerability can really accelerate a security program. The challenge for this is it costs money and needs to be balanced with normal business operations. However, I argue paying the people in your org to accelerate security performance will ultimately cost less than the cost of a security breach.

I personally would like to see an organization take security serious enough where they hold the other C-Suite executives accountable for security by tying their compensation to the security performance of their orgs. By bringing this issue to the forefront people will immediately see the real effects of security performance in their paychecks and they won’t be able to ignore the conversation any longer.

Here are the things I think should be part of an organization wide security performance program:

  • Meeting established security Service Level Objectives (SLOs) for patching
  • Meeting incident recovery or remediation SLOs
  • Deploying any type of infrastructure (OS, network, storage, etc.) without critical or high vulnerabilities
  • Deploying or shipping products and applications without critical or high vulnerabilities
  • Meeting SLOs for resolution of critical security findings from security researchers or external bug bounty programs
  • Resolving security risks discovered and documented during mergers and acquisitions within a set time frame (e.g. 1 year or less)
  • Requiring other C-Suite executives to carry a security performance goal for their organization that is tied to their compensation (same with their org)
  • Establishing and compensating employees via an internal bug bounty / security issue disclosure program
  • Closing security exceptions on time or before the due date
  • Achieving all security audit requirements (e.g. FedRAMP, SOC, ISO, etc.)
  • Having the entire organization go a set time frame without a phishing incident or BEC

This isn’t an exhaustive list, but I think you get the idea. Organizations should start structuring performance and compensation goals to help the security org and ultimately hold the rest of the business accountable for the security performance of the things they own. This can help remove the adversarial relationship that often develops between security and other groups and push security into the forefront of the decision making process for the rest of the business.

Vendors: If You Want To Reach CISOs Stop All The Noise

I had an interesting conversation with a bunch of CISO friends last week about the current problem with vendor communications and particularly how there is a high volume of noise in the industry right now. The current problem is predicated on the assumption that more volume of communication will lead to sales leads, which will lead to an actual deal. I’ve been both on the sales side and the corporate side and I can tell you that vendors view it as a numbers game. However, I’m here to tell you the numbers game isn’t helping you achieve your goals because you are annoying the very people you are trying to connect with.

Bad Behavior

Most CISOs and security professionals I know don’t start out having a contentious relationship with vendors, but once you achieve a specific level and title, the calls, texts and SPAM inevitably start. If you are a vendor, here are a few things guaranteed to get you blacklisted from the people you are trying to reach:

  • Claiming you were referred by someone else in the company (when you weren’t. Yes we check.)
  • Sending repeated SPAM (it is unsolicited for a reason and we have no obligation to respond so stop it.)
  • Calling our personal cell phones (at any time)
  • Reaching out to our boss and saying we aren’t responding to you (you’re kidding right?)
  • Sending unsolicited texts
  • Sending unsolicited gifts

I know the behavior above is referring to a small minority of vendors, but the problem is the volume of noise from these vendors drowns out any legitimate outreach from anyone else.

What Can CISOs Do To Cut Down On The Noise?

If you have a public profile on any social media platform it is inevitable vendors will find you. My suggestion is to use a custom email address for each platform so you know if the social media platform sells or leaks your information. You can do the same thing for conferences or if you fill in your email for webinars and other vendor activities. If you don’t have the ability to create custom email addresses you can also use throw away or temporary free email services. If the vendor won’t accept these temp email services then move on…trust me that white paper isn’t worth it.

Controlling your email is not the only way to cut down on the noise. If vendors, conferences or other activities require you to input your work address you can use the primary address field or secondary address field to list the name of the vendor. This way if they send you junk mail or gifts you know who it came from.

Aside from controlling your information, I know several CISOs that set specific rules for how to engage with them. These are typically rules like only going through a certain Value Added Reseller (VAR) and only coming to the office and presenting on certain days (like the last Friday of every month). These specific days are good ways for security groups and CISOs to learn about a lot of different technologies at once.

What Can Vendors Do To Reach CISOs?

The biggest thing vendors can do to reach CISOs is respect boundaries. Treat CISOs and their security teams the same way you want to be treated. Most importantly, no one wants to feel like a transaction. The security industry is small and if you truly want results, cultivate lasting relationships that can follow you wherever you go. If you abuse people’s time and attention you will get blacklisted. We all talk and word will get around quickly.

The most effective way for vendors to connect with CISOs is to go to vendor days or partner with VARs and get on their docket for innovation days where the VAR shows you off along with a list of other companies that are of interest to the CISO.

Wrapping Up

Being a CISO is stressful enough without getting harassed constantly by vendors. CISOs need to set rules for how vendors can engage with them so vendors know not to resort to bad behavior to get their attention. On the other hand, vendors need to respect boundaries and cultivate lasting relationships independent of the technology company they represent. If you resort to bad behavior you will be blacklisted and word will get around not to do business with you.

When Risk Management Goes Wrong

Last week I took the opportunity to take some time off and spend a few days with my family at a popular amusement park in California. On the second day my kids and I decided to go to the water park to go down the water slides and during this experience my kids and I were on the receiving end of risk management gone wrong.

Let me explain…

Let’s assume I’m the CSO of this amusement park company and I’m helping the legal team and rest of the C-Suite evaluate the risks involved in this particular activity. So the general question is: “What are the risks involved in going down a water slide?” Here are a few easy examples:

  1. Someone could go down the slide and not be able to swim so they could possibly drown in the splash pool.
  2. Someone could go down the slide and collide with another person in the splash pool injuring them both.

How would you manage these risks as a business to make sure you aren’t continually sued by your guests?

It is easy to take the extreme case and simply try to manage these risks to the point where you have minimized your liability. In example 1, you can simply hire and staff life guards at the splash pool to make sure people don’t drown. You can also minimize the depth of the pool so there is enough water for a safe landing, but not so much that it is over people’s heads. Overall, these risk management techniques would make guests feel safe and not really impact the overall experience of the ride in anyway.

Example 2 is where it gets interesting. How do you make sure people don’t crash into and injure each other? On the extreme case you can make guests wait until the current guest is all the way out of the pool. This would minimize the risk of crash injury and be the safest option, but it comes at a tradeoff. The tradeoff is wait times for the ride, which ultimately impacts guest experience and satisfaction. Waiting for each guest to exit the pool took anywhere from 30 seconds to a few minutes (depending on the guest) before they would let the next guest go down. This doesn’t sound like a lot, but when you add up the time it takes for someone to go down (let’s say 30 seconds) combined with the time for them to get out of the pool (another 30 seconds to several minutes) you are looking anywhere from one minute to several minutes between guests. This means if the line is long your guests are waiting a really long time to ride this ride.

Risk is a difficult concept for companies to navigate because it is subjective. In order to get a reasonable risk outcome you need to use an objective process to make sure you are assessing and managing risk in a consistent way. It is up to the CSO organization to communicate and advise on how to manage risk in a way that is conducive to the business. Most importantly, you can’t let one group dominate the conversation about risk without taking into account other stakeholder perspectives (like legal, sales, finance, HR, IT, etc.).

In this example above the amusement park business minimized their risk at the expense of customer experience. This is equivalent to having extremely long latencies on your e-commerce site as a result of security checks. It may sound like a great idea, but ultimately will impede your business in the long run. In this particular case I am unlikely to go back to that particular amusement park because of the frustratingly long wait times.

As the CSO, it is your job to effectively communicate risk. You could be advocating for a new control to reduce risk, a new process to manage risk or an exception to accept risk. These are all acceptable outcomes as long as the business owners are involved in and acknowledge the ultimate decision. Most importantly, you need to balance customer experience, user experience and the needs of the business when implementing controls and processes to manage risk. At the end of the day not all risk can (or should) be managed because the business has to function and this comes with inherent risk.

A CISO Primer On Navigating Build vs Buy Decisions

Every year CISOs propose and are allocated annual budgets to accomplish their goals for the upcoming year. Within these budgets are allocations for purchasing tooling or hiring new headcount. As part of this exercise CISOs and their respective security teams are asking: should we build this thing ourselves or should we just buy it? It may be tempting to simply buy a tool or to build it yourselves, but both options have advantages and disadvantages. Here are my thoughts on how CISOs should think about this classic business problem.

Strategic Considerations

The first question I ask myself and my team is – will building this thing ourselves become a strategic capability or differentiator for our business? If we build it can we use it ourselves and sell it to customers? Are we building a capability unique to the industry that could lead to patents or a competitive advantage? Most importantly, do we have the resources to develop, maintain and support this capability for the indefinite future? If the answer to these questions is yes, then you should consider building the capability yourself, but this also comes with a cost in terms of people resources.

Use of People resources

While building a capability can look attractive at first, it generally has long term costs that can easily add up to be more than the cost of just purchasing a tool or capability. This is because CISOs will need to staff engineers or developers to build the thing. This means they will need to hire (or borrow) resources who will need coding skills, database skills, AI/Big Data Skills and a bunch of other skills that aren’t typically key skills in a traditional security team.

Let’s say you will need to hire or borrow people to build your new thing. These people have salaries, benefits, bonuses, equipment costs, facilities costs and other expenses that can easily cost as much as (if not more than) the annual cost of purchasing a tool. Additionally, if you hired people, they can’t just move on once the thing is built. They will need to support it, maintain it, etc. If you borrowed resources then you will need to figure out who is going to handle ongoing operations and maintenance of the tool and you need to consider the opportunity cost of using these borrowed resources to build something for you instead of doing something else for the business that could have value.

The point is people aren’t cheap and they tend to be the most valuable resources for a business. Using these resources wisely and in a cost effective way is an important consideration for every CISO.

Financial allocation (CAPEX vs OPEX)

One other consideration for Build vs Buy is how your company allocates financial costs towards either CAPEX or OPEX. The reason this is something to consider is it may be easier to get OPEX budget than CAPEX (or vice versa). This can influence your decision to buy something over building it depending on how finance wants you to allocate the cost (or how easy it is to get budget in one of these buckets).

Time to deploy

Another consideration for Build vs Buy is – when do you need the capability and how long will it take to build it vs how long it will take to buy something and deploy it? If you need the capability immediately it may make sense to buy the tool and deploy it rather than trying to hire resources, onboard them, build the thing, support it, etc.

Integration Costs

Similarly, integration costs can be a huge factor towards whether the capability is truly effective or not. For example, if you can stand up the new thing relatively quickly, but it takes six months or a year to integrate it into your existing tools then that could throw your overall timeline off and may sway your decision towards building it yourselves instead.

Security Considerations of SaaS / Cloud Products

Lastly, and most important, CISOs need to think about the security considerations of buying a product vs building it in house. Software supply chain security is a top security risk for businesses and CISOs need to evaluate new tooling to see if they are adhering to the security requirements required by the CISO. If the product is a SaaS or Cloud Product then CISOs need to think about the risk of sending their data, source code or other information to a third party environment they don’t directly control. Similarly, if the CISO chooses to build the capability in house then they will need to make sure the team is making the new capability as secure as possible so the business and their customers aren’t exposed to unnecessary risk.

Wrapping Up

Choosing to build or buy a new capability isn’t an easy decision. Both decisions have explicit and hidden costs that can be difficult to navigate. Like any decision the CISO should weigh the risk of the decision and ultimately choose the option that supports the strategic direction of the business, meets financial and budgeting requirements and is sustainable by the security organization for the life of the capability.