Cultural Change Agent – Part 2 What Does Work?

In Part 1 of this post I discussed how security teams are often challenged to change behavior and ultimately culture. This is an exceptionally difficult problem and I covered a few things that don’t work when trying to effect change. In this post I’ll discuss my observations for what does work when trying to change behavior to a security first mindset.

What Does Work?

Assume Good Intent And Be A Patient Teacher

This one speaks to mindset. When coaching my team I often find they are frustrated with another group unnecessarily. When diving into the problem there is usually a lot of blaming and an us vs. them attitude. There is also an underlying assumption that the team is not doing something because they don’t want to.

Instead, I coach my team to adjust their perspective and lead with empathy. It is exceptionally rare that a team flat out refuses to do something security related. Instead, I find a team wants to do the thing, but just doesn’t know how. Changing your mindset from a negative one (they aren’t doing it) to a positive one (they want to do it, but don’t know how) changes the entire dynamic of the conversation with the other team. Practicing empathy, and ultimately patience, with other teams by coaching them to the outcome you want is far more effective than blaming them for failing to comply.

Make Security Easy (Without Compromising Effectiveness)

I touched on this briefly in Part 1, but I’ll say it again here because it is worth repeating. Humans will take the easy path, follow habit or both. Security teams need to find solutions that make security easy, without compromising effectiveness.

A good example is patching. Make it easy for teams by publishing hardened, up to date images, containers, etc. that they can pull and use for their products / services. Take the burden off of them to have to patch an OS by providing them with a patched version up front. (Yes there is a lot of nuance to this. I’ll discuss this more in a later post).

Enlist Allies

Enlisting allies on other teams can really help accelerate your security goals. Teams within IT, the CTO group and even senior engineers can help get momentum behind key initiatives and act as an extension of the security team.

In my current function we use these allies in the following ways:
  • Early Adopters – New processes and technologies always have issues. Having a team of early adopters outside the security function helps iron out the bugs before trying to get the entire organization to adopt it.
  • Security Champions – In my current function we use our security allies to help champion initiatives or act as an extension of the security team. This is invaluable because it helps explain things in a way that is understood natively by the receiving team and it is delivered by someone within their own group. This can be particularly effective when one team is behind on something (like patching) and needs a little extra encouragement.
  • Extended Vulnerability Response Team – During Log4j we found the security allies to be invaluable in responding to the vulnerability. They were able to take the new patches / fixes and translate that to their products or services. They were also able to come back with suggestions and improvements to processes to speed up remediation. By having key allies represent the engineering groups we were able to keep our War Room relatively small and relied on them to deliver marching orders to their teams so we could tackle the issue at scale.

Set The Example (Lead From The Front)

Sometimes teams need to see what good is before doing it themselves. The CSO / CISO and ultimately the security team should set the example for the type of behavior they want to see from the rest of the organization. This shouldn’t create an us vs. them or sense or superiority, but instead should act as an eat your own dog food type of exercise. If you are asking teams to patch things, then your team should follow the same processes. By participating in the very same processes you are asking the rest of the org to follow it creates accountability for your team to make sure those processes work appropriately.

Trust But Verify

Simple, but effective. Early in my current role I often had teams tell me one thing and I took their word for it. I did this because I simply didn’t have the staff to chase down every little thing and I assumed good intent. As my team matured and we developed better processes for collecting data, we found what people told us didn’t match up to reality. This wasn’t a case of malicious intent, but instead the reality of working in a large organization. Things change, people leave, nothing is static. One of my history professors used to say “the palest ink is better than the fondest memory.” This is true with security data. Talk to people, get their perspective and then go check it out to see if that is true. If it isn’t, then that is an area to focus on.

Like any skill, practice makes perfect. I often find when I am getting frustrated it is because I am slipping back into bad habits like failing to assume good intent or leading with empathy. A simple mindset shift usually snaps me out of it and I spend a large amount of my time reinforcing these principles across my team. I hope you find these skills useful and are able to put them to use in your organization to achieve cultural change.

Cultural Change Agent – Part 1 What Doesn't Work?

When you are hired for a security leadership at a company you are being hired because the company has a need for your experience. The company may need help with reducing risk to a new business area, successfully passing a new compliance audit, implementing a new security capability or simply growing the team to match the growth of the company. All of these are possibilities, but I often find one of the most important underlying reasons a company hires a new security leader is to effect behavioral and cultural change. Changing behavior, and ultimately the culture, of a company is an exceptionally difficult challenge for a security leader and the following are some examples of what doesn’t work when trying to effect change. In Part 2 I’ll cover what does work.

What Doesn’t Work?

First, let’s talk about what doesn’t work. In some cases I have learned these lessons the hard way. In other cases, I’ve learned these lessons through observation or listening to my peers.

Don’t Be The Mafia (Or What I Like To Call “The NO Factory”)

First up, avoid creating an adversarial relationship between your team and the rest of the company. If the security team constantly tells the rest of the company NO, then eventually the rest of the company will start ignoring your team and this will become a problem in terms of risk management, security enforcement, support during incidents, etc.

I find a much better approach is to figure out how to enable the business with security. What does this mean? It means working with the non-security teams to put processes, people or technology in the right places so they are minimally invasive, while achieving the security objectives.

Example – Architectural Review Process:

Several years ago my team was responsible for reviewing new architectures for security issues before they went into production. This was a very lightweight process with two phases – a conceptual review (does this make sense) and then a more detailed post production review (did you do what you said you were going to do). In order to achieve the results of managing or reducing risk appropriately my team ended up acting in a consulting role to work alongside the other teams to help improve their security as part of the Architectural Review Process. We even eventually automated several of the compliance checks and scans to shorten the feedback loop and allow teams to self service as they were building their new thing. The end result was the security team built trust with the rest of the business and we ultimately reduced risk. If, instead, my team told the non-security teams NO every time they submitted something that wasn’t up to security standards, they probably would have just deployed to production anyway without meeting our security requirements and this would have put the company at significant risk.

Don’t Confuse Accountability With Ownership

If you have ever read Extreme Ownership by Jocko Willink then you are familiar with the concept that good leaders should take ownership for everything in their world. In the case of security this would mean the security function should be responsible for patching, hardening, scanning, etc. of everything in the company. Yet, I argue this is not only impossible, but the wrong approach.

Why is this impossible?

The team building the product or service needs to be responsible for the security of that service. If you aren’t the team building the product or service then you may not have the right permissions, you may not understand the code and you may not even know it exists.

Why is it the wrong approach?

Asking another team, other than the one who produced the product, to be responsible for patching, hardening, scanning, etc. of a product or service, passes the responsibility and accountability for security away from the team that actually owns the product or service, which means the original team will have zero incentive to develop a product or service that meets the appropriate security requirements.

As a CSO/CISO you are accountable for the security posture of the company, yet the security function doesn’t actually own the products or services that the company sells to customers. The teams who build the products or services are responsible for patching them, scanning them and fixing the security issues that are discovered in them. The non-security leaders over these teams need to hold their teams accountable for meeting the appropriate security requirements and the CSO/CISO needs to hold these leaders accountable (with things like dashboards, scan results, etc.).

Don’t Underestimate The Human Factor

When beginning a new security leadership position it is usually easy to identify the technological solution to a problem. Need more visibility? Improve logging and alerting. Need to know who owns something? Improve asset inventory. Need to shorten the vulnerability window? Improve patching. However, it is often extremely difficult to diagnose and understand the behavior of the humans who are behind the security problem you are dealing with.

Technology is easy, humans are complex

I often say this to my team to remind them that people are the most valuable asset for a company, but are also the most challenging. It also reminds them to not jump to a technological solution. Spend time talking to people to understand how and why they the things they do so you and your team can develop ways for them to achieve your security goals as quickly and easily as possible.

Changing patterns or behavior within a company or group takes time and is difficult. Humans will do what is easy, what is habit or both, often at the expense of security. Make small changes to processes or technology to gradually nudge teams into compliance.

I hope this helps you avoid these pitfalls so you and your team can be successful in achieving your security objectives via cultural change. Up next – Cultural Change Agent – Part 2 What Does Work?.

Building A Strategic Plan

In my first blog post I talked about how to create a security organization. Once you have an idea of how you want to structure your security org the next step is to build a strategic plan. This plan will cover what you need to do, what order you need to do it and why you need to do it. It will also inform how to employ your resources or what type of resources to request if you need to hire. You may start working on this plan from day one, but most likely the plan really won’t become clear to you for several months or longer. The way one of my mentors described it is – the higher up in the org you are the more impactful your changes are, so you want to take longer to develop and execute the plan to minimize negative effects of the changes you will make. The only way to minimize the effects is to talk to everyone possible and get buy in from key stakeholders to the security org.

The rest of this post will talk about the steps I used to build the plan that I still use today. The steps are adapted from the CIS Critical Security Controls and from the CIS RAM. We adapted their process so it worked better for us.

Step 1: Choose A Framework

There are a lot of frameworks out there, but by far the most popular are NIST Cybersecurity Framework (800.53) and the CIS Critical Security Controls v8 (Top 18). Both of these frameworks have mappings to each other so you can go back and forth if needed.

Why use an industry standard framework?

  1. Standard terminology when you talk to other companies, auditors, internal groups, etc.
  2. Clear completion requirements (avoid scope creep and never ending projects)
  3. Prioritized list
  4. Objective, repeatable and scalable process for measuring security maturity and risk

I personally like the CIS controls because it is simpler to follow and has accompanying plans like the CIS Risk Assessment Method. However, both CIS and NIST are great and you really can’t go wrong with either one.

Step 2: Review and Prioritize The Controls

Review & Adjust

This is a really important step because not all of the controls will apply to your business or some of the controls may need to modification to make them relevant. Spend the time one this part up front because it will help you later when you try to close things out.

Prioritize

Once all the controls are clear, go through and prioritize the order you want to do them. In the case of CIS there are three types of implementation groups – basic, foundational and organizational. You can start with the implementation group that relates to your organization size, or you can treat the implementation groups as maturity levels (which is what we did). As we matured we moved from the basic to the organizational implementation groups and this was helpful in the earlier days to help us focus and prioritize. Each implementation group has different sub-controls that specify what you will need to do and they increase in scope.

Step 3: Baseline Risk

The next step is to gather a risk baseline against all of the controls. This baseline indicates how much risk your business has if they did nothing against the controls. This is really important because risk is subjective and no business will have the same risk baseline, risk tolerance, etc. You can follow the CIS RAM for this step or follow what I did (which was based on the CIS RAM). In my case I used a scale of 1-5 to baseline risk against the following:

Impact – What is the impact on our ability to respond or conduct business by not having this control implemented in the event of an incident?

  1. An imperceivable impact on our operations, services or revenue
  2. A noticeable, but low impact on our operations services or revenue
  3. A noticeable, but medium impact on our operations, services or revenue
  4. noticeable, but high impact on our operations, services or revenue
  5. A catastrophic or unrecoverable impact on our operations, services or revenue

Likelihood – What is the likelihood of a security incident if we don’t have this control implemented?

  1. Not foreseeable – We don’t think it can happen
  2. Foreseeable, but timing uncertain – We think it can happen, but aren’t sure when
  3. Expected, but uncommon – We know it can happen, but it doesn’t happen often
  4. Common – We know it can happen and it happens often
  5. Imminent – We know it can happen and it is happening now (or all the time)
Example: Assess each control for the maximum risk to the business as if you did nothing

CIS control 8.2 – Establish Audit Logging

Likelihood – Noticeable High (4)

Impact – Common (4)

Next we multiply the two scores together to get the maximum risk for the control, which in this case is 16 (4×4).

Next evaluate the risk against a scale. We created the following risk scale to evaluate risk as low, medium or high to help us prioritize. The risk scale has a minimum of 1 and a maximum of 25 (you get these from multiplying the lowest and highest of likelihood and impact together). So the risk scale looks like the following:

Low < =5

Medium >5 & <=12.5

High >12.5

This scale is subjective and I recommend adjusting it based on the risk tolerance of your business. In our case, we created this scale to orient around the middle (3×3) in attempt to equitably distribute risk between low, medium and high to try to avoid bias towards one extreme.

Step 4: Create A Progress (or Maturity) Baseline

Baselining your plan and risk is an important step. It tells you where you are doing well, where you need to focus, how to prioritize, how to invest and how to assign resources. Most importantly, it documents the progress you are making to reduce or manage risk and it is a useful way to align stakeholders.

To baseline progress we used the following scale:

Not Started – Gray

<=50% – Red

>50% & <=80% – Yellow

>80% – Green

Verified Complete – Blue

The nice thing about this model is you can assess progress for multiple groups at whatever level you want (project, product, group, division, etc.) or you can just do one for your entire org. In our case, we broke it out by division and then averaged it to provide a single view for the whole org.

Step 5: Link Progress to Risk

Now that you have a risk baseline – (maximum risk for not doing anything for each control) and you have a progress baseline (where you are starting from) you can now link the two together. This is important because as you make progress on the different controls it will reduce risk for your business. In our case we used the following scale:

Not Started – Gray    No Change From Baseline (Multiply Risk by 1)

<=50% Complete – Red    Multiply Risk by .75

>50% & <=80% – Yellow    Multiply Risk by .5

>80% – Green    Multiply Risk by .25

Verified Complete – Blue    Multiply Risk by .1

Example:

CIS control 8.2 – Establish Audit Logging

Baseline Risk – 16

Baseline Progress – >50% & <=80% Yellow

New Risk Score – .5 * 16 = 8

This means risk has been adjusted from High to Medium because we have made progress on logging

Step 6: Repeat This Process For All Controls To Build A Full Dashboard

Ok let’s bring this all together. At the time of writing this blog the CIS site to download v8 isn’t working so I’m going to use a copy of v7.1 that I have.

To simplify, I’m just going to use one control from each control group so for v7.1 there will be 20. This example shows a fictional organization that started by having zero controls implemented at the beginning of the year and then were able to make progress against all of the controls to have their progress complete and verified by the end of the year (I wish!). I’ve added coloring to create a classic stoplight chart and a graph showing the risk reduction by quarter.

Final Thoughts

Whether you use the CIS Controls and RAM or you use my example, this is a great way to create a strategic plan and begin focusing your resources on the highest risk areas. My team still uses our original plan as reference and we update it periodically as we make progress against various controls. In fact, we just updated it for a board meeting and I was able to show our progress against risk over the past three years (over 70% reduction).

We even used this plan to gamify how our risk reduction would change based on what controls we chose to implement next. This allowed us to plan ahead and forecast what risk reduction we were expecting each quarter or after a major control was implemented.

One important thing to note is that there will always be residual risk within the organization. You can never make risk go to zero. If you happen to have an area that is still high risk (even though you implemented a sufficient control) then that is an area you may want to focus additional people, processes, etc. to try to effectively manage that risk in parallel with technology.

Finally, this plan has limitations. While it is an effective way to measure static risk, it isn’t measuring dynamic risk.

What’s the difference?

Static risk – is measuring how well a control was implemented.

Dynamic risk  – is measuring if that control is effective against the changing threat landscape.

I’ll talk about this more in a later blog post.

Reference Files: Use the template below to follow the process above. Simply click the drop down for the appropriate cell under Risk or Maturity and everything will get calculated for you including the graph.

Link to CIS 7.1 Strategic Plan Template

Welcome to 370Security!

Welcome to the 370Security blog where I plan to share my experience of being a Chief Security Officer at one of the largest cloud companies in the world. Through this blog I plan to cover a variety of topics such as leadership, organization development, industry trends, tools, risk and more. If you aspire to be a CSO / CISO or you ever wanted to know what the job is like, then bookmark this site. What I can promise you is these posts will draw from real world experiences and this site will catalog those experience so others can learn from them.

Why 370Security?

I’ll be honest I struggled with the name for a long time, but ultimately decided on 370Security because it is directly related to a success story that I want to share with you through this blog. Here it is – over the course of my career, and particularly over my past three roles, I’ve learned how to build a world class security team and security program that can reduce risk to a business by over seventy percent. Want to learn more? Stay tuned…

Security Team Assemble!

There are no bad teams only bad leaders and by corollary the success of a leader is directly attributed to the performance of their team. Now, a good leader can change the performance of a team from bad to great, but I’m getting ahead of myself. Let’s talk about building a security team and how to think about it.

First, it is rare for a leader to join an organization with a blank slate unless they are joining a very early startup. Most organizations will have some if not all parts of a security function and so a leader may need to assess if the existing org structure is meeting the needs of the business. In my particular case, I inherited a very small, flat security organization and it effectively allowed me to start from scratch.

What type of people does a security org need?

In general, a security org needs the following archetypes:

  • Builders – these are people who create things and are your engineers, product and application security specialists, devsecops folks, etc.
  • Defenders – these people find weaknesses and help protect against them. They are your security operations people, red team, blue team, physical security, threat intel, researchers and threat hunters.
  • Thinkers – folks who think about where the industry is going, what capabilities are needed, what policies and standards are needed, etc. They are your architects, governance, legal, privacy and security leadership at all levels.
  • Cat Herders (Organizers) – your organization may or may not need people to help organize, validate and keep things on track. If you do, these are your scrum masters, program and project managers, auditors, risk and compliance people.
How to structure it?

In my experience, a security organization will ultimately be structured as a reflection of the broader organization. What do I mean by that? If your broader org has separate legal, IT, sales, development, finance, hr, etc. orgs then generally you will need unique parts of your security organization to interact with each of these larger parts of the organization. Similarly, the maturity of the broader org will also influence how your organization is structured. If your IT group doesn’t have an accurate asset inventory or has poor hygiene around patching, posture, etc. then you may need to have a security function to augment this group until they come up to par.

In my particular case I started out with three groups – engineering, operations and governance and risk. I eventually added compliance, program management, product security and privacy and this model scaled really well up to a point (which I’ll talk about in a later post).

Security Org
Initial Security Org
Later Security Org

I was fortunate to have very supportive leadership that allowed me to hire and grow the team aggressively and this was one of the contributing factors in how I was able to reduce risk to the business by over 70 percent in 3 years.

If you are curious how to structure a security org Henry Jiang produced an excellent chart of all the different domains in cybersecurity. You can find it here.

I use this chart a lot in my current role to explain to people what a comprehensive organization should include. You don’t have to structure your org this way, but somewhere in the business someone should be performing the function and be accountable to security.

Ultimately, as a CSO/CISO people are not only your weakest link, but they are also your most valuable asset. The performance of your team and their ability to achieve results within the broader org is critically important to being able to manage or reduce risk to the business. People are the first and most important factor that allowed me to achieve the results I mentioned in my first post.

Up next, we will talk about creating a strategic plan.