September 2023 – 370 Security Blog

Compliance Corner Series: Building A Successful Security & Compliance Program

This blog post is part of the Compliance Corner Series developed in partnership with Milan Patel. This series includes a variety of discussion topics around the intersection of security and compliance. The series includes blog posts, live web streams (with Q&A) and podcasts.

What are the primary factors, considerations, and challenges that a security leader needs to consider to build a successful security and compliance program?

If you were starting a security and compliance program from scratch where would you start?

Lee: I’ve written about starting security programs from scratch in past blog posts, but in my opinion a security leader needs to first select what security controls framework they want to use to inform their strategy, initial assessment and planning. It is rare that a security leader will enter an environment that has zero security and so my suggestion for any leader is to conduct an initial baseline assessment for how well the organization is meeting the security controls within the framework they select. Two of the most popular security control frameworks in the United States are the CIS Top 18 Critical Security Controls and the NIST Cybersecurity Framework. Both of these are excellent frameworks that cover everything a comprehensive security program needs to have along with a recommended prioritized order. It will be difficult for a security leader to start a compliance program without baselining and mapping their environment to a security controls framework because this is the equivalent of never practicing a sport, yet going out and trying to play in a game. Security is the practice and discipline that sets the foundation for successful compliance programs i.e. the sport you are playing. Your security strategy will inform the what skillsets you need to hire, what tooling you need to build or buy and what processes you need to establish to make your security program successful.

Once you have selected a security control framework and have a strategy in place, my recommendation is to then evaluate your business for the markets it addresses, the geographic regions it operates in and the customers or partners it serves. This will inform your compliance team what compliance frameworks and audits they need to consider to make the business successful. For example, if you are a consumer based business and process credit card payments, then you will need to plan to comply with and pass a PCI-DSS audit or if you plan to serve the U.S. Federal Government you may need to achieve some level of FedRAMP. Understanding the business drivers, customer demand and regulatory environment is extremely important to setting up a successful compliance program that enables the business.

Milan: I definitely agree that you must start with a baseline assessment of where you are, but also with what your business goals are. Are you going into financial spaces and require SOC1 Type 2? Do you want to baseline your core expectations with SOC2? What about healthcare, HIPAA anyone? We won’t even get into FedRAMP if you want to sell to government agencies, so I suggest that you need to really understand, and baseline, your business goals in terms of verticals, industries, even governments/regions you want to sell into. The compliance leader will need to balance out these requirements (including privacy, accessibility, etc), to ensure that for whatever region/country/industry you want to go into, you can cover the core baseline controls.

Once you have that, you need a way to track and ensure you can meet those controls. Like we talked about before, and as I talk about now, integrating this with engineering and the appropriate metrics are key. This includes how to “pre-audit” your services for new frameworks/certifications so you can minimize issues during an actual audit. If you don’t do this from the start, retro-fitting it in after the fact is difficult and costly.

You also then need personnel that understand these controls and requirements, and importantly how to translate and negotiate these with engineering. If the people you hire cannot do that, then you fall into the same old challenge of friction between engineering and compliance that introduces risk. Having the right personnel that can translate and bring harmony among the pillars required for compliance (compliance, engineering, security, and operations) will be key to standing up a successful compliance team.

2. What if you inherit a security and compliance program that has been established by your predecessor? What advice do you have for leaders when adopting and continuing a program established by someone else?

Lee: Congratulations if you have been promoted or landed a new role as the leader of a security or compliance program! In my opinion the fundamentals don’t change whether you are starting a program from scratch or inheriting an existing program. I think it is critically important for a leader to baseline and understand what is already in place, what gaps exist and then build their strategic plan to address those gaps. The big difference when you inherit a program is it may take longer for the business to shift priorities towards the new plan due to organizational momentum and budget cycles. Technologically and practically, it may not actually take that long to do the things on your plan, but psychologically the organization will need to adjust and adapt. This means you will need to spend more time building alliances, getting teams on board, changing processes and shifting the culture to support your new vision. Keep in mind that closing the gaps in security control frameworks or compliance activities may take a significant amount of time because the organization may need to procure new tools, hire new talent or create new processes to satisfy a specific control. This has a knock on effect to your compliance program because most compliance audits have both readiness audits and the actual audit themselves. Both of these can take months to complete and so expectations need to be set for timelines, critical paths and when objectives will be met. If the business is expecting to get FedRAMp or HyTrust overnight, that is unrealistic because there are robust processes and timelines that need to be followed to pass those compliance activities.

Milan: I still agree that it doesn’t really change much from standing up a new compliance program, other than you may have a bit more work to change the culture from bad practices to new ones. The key is still to align your compliance strategy with engineering goals/metrics. Inheriting a program that doesn’t align to engineering requires immediate attention, as until you can align it, it will be difficult to help the cultural and behavioral changes needed to more closely align priorities and drive the actual execution.

As you say, it will likely still take time to close gaps, hire individuals, build tooling, but if the actual metrics and success criteria isn’t aligned with what engineering will need to have to actually change priorities, then most of that work may become “wasted” as while it may get you through a single audit, it won’t be efficient or at scale. I would still suggest that new compliance org leaders take a deep look at how the compliance team articulates and aligns success metrics first, before doing anything else, and change/align them to engineering first otherwise they will continue to have friction, and risk, with executing the compliance programs that require engineering (or even operations and security) changes.

3. What challenges can a leader expect to face when building or maturing a security and compliance program and what advice do you have to overcome these challenges?

Lee: There are a few challenges that I have run into over the course of my career. The first challenge is mis-alignment of expectations between stakeholders. This could be a mismatch from the board, from other C-Suite executives or from the rest of the business (like sales). In these cases, it is imperative for the security leader to work closely with these stakeholders to align expectations. You are the expert in this field and you need to help these other leaders and teams understand what good looks like and when they can expect to achieve it.

Expectation mis-alignment is also closely related to cultural mis-alignment. For example, if the technical culture is extremely permissive and allows teams to do whatever they want, then the security leader will need to spend time explaining the advantages of increased security control and how that can actually accelerate the business, while reducing risk. Culture mis-alignment is often the biggest challenge I’ve run into over the course of my career and it can be very time consuming to fix. Security leaders are often brought in to improve a security program, when in reality the culture is at issue. This often means the security team becomes a cultural change agent and they cannot change the culture alone. Security leaders will burn out or leave extremely quickly if the culture is not aligned and they don’t have the support of other executive leaders to achieve the security goals specified in the security strategy.

Finally, one of the other issues I’ve run into is budgetary. This can come in two forms – first, the security team struggles to get appropriate funding and second, the business insists the security or compliance team performs activities that may not actually make sense financially (like achieving a specific compliance audit). The first budgetary issue – not getting appropriate funding can be a real problem for the security team if the business arbitrarily cuts budgets instead of choosing to fund the security team based on risk. This funding issue is closely related to a mismatch of stakeholder expectations and a cultural misalignment. The second budgetary issue is often an issue between sales teams and the security and compliance team. Often, the sales teams insist a specific security control is necessary or a certain compliance framework is required in order to exist in a certain market or achieve a sale. While this can often be the case, it is important the security leader has the support to be able to push back on these requests where it makes sense. If a security and compliance activity is insisted on by the sales team, I always require a minimum threshold of business to cover the cost of the control and compliance activity. This should be written into the sales teams number, tied to their comp and ultimately they should be held accountable for justifying the need.

Milan: I still think the biggest challenge for compliance leaders is building credibility. I talk about how leaders look at all companies as “making money, saving money, or costing money”, and compliance has always been put into “costing money”. So how does a compliance leader build their individual and team credibility with the engineering teams (the main control owners), that they are just not a pain in their collective engineering sides “costing money”?

Compliance leaders must start from the beginning showing engineering how they are “helping them”, which mainly to me is “saving money” (engineering/operations/security time). Time is the key metric that is tracked by most engineering leaders (sprint and engineering time), so it’s incumbent on compliance leaders to show them how they are efficient, or even saving time based on their previous compliance experience. Compliance leaders must be able to show, on paper with metrics, that the compliance work is efficient (I use first time resolution as a critical metric), and how we continuously drive for the ultimate metric, of removing engineering completely from the evidence collection flow via automation, etc.

If you cannot build that trust, then you will always just be a leader that doesn’t understand the engineering priorities, cost them money (i.e. time) by being inefficient, and having no credibility and influence as they have no real reason to engage with you. It all starts with the ability to develop and agree on metrics that actually help engineering minimize the time it takes to do auditing/etc, and providing that value back to the actual engineers.

You get risk mitigation by default with the right metrics, as the evidence is clear and concise, and once first time resolution is high, you can now automate it. We have seen that the risk (and the deviations) have gone way down as we negotiate and approve with external auditors the evidence that will pass that QA for review, we have less “drift” with bad evidence, which reduces overall risk (and actually increases predictability and timeliness in audits).

I also believe that the requirements “accepting” of new frameworks requires a much more detailed review and acceptance than I’ve generally seen. There is no point gaining new certifications if sales doesn’t have a solid plan to get the ROI with customers. It’s costly to go through the certifications, and without appropriate sales to support the cost, it can become wasted work. If the cost/benefit numbers and plans for sales are good, then it becomes much easier to justify the compliance work. Also, if the success metrics and tracking of workload for compliance is also detailed, then justifying resourcing for new projects becomes easier. Compliance teams must continuously run compliance “like a business” being able to show metrics that detail out the work, down to the team and individual level, otherwise justifying new resources becomes challenging.

The Most Powerful Word A CSO Can Say Is No

A Tale Of Two Extremes

A few weeks ago I was sitting across the dinner table from a CIO and a CISO who were discussing how security works within their specific businesses. The CIO was complaining that the security team had unreasonable processes that slowed down his ability to deliver new technology projects within his org and as a result he ignored a lot of their requests. This resulted in an engaging discussion about the best way to balance security requirements against the needs of the business. I found it interesting that some of my fellow CISOs were adamant about having teams meet security requirements without exception, regardless of the impact to the business.

After this discussion I spent some time thinking about own my stance on this issue and how I have tried to balance security requirements against the needs of the business over the course of my career. I pride myself on finding ways to partner with and support the business, instead of just telling them no all the time. However, I have also found that sometimes the most powerful word in my vocabulary is NO. Saying no is particularly useful during the rollout of new processes or security controls where you are changing behavior more than you are implementing a new technology.

Tug Of War

Security requirements and business needs can often be in a perpetual tug of war. This isn’t necessarily a bad thing when you consider the purpose of a CISO organization is to help protect the business not only from attackers, but often from itself. However, it can be difficult when the tug of war is biased towards one side or the other. If the business simply steam rolls and ignores all security requirements then clearly the CISO isn’t valued and the business isn’t interested in managing risk. However, if the CISO says no to everything, then this can be a significant and costly drag on the business in terms of people, processes, technology and time. Lost productivity, lost revenue, inability to deliver a product to market quickly can be difficult to measure, but have real impact to the business. Worse, the business may just ignore you. It is important for CISOs to find an appropriate balance to allowing the business to function, while meeting the desired security objectives to protect it. I firmly believe when done correctly, CISOs can avoid being a drag on the business and can actually enable it.

Just Say No

Despite my general inclination to find ways to keep the business moving forward, I’ve also found saying no to certain things can be extremely useful. For some things, when teams want an exception I usually say no as my initial response. I have found often teams just need to hear an exception isn’t an option and this unblocks them to pursue another alternative to the problem. As a result the teams improve their security while also delivering their business objectives.

Sometimes the teams will ask for an exception a second time. In these cases, I usually reconsider, but often still tell them no. My expectation after telling them no a second time is to either get them to fix the issue or if the issue can’t be fixed to present a plan with different options along with their recommendation. When teams come back for the third time it ends up being an actual business risk discussion instead of an exception discussion. The outcome usually ends up being some sort of compromise on both sides, which is exactly what you want. Taking a balanced approach develops an appropriate level of partnership between security and the rest of the business where one side isn’t always sacrificing their objectives for the other side.

Seriously, Just Say No

Next time a team comes to you with an exception request try saying no and see if they respond differently. You may or may not be surprised when they find an alternate solution that doesn’t require an exception. For me, it has become a powerful tool to nudge teams towards achieving my security goals, while still delivering on their business objectives.

Compliance Corner Series: Compliance Landscape 2023

This blog post is part of the Compliance Corner Series developed in partnership with Milan Patel. This series will include a variety of discussion topics around the intersection of security and compliance. The series will include blog posts, live web streams (with Q&A) and video blogs.

“How has globalization, increasing regulatory requirements from governments and industries change how Security and Compliance must operate internally”

Growing Regulatory Requirements – Impact To The Whole Company vs Single Business Unit

Lee – From a security perspective, it is difficult to standardize security controls and baselines across the whole company vs. a single business. Business lines have different requirements, customers and industries, which are unique to their product lines and customer base. Mandating the entire company meet a global regulatory requirement (like PCI-DSS, FedRAMP or HIPAA) can be costly for those lines of businesses that don’t have customers in those industries where these regulations apply. While the security program may be effectively managing risk for their particular line of business, the difference in regulatory requirements can lead to gaps in customer expectations across product lines or lines of business. It can also be costly to maintain compliance programs since you typically have to audit and produce evidence for each product. This puts a burden on development teams and can take time away from improving the security of their products, vs. generating sufficient evidence to pass audits. For smaller lines of businesses this can be difficult to justify the cost to keep up with global regulations, certify audits and maintain levels on par with the rest of the company.

It is possible to centralize compliance activities across the company, but it also requires every part of the company to standardize their security controls, risk management practices and reporting principles. This may work well for small to medium sized companies that have grown organically or who have done a good job of homogenizing their organization. For larger organizations that have grown via acquisition and have dozens of lines of businesses and product lines, centralizing compliance and security activities may be impossible.

Milan – For compliance, the issue is that many of these new requirements have no “corporate” baseline expectations. They have not been translated (and sometimes agreed to), by individual lines of business. Without a standard “bar” to execute against, it’s very difficult for individual compliance teams to attest that it’s being met. In many cases, these items have no tests, or have not even been implemented. It becomes a real issue at scale, as these are yet another new compliance item that engineering will have to meet, which will continue to compete with their feature development time requirements.

My experience is that it’s challenging for many of the central compliance/legal teams (that try to set these requirements for engineering teams), from both a positional authority, as well as a credibility perspective. There are few engineers on these central compliance teams that are experienced with translating these regulations to actual engineering practices. Also, there is a lack of organizational authority to enforce it uniformly across all business groups. This just increases the challenge as it adds more time to negotiate and drive priority, especially with the speed of these incoming regulations.

2. Security and Compliance Practices – What Has Changed?

Lee – For security, there has been an increasing trend in transparency as well as a reduction in time to fix security vulnerabilities and report security incidents. Often these regulations or new compliance practices have good intent, but will be so costly to implement that they are impractical or are technologically impossible. These problems are further compounded by modern software development practices like DevOps that move incredibly quickly and can amplify deployment of internet scale products across cloud providers. Software supply chain security and reporting for compliance reasons is particularly challenging due to the complexity of modern software libraries that have multiple interlinking dependencies. Third party libraries and open source software is even more difficult because you don’t always know if you can trust the source code of that library. Providing an accurate software bill of materials (SBOM) can be a significant burden on the business to develop and maintain for compliance reporting purposes. Additionally, the priority of the business can sometimes take precedence over other activities like solving tech debt or modernizing technology stacks. Technical debt can not only hold back the business from optimizing and becoming more efficient, but can also make security and compliance activities challenging.

Milan – I’ve been doing this for a long time, on the engineering side with a large multi-billion dollar software product, and more recently on the compliance side across many products. What I can say is that while engineering has changed… DevSecOps, updating features/software monthly, instead of yearly… or even every 5 years when hardware went “end of life” (yes, I’m totally aging myself here), compliance practices basically have not changed internally.

Even major companies are still trying to “throw money” at the problem. The internal compliance teams know that the SLT has to pay for compliance… but I believe the days of the compliance “bat” and “blank checkbook” are gone. It doesn’t scale. I will never get more bodies to manually collect and review evidence for large amounts of controls and services (and not regulatory requirements) that we will have to test and account for. Spreadsheets and email won’t work anymore. Compliance needs to change, and be run more like a business. Even the third party auditors didn’t have incentive to be more efficient… remember, they mostly bill hours… So the more inefficient and longer it takes for a company to do an audit, the more money they make. We’ve changed that quite a bit at Oracle, but it’s still early in the journey.

3. Looking Ahead: How Must Security and Compliance Practices Change?

Lee – For security, teams need to shift to a continuous compliance model. Instead of periodically assessing the business once a year (or some other interval) teams need to implement security controls and reporting mechanisms that can automate and continuously evaluate how the security controls meet the various regulatory requirements required for the business. By moving to a continuous compliance model it will force the business to modernize their Software Development Life Cycle (SDLC), modernize their technology stacks and resolve technical debt, which will ultimately benefit security programs. A continuous compliance program will allow security teams to tackle security at multiple points across the technology stack. First, it will evaluate, report and control how software is written. Second, it will evaluate, report and control how infrastructure is deployed and third, it will evaluate, report and control how security posture changes over time so teams can react to security drift.

Security and compliance practices must also change by holding the rest of the business accountable for their decisions. It is impossible for a security or compliance team to be successful without support from the rest of the business because the security and compliance teams don’t “own” the infrastructure and products the business uses to make money. Therefore, there needs to be support at the highest levels of executive leadership to hold the lines of business accountable and tie performance and incentives to how these businesses satisfy their security and compliance requirements.

Milan – Well, I think it must happen on two levels. First level is central oversight and governance.

Without a strong central compliance function that can translate, and negotiate requirements, enforce and govern equally with the engineering leaders, it won’t scale effectively. The main challenge now is that the “fox is guarding the hen house” so to speak. When the engineering team compliance is solely under engineering leaders in the product group, there is the ability (or even just the appearance) that decisions are made without compliance “equality” of decision making. If a compliance item isn’t resolved, who questions whether that amount of risk is acceptable to the senior product leader? Even if accepting the risk is “OK”, any external party can question whether the right level of compliance leadership made that decision, and not just an engineering leader who may prioritize compliance lower than say… a feature request.

This mostly only becomes an issue after the fact. A compliance issue happens, then the questions of “Who made that decision… why was that compliance issue not escalated… etc? Escalate to who? While it’s a great idea that a compliance leader (buried usually multiple levels down in an organization), will be “politically unencumbered” and rise up and strongly disagree with an issue with their senior leader (who by the way signs their checks). My experience is that it can be difficult, as being seen as an antagonist is not great for one’s career in this space. I’ve noticed that compliance positions that have high turnover, this has been one of the main indicators of how effective the compliance leader is in the organization feels they are. If they feel they cannot speak up, or they do and are dismissed “for higher priorities”, they feel they are not effective, or will become the scapegoat if there is an issue.

The second level is that compliance needs to change their success metrics, and run like a business.

Most senior leaders I know, including myself, put everything into three buckets… You’re making money… saving money… or costing money… Compliance has always been seen as “overhead” and costing money. Even today, I’ve had to battle that, until we made some major changes in how we operate, and what metrics we track for success. So the question is, if I’m always seen as “costing money”, how do I change that, and what do I change it to? I realized that while compliance does “make money” (new frameworks open new markets), I don’t really get to claim that. I’m not out there hustling deals… So I realized years ago I could “save money”, but not in the traditional sense. I had to change my compliance principles and metrics. First, everyone accepts that compliance is important, but why did they hate it so much? As an engineer doing this on a large project, the compliance team never understood that my metric was “completing the audit” or “collecting XXX number of evidence pieces”.

My engineering metric of success was TIME.

As a development operations, release manager, and development compliance owner for the large project, I only had so many engineering hours, so I needed to minimize time I spent on each engineering function, including compliance.

So in compliance, I changed the success metric to First Time Resolution (FTR) for audit evidence. Everyone knows you have to collect at least one, but we had so many “kickbacks” for many reasons.. Either the engineering team didn’t provide what was asked for, or the evidence instructions were incomplete or unclear (remember what I said earlier for “billable hours” from the auditors)? I put at least 30 min per kickback. I used to own 1000 pieces of evidence, sometimes I could resolve them in 5 min, sometimes it took days. By raising the FTR up to a goal of 95% (makes it through the first time), I can show, on paper… how much money I’m saving the engineering team (in time). I was able to show this year, saving millions of dollars in audit costs (by going with an efficient auditor, that didn’t bill me for all those hours), and saving Tens of Thousands of engineering hours by eliminating the “unplanned work” of kickbacks.

By changing the metrics to what the engineers care about, it built trust and transparency, and increased the overall value of compliance, plus their willingness to look for more efficiencies.

Needless to say there are a lot more details to talk about for the journey to get here, but the engineering leaders saw, with hard metrics, that compliance “saved money” (via time) for them to use on other features, bug fixing, etc. And more importantly, it built trust with them that the compliance team was a good steward of their currency, which is time.

4. Where do we go from here?

Lee – Milan, it’s been great discussing how we evolved to this state, and some great ideas and positions on what has to change in our respective security and compliance areas. There is so much to discuss here I’m looking forward to continuing this series to dive into topics around the intersection of compliance and security.
Milan – Yes Lee, it’s been great discussing these topics. For me, I feel like this is the beginning of another Industrial Revolution for Compliance, we have seen the change in the engineering and globalization, and we’re on the verge of revolutionizing the Security and Compliance space. We didn’t even really get into the overall governance and continuous security/compliance versus point in time. Maybe that’s the next discussion!