This blog post is part of the Compliance Corner Series developed in partnership with Milan Patel. This series will include a variety of discussion topics around the intersection of security and compliance. The series will include blog posts, live web streams (with Q&A) and video blogs.
“How has globalization, increasing regulatory requirements from governments and industries change how Security and Compliance must operate internally”
- Growing Regulatory Requirements – Impact To The Whole Company vs Single Business Unit
Lee – From a security perspective, it is difficult to standardize security controls and baselines across the whole company vs. a single business. Business lines have different requirements, customers and industries, which are unique to their product lines and customer base. Mandating the entire company meet a global regulatory requirement (like PCI-DSS, FedRAMP or HIPAA) can be costly for those lines of businesses that don’t have customers in those industries where these regulations apply. While the security program may be effectively managing risk for their particular line of business, the difference in regulatory requirements can lead to gaps in customer expectations across product lines or lines of business. It can also be costly to maintain compliance programs since you typically have to audit and produce evidence for each product. This puts a burden on development teams and can take time away from improving the security of their products, vs. generating sufficient evidence to pass audits. For smaller lines of businesses this can be difficult to justify the cost to keep up with global regulations, certify audits and maintain levels on par with the rest of the company.
It is possible to centralize compliance activities across the company, but it also requires every part of the company to standardize their security controls, risk management practices and reporting principles. This may work well for small to medium sized companies that have grown organically or who have done a good job of homogenizing their organization. For larger organizations that have grown via acquisition and have dozens of lines of businesses and product lines, centralizing compliance and security activities may be impossible.
Milan – For compliance, the issue is that many of these new requirements have no “corporate” baseline expectations. They have not been translated (and sometimes agreed to), by individual lines of business. Without a standard “bar” to execute against, it’s very difficult for individual compliance teams to attest that it’s being met. In many cases, these items have no tests, or have not even been implemented. It becomes a real issue at scale, as these are yet another new compliance item that engineering will have to meet, which will continue to compete with their feature development time requirements.
My experience is that it’s challenging for many of the central compliance/legal teams (that try to set these requirements for engineering teams), from both a positional authority, as well as a credibility perspective. There are few engineers on these central compliance teams that are experienced with translating these regulations to actual engineering practices. Also, there is a lack of organizational authority to enforce it uniformly across all business groups. This just increases the challenge as it adds more time to negotiate and drive priority, especially with the speed of these incoming regulations.
2. Security and Compliance Practices – What Has Changed?
Lee – For security, there has been an increasing trend in transparency as well as a reduction in time to fix security vulnerabilities and report security incidents. Often these regulations or new compliance practices have good intent, but will be so costly to implement that they are impractical or are technologically impossible. These problems are further compounded by modern software development practices like DevOps that move incredibly quickly and can amplify deployment of internet scale products across cloud providers. Software supply chain security and reporting for compliance reasons is particularly challenging due to the complexity of modern software libraries that have multiple interlinking dependencies. Third party libraries and open source software is even more difficult because you don’t always know if you can trust the source code of that library. Providing an accurate software bill of materials (SBOM) can be a significant burden on the business to develop and maintain for compliance reporting purposes. Additionally, the priority of the business can sometimes take precedence over other activities like solving tech debt or modernizing technology stacks. Technical debt can not only hold back the business from optimizing and becoming more efficient, but can also make security and compliance activities challenging.
Milan – I’ve been doing this for a long time, on the engineering side with a large multi-billion dollar software product, and more recently on the compliance side across many products. What I can say is that while engineering has changed… DevSecOps, updating features/software monthly, instead of yearly… or even every 5 years when hardware went “end of life” (yes, I’m totally aging myself here), compliance practices basically have not changed internally.
Even major companies are still trying to “throw money” at the problem. The internal compliance teams know that the SLT has to pay for compliance… but I believe the days of the compliance “bat” and “blank checkbook” are gone. It doesn’t scale. I will never get more bodies to manually collect and review evidence for large amounts of controls and services (and not regulatory requirements) that we will have to test and account for. Spreadsheets and email won’t work anymore. Compliance needs to change, and be run more like a business. Even the third party auditors didn’t have incentive to be more efficient… remember, they mostly bill hours… So the more inefficient and longer it takes for a company to do an audit, the more money they make. We’ve changed that quite a bit at Oracle, but it’s still early in the journey.
3. Looking Ahead: How Must Security and Compliance Practices Change?
Lee – For security, teams need to shift to a continuous compliance model. Instead of periodically assessing the business once a year (or some other interval) teams need to implement security controls and reporting mechanisms that can automate and continuously evaluate how the security controls meet the various regulatory requirements required for the business. By moving to a continuous compliance model it will force the business to modernize their Software Development Life Cycle (SDLC), modernize their technology stacks and resolve technical debt, which will ultimately benefit security programs. A continuous compliance program will allow security teams to tackle security at multiple points across the technology stack. First, it will evaluate, report and control how software is written. Second, it will evaluate, report and control how infrastructure is deployed and third, it will evaluate, report and control how security posture changes over time so teams can react to security drift.
Security and compliance practices must also change by holding the rest of the business accountable for their decisions. It is impossible for a security or compliance team to be successful without support from the rest of the business because the security and compliance teams don’t “own” the infrastructure and products the business uses to make money. Therefore, there needs to be support at the highest levels of executive leadership to hold the lines of business accountable and tie performance and incentives to how these businesses satisfy their security and compliance requirements.
Milan – Well, I think it must happen on two levels. First level is central oversight and governance.
Without a strong central compliance function that can translate, and negotiate requirements, enforce and govern equally with the engineering leaders, it won’t scale effectively. The main challenge now is that the “fox is guarding the hen house” so to speak. When the engineering team compliance is solely under engineering leaders in the product group, there is the ability (or even just the appearance) that decisions are made without compliance “equality” of decision making. If a compliance item isn’t resolved, who questions whether that amount of risk is acceptable to the senior product leader? Even if accepting the risk is “OK”, any external party can question whether the right level of compliance leadership made that decision, and not just an engineering leader who may prioritize compliance lower than say… a feature request.
This mostly only becomes an issue after the fact. A compliance issue happens, then the questions of “Who made that decision… why was that compliance issue not escalated… etc? Escalate to who? While it’s a great idea that a compliance leader (buried usually multiple levels down in an organization), will be “politically unencumbered” and rise up and strongly disagree with an issue with their senior leader (who by the way signs their checks). My experience is that it can be difficult, as being seen as an antagonist is not great for one’s career in this space. I’ve noticed that compliance positions that have high turnover, this has been one of the main indicators of how effective the compliance leader is in the organization feels they are. If they feel they cannot speak up, or they do and are dismissed “for higher priorities”, they feel they are not effective, or will become the scapegoat if there is an issue.
The second level is that compliance needs to change their success metrics, and run like a business.
Most senior leaders I know, including myself, put everything into three buckets… You’re making money… saving money… or costing money… Compliance has always been seen as “overhead” and costing money. Even today, I’ve had to battle that, until we made some major changes in how we operate, and what metrics we track for success. So the question is, if I’m always seen as “costing money”, how do I change that, and what do I change it to? I realized that while compliance does “make money” (new frameworks open new markets), I don’t really get to claim that. I’m not out there hustling deals… So I realized years ago I could “save money”, but not in the traditional sense. I had to change my compliance principles and metrics. First, everyone accepts that compliance is important, but why did they hate it so much? As an engineer doing this on a large project, the compliance team never understood that my metric was “completing the audit” or “collecting XXX number of evidence pieces”.
My engineering metric of success was TIME.
As a development operations, release manager, and development compliance owner for the large project, I only had so many engineering hours, so I needed to minimize time I spent on each engineering function, including compliance.
So in compliance, I changed the success metric to First Time Resolution (FTR) for audit evidence. Everyone knows you have to collect at least one, but we had so many “kickbacks” for many reasons.. Either the engineering team didn’t provide what was asked for, or the evidence instructions were incomplete or unclear (remember what I said earlier for “billable hours” from the auditors)? I put at least 30 min per kickback. I used to own 1000 pieces of evidence, sometimes I could resolve them in 5 min, sometimes it took days. By raising the FTR up to a goal of 95% (makes it through the first time), I can show, on paper… how much money I’m saving the engineering team (in time). I was able to show this year, saving millions of dollars in audit costs (by going with an efficient auditor, that didn’t bill me for all those hours), and saving Tens of Thousands of engineering hours by eliminating the “unplanned work” of kickbacks.
By changing the metrics to what the engineers care about, it built trust and transparency, and increased the overall value of compliance, plus their willingness to look for more efficiencies.
Needless to say there are a lot more details to talk about for the journey to get here, but the engineering leaders saw, with hard metrics, that compliance “saved money” (via time) for them to use on other features, bug fixing, etc. And more importantly, it built trust with them that the compliance team was a good steward of their currency, which is time.
4. Where do we go from here?
Lee – Milan, it’s been great discussing how we evolved to this state, and some great ideas and positions on what has to change in our respective security and compliance areas. There is so much to discuss here I’m looking forward to continuing this series to dive into topics around the intersection of compliance and security.
Milan – Yes Lee, it’s been great discussing these topics. For me, I feel like this is the beginning of another Industrial Revolution for Compliance, we have seen the change in the engineering and globalization, and we’re on the verge of revolutionizing the Security and Compliance space. We didn’t even really get into the overall governance and continuous security/compliance versus point in time. Maybe that’s the next discussion!