October 2023 – 370 Security Blog

Exploring The Advantages and Disadvantages of Centralized vs. Decentralized Teams

This blog post is part of the Compliance Corner Series developed in partnership with Milan Patel. This series includes a variety of discussion topics around the intersection of security and compliance. The series includes blog posts, live web streams (with Q&A) and podcasts.

What is more effective – A decentralized or centralized security and compliance team? What are the factors you need to consider, what are the pros and cons of each model, does company size matter, are they simply analogs of organizational maturity or should leaders consider one model over another model for their org?

When leaders are creating or maturing their organization should they consider a centralized or decentralized organization structure?

Lee: If you have the opportunity to create or modify your organization I personally prefer a centralized organization structure. This is because it concentrates the roles, responsibilities and authority for security into a single function that can offer governance and all of the additional expertise expected of a security organization. The rest of the business knows where to go and who to talk to for all security issues. I have seen problems arise in both decentralized and heavily matrixed organizations because it confuses the roles and responsibilities of the function. Who is actually responsible for making security decisions if major parts of security are spread out across the organization? Sharing resources doesn’t really work very well because it is confusing for the individual team members and when sharing resources one side typically loses out to the other side. I have also seen shared resources get mis-used or repurposed for things other than security. This doesn’t mean the security team can’t place resources in different parts of the org, but they should report into and be owned by the security function. In my opinion whoever is responsible for the budget and the headcount truly controls that resource and decentralizing the budget and headcount causes problems.

Milan: Business leaders must first consider what role they want their compliance organizations to have. Will their compliance team actually offer governance, or just auditing? Are they going to cover corporate policies, or just audit frameworks that attest to customer reports? These are important scope questions to answer before setting up (or maturing) a compliance organization. It can drastically change how you fund and scope skills for the team, and whether a decentralized team will meet the overall risk management and corporate goals.

Investment size must also be considered, I get that question all the time, “How much should a business invest in compliance?”. I have seen everyone from flat personnel-project based funding, to actual percent of overall business operations spend. I focus on scope first, as then you can directly cost out what the deliverables/responsibilities are. Governance will drive a big factor of centralized or decentralized teams. Governance requires authority, charter, and appropriate level of independence to actually hold teams accountable. In a decentralized model, governance becomes much more difficult, as the fox ends up guarding the hen house.

Does company size, organizational maturity or other factors influence the decision to have a centralized vs. decentralized organization?

Lee: Company size can definitely influence the initial decision to create a centralized or decentralized function. Smaller organizations or startups may not be able to justify the initial cost of a dedicated security leader and may lump this responsibility under the CIO, CTO or Chief Counsel. As a result the security function may initially grow as a decentralized function until the organization decides it is either time to offload the original leader or they realize they need more specific security leadership and it is time to build out a dedicated function.

Organizational maturity can also impact the decision. Immature organizations may struggle to effectively use decentralized resources and so the weaker the organizational culture the more a centralized security organization will make sense. However, in really large organizations it is common to see a hybrid approach which I like to call a federated model. In a federated model you have a centralized security organization that sets policy, governance, manages risk, makes decisions and has all the authority for anything security. Business units within the large company then staff specific security resources based on expertise for specific industries or to help navigate their specific security and regulatory requirements. This can be advantageous in terms of presenting a single view of overall risk, consolidating processes and leveraging economies of scale for purchases to get a better price for tools or contracts used for security across the organization.

Milan: Company size, and breath of products, can definitely influence the model. In smaller companies, there will likely be less resourcing (and complexity) to consider, which makes a centralized model more affordable and practical. You are not going to have much ability to fund a larger team (and wouldn’t likely need it), so a centralized model pretty much is the only option.

In larger companies, decentralization is used (and we’ll talk about advantages and disadvantages later), but the better model is hub and spoke. A strong central team, chartered with governance, but small “spoke” compliance teams that are the boots on the ground in the team. Small presence that can keep engineering on track, participate in design reviews, threat model reviews, and know enough to ensure that engineering teams and products are on the right track from the start. They also can drive best practices for that team, but they are based on the central team requirements, and can escalate to the central team (that ideally has a governance charter) to ensure adherence at the right senior level.

What are the advantages and disadvantages of each model?

Lee: Centralized models offer consolidation of budget, resources, governance, responsibility and authority. It presents a single function that the rest of the business can go to for anything security related. Centralized models are typically more efficient because it avoids each group having to create and duplicate resourcing, tooling and processes. The one downside of a centralized model is if the security organization forgets that the rest of the business is their customer then it can become extremely difficult to interact with that group who effectively becomes a gatekeeper for business progress.

Decentralized models can offer some initial advantages when companies are extremely small. This is typical during startups or when you are operating in a mode where everyone is doing a lot of different jobs. However, this usually isn’t sustainable long term. I also find people who operate in this mode usually can’t scale to a larger organization where more governance is required. Decentralized models are also more prone to duplication of resources, technology and processes because there isn’t a single leader coordinating strategy and investment. Decentralized functions can also run into problems where the resources are misused or go “native” and stop performing the intended security role. Decentralized functions may end up with different levels of maturity across the different groups in the organization, which can make it difficult to obtain compliance certifications or to standardize processes and technology for a unified approach to security.

Milan: In general, a centralized structure offers the best overall coverage and governance. You can set consistent policies and practices across multiple organizations, which inherently will reduce risk as it’s easier to ensure consistency, and accuracy with one process vs many. You also can provide more controls to validate continuously that processes are working, plus attest much easier. Continuous compliance in a cloud environment is basically the norm now, but not all organizations, especially those with a decentralized model, can effectively ensure compliance of many regulations that come in and now must be enforced at the corporate level, and not just at the product level.

You also reduce cost, as having one set of compliance experts is cheaper, and can provide more optimization of skills. In a decentralized model, you end up having to hire more individuals, as you must replicate specialized skills in multiple areas.

One aspect that is often overlooked in centralized vs decentralized is pricing power. For compliance, for instance, you can collective bargain auditing to drive better prices in a centralized model. In a decentralized model, every team is determining it’s own bidding and metrics, which basically allows for suppliers to cost every team as individuals, reducing the overall negotiating power of the company. In a decentralized model, you usually also have more junior leaders (as the team and overall scope is smaller), and that dilutes the overall governance credibility, as they are not truly objective, as again, this can give the impression of the fox guarding the hen house.

Is there a clear winner here or is this more of a dogmatic approach / “it depends” type of answer?

Lee: Obviously there is always an “it depends” type of answer, but I personally think a centralized team offers far more advantages than a decentralized team. I have operated in decentralized teams, startups, and heavily matrixed organizations and they have all had incredible inefficiencies, process problems, lack of technological standardization and contention between the leaders in control of the different resources. While anyone can demonstrate leadership, the reality is there can only be one leader for a function. If you want to build a strong and effective security organization my personal recommendation is to avoid the decentralized model and strongly advocate for a consolidated, centralized function for all of the reasons I listed above.

No matter what size your company is, at some point your business will get big enough that it will either need to transition to or will need to build a centralized security org. Even when your company gets truly massive a centralized security organization will offer tremendous advantages for coordinating the rest of the functions across the business. This doesn’t mean you can’t have specific expertise embedded within the different lines of business, but there should be one overarching function that sets strategy, governance and has the authority to coordinate everything related to security across the organization.

Milan: I am going to lead off with a “it depends”, but “it depends” on what the SLT wants the function of the Compliance team to be, and how they want them to operate. For example, if they want what they “should” want. Corporate SLT should want an independent compliance organization that has the charter and weight to actually drive governance and accountability. Any decisions made by an engineering leader where the compliance team reports directly to them will be suspect if there is an issue, as how can compliance be seen as impartial if the decision can be overturned by the product or engineering leader directly? Did the right conversation happen, does that decision align with similar decisions with other product groups/lines of business? It can be a real problem if there is an issue and companies have to explain.

That is very difficult in a decentralized model. In a decentralized model where the compliance team, which has to drive hard messages and needs to engineering leaders, are they truly independent and will they speak up, as they tend to be mostly more junior, without any real organizational or peer power with the teams they are supposed to govern? The answer I’ve seen is rarely. I’ve seen and worked with many compliance teams that are frankly afraid to raise issues, or particularly escalate (and if they would escalate, who would they escalate to, as it would be their own management that signs their pay stubs). I’ve seen it both on the compliance and security side, where even mid level leaders will not raise or push issues, as they are worried for their jobs. It’s very difficult to find compliance teams and leaders that can truly be “politically unencumbered” in terms of raising issues, when they report to the fox that likely already doesn’t like having to do compliance work.

I believe that a strong and chartered central team, made up with the right personnel that understand engineering and can translate, and govern engineering compliance practices is the overall best option, particularly for larger organizations where standardization and efficiency must be improved. In a large company, compliance “spokes” with specific charter are important, as it’s the only way to scale the appropriate knowledge down to the teams.

What Causes CISO Burnout?

Burnout isn’t exclusive to the security industry, but it certainly seems like there is a higher incidence of burnout within security and particularly among CISOs. I have had my fair share of burnout and have tried a lot of different techniques to recover, however for this post I want to cover – What are the most common causes of CISO to burnout?

Lack of Appreciation

There are a number of reasons for burnout, but one of the main causes is lack of appreciation. This could be something as simple as celebrating the successes of the security organization more broadly or it can be more complex like advancement to the next level within the company. Given the broad range of CISO levels across the industry, advancement is certainly one of the issues that can manifest as lack of appreciation. For example, I see a lot of head of security positions or CISO level positions posted as Director or Sr. Director level positions. While this may make sense from an organizational standpoint it can put the CISO role at an inherent disadvantage compared to their other peers (like the CTO, CIO, etc.). Celebrating the successes of the CISO, acknowledging their contributions and promoting them to the appropriate level based on their scope and impact is one of the first ways you can reward a CISO, acknowledge their value and prevent burnout.

Lack of C-Suite Support

Another main reason for CISO burnout is the lack of equivalent C-Suite support. If the CISO isn’t supported by their peers and is always at odds with them, this will lead to burnout very quickly. Being on an island all alone is no fun, particularly when dealing with complex issues like security or when attempting to change behaviors across the organization. A CISO needs support and the C-Suite needs to be aligned to the overall security goals of the organization otherwise the rest of the organization will undermine the CISO. Without this support the CISO will spend all of their time just battling for political relevance instead of actually identifying and managing risk and this will lead to burnout.

Too Many Responsibilities

This can vary depending on organization size, but it is not uncommon to see a CISO with additional responsibilities such as Privacy, Data, Risk, Compliance, etc. all in their remit. Typically a CISO does deal with these things, but the organization has to be careful to not lump things under the CISO just because there is no one else to do it. The CISO organization shouldn’t be the janitor or garbage dump for the rest of the org and they shouldn’t be the place where all the misfit toys go. If the organization is going to lump additional responsibilities onto the CISO then those responsibilities need to come with support in terms of additional budget or resources. In addition to responsibilities and resources, the CISO needs to understand their strengths and weaknesses and delegate accordingly. Otherwise, the CISO will take on too much, not be able to scale and burnout.

Operational Burnout

Operational burnout is a big problem. If your operational tempo requires the CISO to constantly deal with incidents, respond to investigations, answer regulatory questions, deal with lawsuits, etc. then the CISO won’t be able to focus on other parts of the job like driving strategy, managing risk or prioritizing resources. Instead, they will constantly be reacting to situations which causes stress and takes a toll on personal health. Operational tempo can be difficult to manage, particularly if the CISO organization is understaffed, which means the team can’t maintain normal working hours. Personal care, such as maintaining normal routines to eat, sleep, exercise and decompress, can be seriously impacted if not managed properly during operational situations and this will lead to burnout extremely quickly.

Another area of operational burnout is constantly dealing with vulnerabilities, keeping up with the the latest trends, dealing technical debt or responding to increased reporting requirements. All of these scenarios have a never ending aspect to them and can cause the CISO to begin to feel like the situation is hopeless.

Risk Tolerance Misalignment

This is a very common area of frustration for CISOs and it boils down to not feeling appreciated. If the CISO is constantly making reasonable recommendations for how to manage risk and the business ignores them then there is clearly a risk tolerance misalignment, which will ultimately make the feel CISO unappreciated. To be clear, I’m not expecting every recommendation to be followed because you don’t want to get into a chicken-little type of scenario, but the CISO needs to have enough organizational credibility that the recommendations are acknowledged, considered and discussed. Organizations that don’t do this will rapidly find their CISO burned out and seeking other opportunities because you can only be ignored so many times before taking the hint and moving on.

Shiny Object Syndrome

At the next conference you go to, take a look around at the vendors and read their messaging. I bet you find it is hard to tell the difference between several of the companies there. Buzzwords like threat intelligence, machine learning, block chain, artificial intelligence, next generation, zero trust, etc. all hype up companies with buzz words, but remove differentiation for decision makers like CISOs. Keeping up with the Gartner Hype Cycle and the latest product buzz words is tiring because CISOs really just want to know what your product does and if it will be useful to solve their problems. Continually having to sift through the noise of buzzwords and hype is exhausting to CISOs and can burn them out quickly to dealing with vendors.

On the other side of this equation is technological churn. If your organization is continually implementing new tools and then replacing them after a short period this can also cause burnout. A security function needs a certain amount of stability and predictability to be effective. Shiny object syndrome from executive leadership or other parts of the business can make it difficult for a security team to find their natural rhythm or implement effective processes around those tools. Shiny object syndrome can quickly burn through the credibility or effectiveness of a CISO, which can ultimately lead to burnout.

Impossible Goals

It takes a considerable amount of effort for a new CISO to make their mark, effect change and achieve their goals at a new organization. This effort alone can cause CISOs to burn out, but it is made worse when the organization has impossible expectations or sets impossible goals for the CISO and their team. Examples of impossible goals are – achieving a compliance certification within an impossible timeframe, improving security posture when there is a significant amount of technical debt or trying to meet expectations for response times without appropriate staffing. The CISO needs to set realistic goals and have the latitude to adjust those goals as needed to avoid burning out.

Lack Of Accountability

The last situation that is sure to cause burnout for a CISO is lack of security accountability in the rest of the organization. If the business expects the CISO function to magically fix all of their security problems without support then that is unrealistic. The business (think CEO) needs to hold the other C-Suite members accountable for supporting and meeting the security objectives set by the CISO. If this accountability is not in place then other parts of the business will keep making decisions that increase risk or incur security technical debt, which places the CISO in an impossible situation and will cause burnout.

Wrapping Up

Burnout is an unfortunate byproduct of an otherwise exciting industry. CISOs are particularly ripe for burnout due to the immaturity of the role with respect to other C-Levels and the wide range of responsibilities that can be lumped under a CISO. Additionally, industry hype, lack of resources, lack of accountability and operational tempos can all cause CISOs to burn out. A CISO who is burned out is not as effective at their role and the level of burnout will take a proportional level of recovery. Hopefully, the examples above can help you recognize common situations to avoid or if you find yourself in that situation, recognize that it will quickly lead to burnout so you can make proactive changes and keep leading your team effectively.

What happens post incident?

One of the most exciting, stressful and true tests of a CSOs ability to lead during crisis is during a security incident. Unfortunately, it is inevitable that CSOs will experience several security incidents during their tenure. This could be something as small as a configuration error, or as large as a full on public data breach. I also think CSOs should assume they are operating in some state of compromise such as a malware infection or an attacker with complete remote access. Given this volatility of enterprise environments and the inevitability of some sort of security incident, the question becomes what happens afterwards? Just because you are no longer under active attack doesn’t mean your work is done. As a CSO, here are some things you should consider after an incident:

Retro / Post Mortem

First, I highly recommend conducting a retro or post mortem on the incident. This is a blameless session to discuss what happened, why it happened and most importantly what the team learned from the incident and what they are collectively going to do differently. During this exercise the CSO should plan to ask questions and listen a lot. I find being the note taker is exceptionally helpful because it is difficult for me to talk while taking notes. I want to hear opinions and ideas from the team on what happened and how we are going to improve. The result of this retro will likely kick of follow on activities such as increased training, requests for investment, development of new processes, creation of new detections or even additional automation. The point is it is a time to learn and rebuild.

Investment

Never let an incident go to waste. Instead of viewing the incident as a failure, I choose to look it as an opportunity for growth. It is easy to point the finger after the fact, but no environment is 100% perfect and secure. Therefore, after the retro is completed it is important to capitalize on the incident to ask for additional investment to respond to the improvement areas raised in the retro. This could mean asking for additional personnel to respond to events, it could mean additional training to fill knowledge gaps, it could mean a new tool or technology to improve detection and response or new processes to improve responses. As the CSO you need to distill down the retro suggestions into an actionable plan that focuses on the risks along with areas for improvement and investment.

Increased Regulatory Requirements

Unfortunately, there are consequences for having a security incident. For publicly traded companies this can come in the form of increased regulatory requirements such as being required to have an independent 3rd party audit your security practice on a periodic basis. It could also mean fines or reporting the incident as material in your upcoming SEC filings. You may even get inquiries from various regulatory bodies asking what happened and what you are doing to prevent it in the future.

Legal Ramifications

Similar to the regulatory requirements your company may face increased legal pressure as a result of the security incident. This could come from a number of different sources such as: lawsuits from outside entities (such as a consumer group or class action), increased contractual obligations from customers who are now concerned about your security practices, law enforcement investigations and costs for outside counsel who have expertise in your specific business area and are helping your company limit the damage.

Financial Implications

This is probably the biggest area for a company to navigate after an incident. Financial implications can manifest in a number of ways. Here are a few:

Increased Cyber Insurance Costs

As a result of the incident, your company may face an increase in cyber insurance premiums. These premiums may even become so expensive that your company can no longer afford to pay them, or your company could be deemed uninsurable.

Customer Impact

Depending on your business, this could be something simple like contracts not getting renewed and loss of new business or it could be something more material like having to pay to notify all of your consumers, paying for credit monitoring and having to compensate your loyal customers in some way to retain their business.

Market Impact

This is a broad area, but post incident your company could face a decrease in stock price. It may even be more difficult for your company to secure lines of credit because of the business risk. The severity and financial impact of the incident could potentially put your company at risk for M&A takeover or may require the business to declare bankruptcy and look for a buyer who has enough capital to weather the storm.

Budget Impact

Again, this is a broad area, but whenever I have an incident I try to keep track of the number of people involved, number of hours it took to deal with it and the opportunity cost of dealing with this versus doing something else I had planned. As an example, the opportunity cost could be something like “as a result of this incident we had to delay project x for 3 months while we dealt with the incident.” All of this will have an impact to timing, budgets and manpower.

Can You Calculate The True Cost Of An Incident?

Unless your company keeps track of everyone’s time and is able to create a time code entry for each specific incident, I find it unlikely that we will ever know the true cost of an incident. There are many direct costs, but there are so many indirect costs that are hard to estimate. The ramifications of an incident may stretch on for years and may change hands from one CSO to the next. However, I do think a CSO should have enough grounding in business principles to be able to estimate the cost of an incident. This can be useful when gaining support from your other C-Suite peers, when presenting to the board or when making investment requests to the CFO. Having context for all of the ramifications of an incident along with potential areas of growth and improvement can be a valuable story to tell.