fbpx
713.893.3940

CISO Forum Summary – Measuring Success Through Metrics..

CISO Forum Summary – Measuring Success Through Metrics

Every month, Fortify Experts holds CISO Round Tables discussing the latest trends and topics.  Recently, we discussed how these successful security leaders have been able to create higher-performing security teams.  Here is a summary of their top tips or suggestions on how to measure the success of your security program?

  1. Purpose of Metrics

As a CISO, “You want to be able to tell a story, so what story are you telling.” The key question is, “How are you using those metrics?”

You should be asking, “What is my priority here?” every time you develop a metric. This is the question that every business person and every executive is always asking as they are allocating resources.

  • Board wants to know, “Are we investing in the right security capabilities to protect our assets?”
  • CFO wants to know, “How do we show value while managing costs?”
  • CEO wants to know, “What are the risks of a financial loss if we don’t take action?”

Metrics:

  • provide the platform for creating the story that you need to tell, and you need to be able to carry around with you on a day-to-day basis
  • are the containers that carry all of that detail which are important to our jobs.
  • build confidence within the executive team that your security and compliance program is actually working.
  • help you stay more strategic and less reactive.
  • provide a view into what has to be done going forward.
  • Are key performance & key risk indicators that help us justify our existence to senior execs.

KEY POINT:  Metrics can change based on the behavior that needs to change.  Larger vs. Smaller companies, regulated vs. non-regulated, immature vs. mature, technology vs non-tech dependent, all play into which metrics boil up to be critical to measure.  There is no one size or one set of metrics that fits all.

CIA TRIAD:

The CIA Triad can be used as a foundation for Metrics: https://whatis.techtarget.com/definition/Confidentiality-integrity-and-availability-CIA#:

  • Confidentiality
    • Confidentiality limits access to information and is more important from a regulatory or compliance point of view (progress of GDPR, servers not on 2FA, unprotected PII data repositories)
  • Integrity:
    • Integrity is the assurance that the information is trustworthy and accurate unauthorized access is not allowed (related to patching)
  • Availability
    • Availability is more important in operations because you care about the volume of applications you’re processing and the throughput how quickly they are able to be addressed and circulated (impacted by ransomware,
    • Day-to-day business is going to care about the flow so maybe like what controls are around availability of data to make sure that it’s unencumbered and what security measures ensure that availability.

KEY POINT:  In the CIA Triad, availability is probably the highest most scrutinized piece to keep data flowing, followed by integrity of data from an operational standpoint.  There’s an assumption that privacy that’s probably going to help drive confidentiality.

Most Useless Metrics: 

A lot of Metrics create a “So What?”  Here are some of metrics that leaders have been asked to provide but can have little actionable value:

  1. Number of malicious or phishing emails that come in (spam)
  2. Total number of unpatched vulnerabilities
  3. Number of attacks thwarted
  4. # of dropped packets on the firewall
  5. Ranked Vulnerabilities when there is no business context to know the actual risk.
  6. # of actions items required to get a system compliant
  7. # of systems that are compliant

KEY POINT:  Instead just posting up metrics when asked, focus on high value metrics.  Those are the ones that lead to a decision point and drive change.

  1. IT Operational Metrics:

Operational metrics are useless without the business context behind them.

Phishing Training (often provided by the Phishing tool):

  • # of people positively report a Phish without interacting with it
  • # who failed by clicking
  • # of people who did nothing with it (i.e. Apathy score) – most concerning?

Cloud:

  • With the migration to the cloud, measuring the CIA metrics in your DevOps chain becomes much more complicated and difficult to monitor.
  • Companies that don’t do DevSecOps management well in the future are going to lose ground very quickly in agile environments.
  • DevOps & Security Ops should partner to defining how they can work together more efficiently.

Track Ugly Metrics:

  • You need to be ready to track some ugly metrics right now like:
    • How many how many applications aren’t passing vulnerability scans?
    • How many applications aren’t being screened for security criteria?
    • When they’re screened, how many of them have vulnerabilities and how many of them don’t?
    • How many emergency changes occurred where DevOps didn’t do their security checks or made changes outside of the process.
    • How many critical servers with a high vulnerability remain unpatched.
  • Use the CVSS scores to classify and highlight the % of exploitable or highly exploitable vulnerabilities that exist.

KEY POINT:  Measuring the Ugly Metrics in DevOps helps teams focus on the stuff that really matters and less on the stuff that doesn’t and moves a really unstable state towards a stable state.

Other Operational Metrics:

  • Measuring % of devices which were N vs. N-1 (i.e. Within a standard or compliance vs. a variant of the standard or out of compliance.)
  • Patching performance metrics against defined risk categories or vulnerabilities that have been identified over time (i.e. 6 months).
  • # of people who didn’t have multi-factor authentication turned on
  1. Business Operational Metrics
  • It takes time to strategically think about these metrics to ask “how can I identify and track what’s important to the business.
  • Define business success factors such as what is needed to ensure Productivity, prevent Revenue loss, Protect Market Share and reputation, etc.
  • What levers can the security team pull to give the business a strategic advantage or to take risks off the table. The determine if they are worth investing in?
  • Define what Risks could take the company off of our mission over the next 18 to 24 months?
  • Develop a data management committee – Include a leader from each business unit to discuss what they care about. Side benefit is that the data management committee is also involved in the board meeting as provides positive feedback because they were involved in the process of establishing what was important to them.

KEY POINT:  Push to hire a cyber expert on the Board to help sell the importance.

KEY POINT:  Only track 3 to 4 things that are the most important to the business.  More than 5 is too much.

Mike Davis shared his detailed report on NIST based Cybersecurity Metrics.  You can download, use and modify this document:  (Scanned Safe) – NIST Scorecard  https://fortifyexperts.com/wp-content/uploads/2021/04/CISO-Scorecard-_-Security-Metrics-Approach-Mke-Davis.docx

  1. Risk Metrics:
  • Measuring how many administrators and privileged access management did not meet policy by introducing accountability for role based access.
  • Defining what metrics were acceptable, unacceptable, and what risks we were willing to accept.
  • Measuring compliance and internal audit risks can have value in the right context. Especially, if they are identifying potential operational risks.

KEY POINT:  Aligning your operational metrics to NIST categories can be valuable because it allows the operational metrics to translate easier to your risk metrics.

KEY POINT:  Move towards a risk-based vulnerability model from a “critical” model.

  1. 3rd Party Metrics
  • What metrics can be measured to understand the supply chain especially in the delivery of software in the cloud?
  • How do you measure Trust in your Supply Chain? Can you build in metrics to understand their risks?
  • Measure and track 3rd parties that are critical to the business success, plus track their supply chain as well.
  1. Benchmarking Metrics:
  • Answers, “How are we doing compared to others within the sector?”
  • We never want to be caught behind where the industry is going. We don’t want to look incompetent against our peers.
  • Financial Cybersecurity Profile (FSP) https://fsscc.org/ – Annual evaluation to know how you compare to others. Regulators want to understand if you understand where you stand and where you need to focus attention.

Other Benchmarking Tools:

KEY POINT:  SecurityStudio – Combines CMMC, NIST CSF, PCI into one simple assessment, plus has automated 3rd Party Assessment Tool and a free Employee Assessment and training tool – Request Demo via Fortify Experts

  1. Metrics Reporting Tools:
  • Excel –  is the most popular because of the flexibility but very difficult because you almost have to really be a master at excel and a very manual process.
  • Phishing Tools provide some metrics
  • Ask your MSSP to provide metrics for executive reporting
  • SolarWinds (https://www.solarwinds.com/)
  • RiskLens (https://www.risklens.com/)
  • Archer as a data repository but too complex and not a good analytics tool.
  • OnSpring – GRC Tool (https://onspring.com/)
  • Or created your own data repository and use a BI tool.
  1. ADDITIONAL RESOURCES:

In Summary:

Thanks to the input of the security leaders who joined our CISO Forum which provided such insightful advice to on how to improve your security program.

If you are a security leader and would like to participate in our monthly CISO Forums where we discuss valuable and actionable information as well as best practices and challenges, please register here.

About Tim Howard

Tim Howard is the founder of 4 technology firms including Fortify Experts which helps companies hire the Best Cyber Talent on the Planet as well as provides expert consulting and NIST based security assessments. 

In addition, he has a passion for helping CISO’s develop Higher Performing Teams through coaching, by creating interactive CISO Forums and by helping them create highly-effective team cultures.

With each new hire, his firm produces an Employee Operating Manual to help clients understand how to motivate and maximize productivity while meeting the needs of each employee.

He also teamed up with Lyndrel Downs to launch Cybersecurity DIVAS to help promote the most influential women in cybersecurity and provide a mentoring program to help encourage and support more diversity within the industry.

Tim is married with 3 kids. He is an avid runner and has completed two IRONMAN Texas events. He is also a graduate of Texas A&M University. 

Invite Tim to connect:  www.linkedin.com/in/timhoward