Cybersecurity In-Depth: Feature articles on security strategy, latest trends, and people to know.
8 AI Risk and Resilience Firms CISOs Should Track
Check out our list of emerging firms that are building technology and services to assess the risk posture of AI systems and ML models.
August 17, 2023
concept art showing white colored gears against a blue and black backgroundSource: Blue Planet Studio via Adobe Stock Photo
As enterprise use cases of artificial intelligence (AI) grow beyond research edge cases, cybersecurity leaders are increasingly being called to manage a new layer of risk in their technology stack. CISOs need to help their organizations account for new attack vectors, such as adversarial AI attacks like model inversion attacks and data poisoning.But that’s just a slim picture of the risks. Resilience, reliability, and trust issues like model brittleness, AI bias, and explainability are all increasingly important factors to manage.
Additionally, AI further exacerbates software supply chain issues, as open sourced AI models and training data are de rigueur for building these systems. Given the situation, new vendors, advisers, and consultants have a greenfield opportunity to help organizations manage these risks. The following are some of the early contenders in what is sure to be a growing new niche of AI risk and resilience technology and services.
Billing itself as a machine learning detection and response (MLDR) vendor, HiddenLayer offers a platform that monitors the inputs and outputs of machine learning (NL) algorithms to look for adversarial ML attack techniques mapped to the MITRE ATLAS framework. Founded by a team of former Cylance threat researchers — Chris Sestito, James Ballard, and Tanner Burns — the firm came out of stealth mode last year blazing a trail with $6 million in seed funding that helped it prove out its technology well enough to win the RSA Conference 2023 Sandbox Innovation contest and start gaining some early momentum.
Founded: 2022
Founders: Chris Sestito, James Ballard, Tanner Burns
CalypsoAI recently shifted its main focus to large language model (LLM) risk management for sensitive industries — such as defense, finance, and pharmaceuticals — and technologies that are seeking ways to safely ride the wave of ChatGPT and LLM innovation. The foundation of the firm's approach is built on its CalypsoAI Moderator technology, which is built to help firms with data loss prevention for sensitive information and intellectual property, jailbreak prevention, and blocking malicious code from executing through LLM use. At the same time, the company is still supporting its earliest product, VESPR Validate, an ML model validation tool that's designed to stress test ML models for mission-critical applications — including defense imaging models — against a range of conditions. Founded in 2018, the firm posted a $23 million Series A-1 round in June, led by Paladin Capital Group.
Founded: 2018
Founder: Neil Serebryany
One of the more mature and well-funded startups in this extremely formative field, Robust Intelligence was founded in 2019 by a superstar cast of ML and data science luminaries — Alexander Rilee, Eric Balkanski, Kojin Oshiba, and Yaron Singer — to come up with technology to stress test and protect AI/ML models in production. The heart of the firm's technology lies in its platform. The platform includes an AI firewall that protects models against undesirable outputs in real time, as well as continuous validation checking technology that ensures the models meet AI risk standards predeployment and monitors them for anomalies once they go into production. The ethos of Robust Intelligence is to assess not just for cybersecurity risks, but also ethical and operational risks that would make a model less trustworthy or reliable. This firm is also the braintrust that came up with the AI Risk Database, a new community platform the company is trying to fashion as the "VirusTotal for AI."
Founded: 2019
Founders: Alexander Rilee, Eric Balkanski, Kojin Oshiba, Yaron Singer
Founded in 2022 by Amazon Web Services' former worldwide leader in AI/ML, Ian Swanson, Protect AI is a new contender in the AI risk management space that just this month earned a $35 million Series A funding round from Evolution Equity Partners, following a December 2022 seed round of $13.5 million. This summer the firm also brought in Diana Kelley to serve as the firm's CISO. A veteran CISO and security advisor, Kelley brings decades of security leadership experience that includes stints at IBM and Microsoft. The firm's early products are AI Radar and NB Defense. Similar to others in this space, AI Radar is designed to give organizations visibility into their ML attack surface and detect risks and threats to ML systems and AI applications. The unique value proposition of this product is its ability to create a dynamic ML Bill of Materials, which provides a list of components and dependencies within an ML system. Meantime, NB Defense is a free Jupyter Notebooks scanning tool that's designed to help address vulnerabilities in the earliest stages of ML experimentation.
Founded: 2022
Founder: Ian Swanson
A seed stage company based out of Canada, TrojAI is focused on building out a portfolio of model stress test, model risk audit, LLM protection, and AI firewall solutions and services for enterprises. Founded in 2019, the firm is led by Dr. James Stewart, a data science and cybersecurity pro who was previously lead data scientist for Forcepoint and most recently successfully founded and sold AI firm EhEye to Patriot One Technology, which operates in the automated video surveillance and analytics space. The firm picked up $2.4 million in seed funds last year, led by Flying Fish Ventures.
Founded: 2019
Founder: James Stewart
Rather than taking a product focus, Adversa AI is putting a stake in the ground in the AI assurance and assessment services space. This bootstrapped firm is building its consulting base by educating its customers' technical and leadership teams about AI risks, leading AI strategy development, and performing a range of threat modeling and ethical hacking checks against enterprise AI/ML environments. Adversa has gained a lot of early attention from its growing body of AI threat research led by the Adversa AI Red Team, which has published research on a range AI jailbreaks, attacks, and vulnerabilities in the past couple of years.
Founded: 2021
Founder: Alex Polyakov
Founded in 2020 and based out of Israel, DeepKeep offers its MLProtect platform, which was built to act as a risk management and protection hub serving data scientists, MLOps, and cybersecurity teams. The platform automates model pen tests and provides ways to optimize AI security policies, increase model explainability, and monitor models in production. The firm is led by founder and CEO Rony Ohayon, a serial entrepreneur who previously founded DriveU Tech, an autonomous vehicle firm. Awz Ventures has invested approximately $5.45 million in seed funds into the firm, and DeepKeep is in a two-year residence with the Awz X-Seed Hub, an early-stage acceleration initiative.
Founded: 2021
Founder: Rony Ohayon
A spin-out company from KPMG startup incubator KPMG Studio, Cranium launched in April as an AI risk firm focused heavily on helping enterprises manage rapidly changing compliance obligations around AI systems. The firm's AI Card technology supports compliance mapping to frameworks including NIST AI RMF and EU AI Act, and offers risk assessment visualizations that make it easier for decision-makers and auditors to understand the risk levels of an organization's model portfolio.
Founded: 2022
Founders: Felix Knoll, Jonathan Dambrot, Paul Spicer
A spin-out company from KPMG startup incubator KPMG Studio, Cranium launched in April as an AI risk firm focused heavily on helping enterprises manage rapidly changing compliance obligations around AI systems. The firm's AI Card technology supports compliance mapping to frameworks including NIST AI RMF and EU AI Act, and offers risk assessment visualizations that make it easier for decision-makers and auditors to understand the risk levels of an organization's model portfolio.
Founded: 2022
Founders: Felix Knoll, Jonathan Dambrot, Paul Spicer
As enterprise use cases of artificial intelligence (AI) grow beyond research edge cases, cybersecurity leaders are increasingly being called to manage a new layer of risk in their technology stack. CISOs need to help their organizations account for new attack vectors, such as adversarial AI attacks like model inversion attacks and data poisoning.But that’s just a slim picture of the risks. Resilience, reliability, and trust issues like model brittleness, AI bias, and explainability are all increasingly important factors to manage.
Additionally, AI further exacerbates software supply chain issues, as open sourced AI models and training data are de rigueur for building these systems. Given the situation, new vendors, advisers, and consultants have a greenfield opportunity to help organizations manage these risks. The following are some of the early contenders in what is sure to be a growing new niche of AI risk and resilience technology and services.
About the Author(s)
You May Also Like
CISO Perspectives: How to make AI an Accelerator, Not a Blocker
August 20, 2024Securing Your Cloud Assets
August 27, 2024