Enterprise

Investing in Distributional

Martin Casado and Matt Bornstein Posted December 14, 2023

Modern AI has allowed us to build the world’s first creative computers. Fields that once seemed uniquely human — like visual art, writing, music, or even the art of conversation — are now accessible to software in ways we couldn’t previously imagine. For many consumer AI apps, this is an incredible and groundbreaking feature. 

For business applications, though, creativity poses a problem. Lack of reliability in AI systems (i.e., their tendency to say things that might offend customers, violate compliance rules, or otherwise damage the business) is one of the biggest barriers to widespread adoption in the enterprise. 

There is probably no closed-form solution to AI reliability. Building perfectly well-behaved applications would defeat the purpose of using AI in the first place (i.e., its creative nature). But it is possible to put bounds around acceptable model outputs, if developers have the right tools and primitives to express their intentions. And it is certainly valuable to at least have a better understanding of whether AI behavior has changed. Building this framework is a massive opportunity. 

That’s why we’re thrilled to announce our seed investment in Distributional. Distributional is building a platform for robust and repeatable testing of AI and machine learning models. The company’s core observation is that rigorous testing, as part of the CI/CD process, has been critical to achieving reliable performance in traditional software development. Good software teams don’t push to prod without passing their build tests first.

The Distributional team believes the same approach will work in AI, but that the right tools don’t yet exist. Traditional testing frameworks only provide static checks — confirming that a piece of code produces the same result every time — while most AI/ ML tools are based on bespoke evaluation functions or manual “vibe checks.” In a sense, Distributional combines these approaches. It provides a standard set of tests (e.g., Python functions) designed to run in the CI/CD pipeline, which work natively with dynamic systems (e.g., checking that code produces results within a certain acceptable range, or distribution, with each run).

The Distributional team has met with more than 25 AI leaders from Fortune 100 companies, AI startups, and financial institutions, and has received early validation on its approach. Each leader identified testing as one of the top barriers to AI adoption within their organization, and many even agreed to become early beta customers, provide ongoing product feedback, or support the company in an effort to help solve the problem. It’s easy to see why users are so eager to find a solution, when you consider what’s at stake: One CTO from a Fortune 100 technology company believes a lack of reliability in AI systems costs their company $10 million a day.

We first met Scott Clark, the CEO of Distributional, back in 2016 through our investment in his prior company, SigOpt. SigOpt was an MLops company in the supervised learning era, and was ultimately acquired by Intel in 2020, where Scott became a VP and GM for AI. He lived the AI-reliability problem every day through the customers he supported at those companies. He’s a zealot for solving hard statistical problems in the real world, and he’s now applying his passion and experience to tackle AI reliability.

We couldn’t be more excited to work again with Scott, Nick Payton, David Rosales, Michael McCourt, and the rest of the team to help make creative computers a bit more well-behaved.

Want More a16z Enterprise?

News and resources for navigating the world of B2B technology, from AI and data, to security and SaaS, and more.

Learn More
Recommended For You
Fintech

new Investing in Glimpse

Joe Schmidt and David Haber
Infra

new Investing in Deeptune

Marco Mascorro and Martin Casado
Growth

Investing in Mind Robotics

Sarah Wang
Growth

Investing in Nexthop AI

Raghu Raghuram, Shangda Xu, and Guido Appenzeller
Fintech

Investing in Lio

Seema Amble, James da Costa, Eric Zhou, and Brian Roberts

Want More Enterprise?

News and resources for navigating the world of B2B technology, from AI and data, to security and SaaS, and more.

Sign Up On Substack

Views expressed in “posts” (including podcasts, videos, and social media) are those of the individual a16z personnel quoted therein and are not the views of a16z Capital Management, L.L.C. (“a16z”) or its respective affiliates. a16z Capital Management is an investment adviser registered with the Securities and Exchange Commission. Registration as an investment adviser does not imply any special skill or training. The posts are not directed to any investors or potential investors, and do not constitute an offer to sell — or a solicitation of an offer to buy — any securities, and may not be used or relied upon in evaluating the merits of any investment.

The contents in here — and available on any associated distribution platforms and any public a16z online social media accounts, platforms, and sites (collectively, “content distribution outlets”) — should not be construed as or relied upon in any manner as investment, legal, tax, or other advice. You should consult your own advisers as to legal, business, tax, and other related matters concerning any investment. Any projections, estimates, forecasts, targets, prospects and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Any charts provided here or on a16z content distribution outlets are for informational purposes only, and should not be relied upon when making any investment decision. Certain information contained in here has been obtained from third-party sources, including from portfolio companies of funds managed by a16z. While taken from sources believed to be reliable, a16z has not independently verified such information and makes no representations about the enduring accuracy of the information or its appropriateness for a given situation. In addition, posts may include third-party advertisements; a16z has not reviewed such advertisements and does not endorse any advertising content contained therein. All content speaks only as of the date indicated.

Under no circumstances should any posts or other information provided on this website — or on associated content distribution outlets — be construed as an offer soliciting the purchase or sale of any security or interest in any pooled investment vehicle sponsored, discussed, or mentioned by a16z personnel. Nor should it be construed as an offer to provide investment advisory services; an offer to invest in an a16z-managed pooled investment vehicle will be made separately and only by means of the confidential offering documents of the specific pooled investment vehicles — which should be read in their entirety, and only to those who, among other requirements, meet certain qualifications under federal securities laws. Such investors, defined as accredited investors and qualified purchasers, are generally deemed capable of evaluating the merits and risks of prospective investments and financial matters.

There can be no assurances that a16z’s investment objectives will be achieved or investment strategies will be successful. Any investment in a vehicle managed by a16z involves a high degree of risk including the risk that the entire amount invested is lost. Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by a16z is available here: https://a16z.com/investments/. Past results of a16z’s investments, pooled investment vehicles, or investment strategies are not necessarily indicative of future results. Excluded from this list are investments (and certain publicly traded cryptocurrencies/ digital assets) for which the issuer has not provided permission for a16z to disclose publicly. As for its investments in any cryptocurrency or token project, a16z is acting in its own financial interest, not necessarily in the interests of other token holders. a16z has no special role in any of these projects or power over their management. a16z does not undertake to continue to have any involvement in these projects other than as an investor and token holder, and other token holders should not expect that it will or rely on it to have any particular involvement.

With respect to funds managed by a16z that are registered in Japan, a16z will provide to any member of the Japanese public a copy of such documents as are required to be made publicly available pursuant to Article 63 of the Financial Instruments and Exchange Act of Japan. Please contact compliance@a16z.com to request such documents.

For other site terms of use, please go here. Additional important information about a16z, including our Form ADV Part 2A Brochure, is available at the SEC’s website: http://www.adviserinfo.sec.gov.