Enterprise

Blueprints for data infrastructure; “lakehouse” storage; data is not the new oil, more

a16z editorial Posted October 27, 2020

Blueprints for data infrastructure; “lakehouse” storage; data is not the new oil, more Table of Contents

This first appeared in the monthly a16z enterprise newsletter. Subscribe to stay on top of the latest in enterprise and B2B.

As much as we hear about artificial intelligence and machine learning, they are just one facet of an even bigger trend within the enterprise: the shift from software systems based on code to those based on data. Here we focus on data and round up our most important stories and pods on the ways it is changing the enterprise, from new technology stacks and business models to who gets to use the data and how.

The blueprints for data infrastructure

The emergence of AI/ML has added complexity to data infrastructure — the tools that centralize and process data — and created two parallel ecosystems. The first is for operational systems (including AI/ML) that power data-driven products, and the second is for traditional analytic systems (including business intelligence tools). So, how does all the innovation in data tools and technology around these two ecosystems come together in the enterprise?

We asked 20+ practitioners from leading data organizations: (a) what their internal technology stacks looked like, and (b) whether it would differ if they were to build a new one from scratch today. The result: a unified architecture for a modern data infrastructure and three common blueprints for implementing it.

Unified Data Architecture

Read the full report »

A new way to store data — the “lakehouse”

Many enterprises have two separate types of storage: data lakes and data warehouses. The data warehouse handles “offline” analytics, such as BI dashboards and reports, that describe what has or is happening in a business. And data lakes store messier, usually unstructured data, which powers “online” models for data science and the computations behind the models that run the business (for example, when Lyft or Uber set prices on rides).

But as data lakes mature, they are more capable of handling BI and analytics use cases directly. Could the future be a data “lakehouse” that provides a single storage platform for both business intelligence and data science? Ali Ghodsi, CEO and founder of Databricks, explores the history of data architectures and where they may be headed in this hallway-style conversation with a16z general partner Martin Casado.

Listen to “Data is Not Enough: The Evolution of Data Architectures” »

Data & the problem of complex systems

In a code-based system, the goal is to construct software that, given certain inputs, produces a certain set of outputs. The task is fundamentally an engineering one: you can modularize, you can build integrations, you can control all the primitives. But data systems, especially AI/ML systems, are more akin to metaphysics than engineering, in their attempt to model and rein in the complexity of the natural world.

Or so argue Peter Wang, founder and CEO of Anaconda, and Martin Casado, who discuss the practical implications for the margins, organizational structures, and building of AI/ML businesses in this episode of the a16z Podcast.

Listen to “Reining in Complexity: Data Science & Future of AI/ML Businesses” »

Why data is not the new oil

Once oil is extracted, it can’t be extracted again, and it holds value over time (market changes aside). Data, on the other hand, can be extracted endlessly; typically becomes less valuable over time; and, as a data corpus grows, the cost of acquiring more data goes up as the incremental value of adding data goes down.

That isn’t to say that data isn’t valuable. Data is fundamental to many companies’ product strategies, and there are ways it can contribute to defensibility — but it’s not a magical moat. Long-term defensibility is more likely to come from differentiated technology and domain expertise, with data a key fuel for both.

Read “The Empty Promise of Data Moats” »

Data-driven decisions when everyone is an analyst

As more industries and organizations move from manual paper-based processes to automated digital processes (buzzword bingo: digital transformation), business intelligence (BI) is giving way to operational analytics (OA). With BI, a team of specialists creates dashboards that executives and managers then use to understand a business’s past performance and make high-level decisions. In operational analytics, everyone becomes an analyst, accessing near real time data for more operational decisions and opening new opportunities for operational analytics infrastructure, industry-focused applications, and role-based tools. So what does that mean for workers and the enterprise?

Watch “Everyone is an Analyst: Opportunities in Operational Analytics” »

Want More a16z Enterprise?

News and resources for navigating the world of B2B technology, from AI and data, to security and SaaS, and more.

Learn More

Want More Enterprise?

News and resources for navigating the world of B2B technology, from AI and data, to security and SaaS, and more.

Sign Up On Substack

Views expressed in “posts” (including podcasts, videos, and social media) are those of the individual a16z personnel quoted therein and are not the views of a16z Capital Management, L.L.C. (“a16z”) or its respective affiliates. a16z Capital Management is an investment adviser registered with the Securities and Exchange Commission. Registration as an investment adviser does not imply any special skill or training. The posts are not directed to any investors or potential investors, and do not constitute an offer to sell — or a solicitation of an offer to buy — any securities, and may not be used or relied upon in evaluating the merits of any investment.

The contents in here — and available on any associated distribution platforms and any public a16z online social media accounts, platforms, and sites (collectively, “content distribution outlets”) — should not be construed as or relied upon in any manner as investment, legal, tax, or other advice. You should consult your own advisers as to legal, business, tax, and other related matters concerning any investment. Any projections, estimates, forecasts, targets, prospects and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Any charts provided here or on a16z content distribution outlets are for informational purposes only, and should not be relied upon when making any investment decision. Certain information contained in here has been obtained from third-party sources, including from portfolio companies of funds managed by a16z. While taken from sources believed to be reliable, a16z has not independently verified such information and makes no representations about the enduring accuracy of the information or its appropriateness for a given situation. In addition, posts may include third-party advertisements; a16z has not reviewed such advertisements and does not endorse any advertising content contained therein. All content speaks only as of the date indicated.

Under no circumstances should any posts or other information provided on this website — or on associated content distribution outlets — be construed as an offer soliciting the purchase or sale of any security or interest in any pooled investment vehicle sponsored, discussed, or mentioned by a16z personnel. Nor should it be construed as an offer to provide investment advisory services; an offer to invest in an a16z-managed pooled investment vehicle will be made separately and only by means of the confidential offering documents of the specific pooled investment vehicles — which should be read in their entirety, and only to those who, among other requirements, meet certain qualifications under federal securities laws. Such investors, defined as accredited investors and qualified purchasers, are generally deemed capable of evaluating the merits and risks of prospective investments and financial matters.

There can be no assurances that a16z’s investment objectives will be achieved or investment strategies will be successful. Any investment in a vehicle managed by a16z involves a high degree of risk including the risk that the entire amount invested is lost. Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by a16z is available here: https://a16z.com/investments/. Past results of a16z’s investments, pooled investment vehicles, or investment strategies are not necessarily indicative of future results. Excluded from this list are investments (and certain publicly traded cryptocurrencies/ digital assets) for which the issuer has not provided permission for a16z to disclose publicly. As for its investments in any cryptocurrency or token project, a16z is acting in its own financial interest, not necessarily in the interests of other token holders. a16z has no special role in any of these projects or power over their management. a16z does not undertake to continue to have any involvement in these projects other than as an investor and token holder, and other token holders should not expect that it will or rely on it to have any particular involvement.

With respect to funds managed by a16z that are registered in Japan, a16z will provide to any member of the Japanese public a copy of such documents as are required to be made publicly available pursuant to Article 63 of the Financial Instruments and Exchange Act of Japan. Please contact compliance@a16z.com to request such documents.

For other site terms of use, please go here. Additional important information about a16z, including our Form ADV Part 2A Brochure, is available at the SEC’s website: http://www.adviserinfo.sec.gov.