A new architectural era in computing is upon us, and the datacenter is changing to accommodate it. The cloud generation of companies has ramped their dominance and proven their models, and the legacy enterprise is close behind in making this massive shift. These new datacenters—as pioneered and designed by Facebook, Google, and Twitter—are defined by hyper-scale deployments of thousands of servers, requiring a new software architecture to manage and aggregate these systems. Mesosphere is that software, and we believe this architecture will be as disruptive to the datacenter as Linux and virtualization have been over the past decade.
Today’s application architectures and big data workloads are scale-out, stateless, and built to leverage the seemingly infinite processing capacity of the modern datacenters. These modern hyper-scale datacenters are the equivalent of giant supercomputers: they run massively parallel applications that serve millions of user requests a second. We are moving from a collection of servers running discrete, stateful applications, to massive scale-out applications that treat the hardware as one giant server.
In that “giant server” view of the world, Mesosphere is the obvious foundation for this new cloud stack and adoption is scaling fast. Look under the datacenter hood in many forward-looking, hyper-scale environments, including Twitter, Airbnb, eBay, and OpenTable, and you will find Mesosphere.
The Future of the Datacenter is Aggregation (not Virtualizaton)
Ten years ago, virtual machines (VMs) revolutionized the datacenter. This was because while the servers were getting bigger and bigger, the apps running on them pretty much stayed the same size. In order to make better use of those large servers, it made sense to virtualize the machines so that you could run multiple applications on the same machine at the same time.
Today, aggregation is fomenting a similar revolution and applications don’t fit on single machines anymore. In today’s world, applications run at a much larger scale (millions of users, billions of data points, and in real-time) and they are essentially large-scale distributed systems, composed of dozens (or even thousands) of services running across all the machines (virtual and physical) in the datacenter. In this world, you want to stitch together all of the resources on those machines into one common pool from which all the applications and services can draw.
Aggregation has proven itself in the A-lists of hyperscale companies, like Google and Twitter. They’ve demonstrated that it’s much more efficient to aggregate machines—pooling all of the resources—and then build applications against the datacenter behaving as a single machine.
Aggregation, and the tools to manage it at scale, is what Mesosphere is bringing to everybody —and it’s what we believe the future of the datacenter looks like.
The companies that buy into this architecture do not abandon virtualization, containers, or other approaches. These become important infrastructure components. But the way they manage their entire datacenter will evolve beyond the duct tape and band aid, highly manual approach to scripting IT operations tasks and “recipes”, and configuring dependencies each time a new application is brought online or a server goes down.
Mesos: From UC Berkeley to Reality
In 2009, Mesosphere Co-founder Florian Leibert was working at Twitter to scale the application in response to its exponential growth. At the time, he spotted a new open source technology that had been built at UC Berkeley called Mesos and he helped Twitter bring it into full production.
Today, almost all of Twitter’s infrastructure is built on top of Mesos, which is now an Apache open source project and is at the core of Mesosphere’s products. The Mesosphere stack, which includes Apache Mesos, is not a hypothetical technology. It’s highly mature and battle-tested, in large-scale production, running in both private datacenters and in public cloud environments. Other organizations using Mesos include: Hubspot, Airbnb, Atlassian, eBay, OpenTable, PayPal, Shopify, and Netflix.
Mesosphere is harnessing the core open source technology of Apache Mesos, and making it possible for everyone to tap into its power. By building an entire ecosystem around Mesos, they are making it easy to install, operate, and manage. For developers, Mesosphere provides simple command-line and API access to compute clusters for deploying and scaling applications, without relying on IT operations. For IT operations, Mesos abstracts the most difficult low-level tasks related to deploying and managing services, virtual machines, and containers in scale-out cloud and datacenter environments, and provides true automation, fault tolerance, and server utilization for modern scale requirements. Finally, Mesos allows applications to move between different environments without any change to the application.
Mesosphere will help define the next generation datacenter. I am honored to be joining the board of a team of dedicated system-level software engineers who will change the face of enterprise computing.
The views expressed here are those of the individual AH Capital Management, L.L.C. (“a16z”) personnel quoted and are not the views of a16z or its affiliates. Certain information contained in here has been obtained from third-party sources, including from portfolio companies of funds managed by a16z. While taken from sources believed to be reliable, a16z has not independently verified such information and makes no representations about the enduring accuracy of the information or its appropriateness for a given situation.
This content is provided for informational purposes only, and should not be relied upon as legal, business, investment, or tax advice. You should consult your own advisers as to those matters. References to any securities or digital assets are for illustrative purposes only, and do not constitute an investment recommendation or offer to provide investment advisory services. Furthermore, this content is not directed at nor intended for use by any investors or prospective investors, and may not under any circumstances be relied upon when making a decision to invest in any fund managed by a16z. (An offering to invest in an a16z fund will be made only by the private placement memorandum, subscription agreement, and other relevant documentation of any such fund and should be read in their entirety.) Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z, and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by Andreessen Horowitz (excluding investments and certain publicly traded cryptocurrencies/ digital assets for which the issuer has not provided permission for a16z to disclose publicly) is available at https://a16z.com/investments/.
Charts and graphs provided within are for informational purposes solely and should not be relied upon when making any investment decision. Past performance is not indicative of future results. The content speaks only as of the date indicated. Any projections, estimates, forecasts, targets, prospects, and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Please see https://a16z.com/disclosures for additional important information.