Apache Mesos users focus on big data, containers

Mesosphere, the main commercial outfit behind the Apache Mesos datacenter and container orchestration project, has taken a good look at its user base and found that they gravitate toward a few fundamental use cases.

Survey data released recently by Mesosphere in the “Apache Mesos 2016 Survey Report,” indicates that Mesos users focus on running containers at scale, using Mesos to deploy big data frameworks, and relying heavily on the core tool set that Mesos and DC/OS provide rather than using substitutes.

We got this contained

Created in 2009, Mesos was built to run workloads of all types and sizes across clusters of systems. DC/OS, released back in 2015 by Mesosphere, automates the deployment, provisioning, and scaling of applications with Mesos as the underlying technology. Thus, it casts Mesos as a commodity similar to Docker, which offers ease in working with long-standing containerization techniques.

The Mesosphere survey doesn’t cover a very large sample of users — fewer than 500, with 63 percent of those surveyed running Mesos for less than a year. Deployments are also modest — the overwhelming majority are fewer than 100 nodes — and by and large favor generic software/IT industry settings. Retail, e-commerce, telecom, and finance made up about 19 percent of the total combined.

Among the workloads deployed in Mesos, the largest slice (85 percent) covers containers and microservices, with 62 percent of all users deploying containers in production. Containers have long been a major part of Mesos’ and DC/OS’s focus, but Mesos sets itself apart from other container projects by providing a robust solution to container management, including native support for GPU-powered applications.

Do it yourself

The second biggest slice of the pie is data-centric applications. No prizes for guessing the top entry in that category: Apache Spark (43 percent of users), followed by other major big data infrastructure components like the Kafka messaging system (32 percent), the Elastic search system (26 percent), and the Cassandra NoSQL database (24 percent). Hadoop is in the mix as well, but only at 11 percent.

If there’s a takeaway to be found, it’s that specific solutions like Spark demonstrate more immediate payoffs than general solutions like Hadoop, especially when projects like DC/OS make them easier to deploy.

The survey also makes clear that Mesos users have historically put together projects themselves, but they like the idea of having the option to not have to. Of those who use Mesos, few currently do so with DC/OS’s automated deployment. Only 26 percent of those surveyed are running it in a production context, with another 12 percent “piloting for broader deployment.” That implies that many existing Mesos-powered deployments are hand-built.

However, newly minted Mesos users are going straight to DC/OS to get their Mesos fix. Eighty-seven percent of users who started with Mesos in the past six months did so through DC/OS. Thus, it’s safe to assume as DC/OS becomes more widely used and Mesos continues to evolve (it recently hit a 1.0 release), DC/OS will become the predominant preference to deploy both Mesos and all the apps that run with it.

It’s important to think about Mesos and DC/OS as complementary technologies to the rest of the container world, not total replacements for it. Kubernetes, for instance, can run in Mesos (and 8 percent of the respondents do use Kubernetes somewhere, according to the survey). Rather than eclipsing such arrangements outright, it’s more likely that DC/OS and Mesos will provide a more convenient option to build with them.

[Edited to clarify Kubernetes use amongst the survey respondents.]