Mesosphere co-founder and CEO Florian Leibert joined InApps founder Alex Williams and co-host Donnie Berkholz of 451 Research for this edition of InApps Analysts podcast, an engaging and informative discussion about the virtues of Apache Mesos, and the concepts and tools involved with architecting big data.
For more episodes, check out the podcast section of InApps.
#48: Mesosphere’s Florian Leibert on the New Data Center OS
Listen to all TNS podcasts on Simplecast.
This podcast is also available on YouTube.
Leibert joined Alex and Donnie to talk about the general availability of the Mesosphere Datacenter Operating System (DCOS), announced earlier in the week, and generally to discuss a new thinking about the data center. He describes these giant systems as a new form factor that is transforming data center operators into what he describes as “framework developers.”
Alex recalls a recent conversation with Sheng Liang of Rancher Labs, during which Sheng observed that while Docker, through its user interface, has done a great job of making things accessible and enjoyable for the developer, there remains a need for “purpose-built infrastructure” for these new microservices environments.
“What we view as some of the key components of that,” says Florian, “are things in the realm of big data: continuous integration, continuous delivery, NoSQL or SQL — basically all the stateful applications — and then of course other infrastructure that’s essential, like Kafka, which is a message queue. There are a lot of pieces in the stack.” He notes that many of these processes get packaged into containers and integrated with the ecosystem in a manner that is similar to converged infrastructure.
“They are really microservices themselves that you’re integrating with one another,” he continues. “For us, that was really big opportunity when we launched DCOS, to showcase that you can start a Marathon — our take on a PaaS — and a Jenkins, which is used for continuous integration, and then tie them together for continuous delivery, and on the same cluster you can run your Spark jobs, your Hadoop jobs, and your NoSQL services like Cassandra, or even things like MySQL.”
“You run them all on a shared environment,” he continues, “and you get to solve the provisioning problem in doing so. You can give each of these services a different priority, then they can self-recover in case there’s an outage in your data center.”
“On top of that,” he says, “it makes it much easier to set up this pretty complex system.”
“Fifteen years ago I used to do high-performance computing stuff,” says Donnie, “and the sense I get, looking at what’s happening here, is that we’re kind of reinventing all of that for a different user base. Instead of for scientists written in Fortran, we’re doing the same thing for app developers and data engineers written in whatever is the new language-of-the-day.”
Later in the show, while discussing the Hadoop ecosystem, Donnie refers to his recent research which charted the quick adoption of Spark, and offers his theories on why it’s become so popular for processing large datasets.
Meanwhile, Florian traces his own path toward Mesosphere through his career, beginning with his first exposure to Mesos, when the project was called Nexus while in development at UC Berkeley, at what’s now known as the AMPLab. From working as a software engineer and applications developer at Twitter, to helping to scale Airbnb, Florian found that the Mesos project consistently fit the bill for solving big problems.
“You have an isolation mechanism. You have a mechanism to send messages across all of the hosts that partake in your cluster. You have things like an abstraction for state, so that you can store your working set in memory,” says Florian. He also makes a case for the level of automation it provides.
“It’s not just for one use case,” Florian says. “Any sort of distributed system that you want to build, you can leverage Mesos to get the job done much quicker, and with much less effort than if you were to just use plain sockets, if you will.”
Docker is a sponsor of InApps.
Featured image: Gustave Dore, detail from “The Creation of Light” (1866), via WikiArt.org.
InApps is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker.
