Mesos is an open source project that helps developers manage server clusters more efficiently while providing resource isolation. It’s currently being used by the likes of Twitter and Airbnb, where much of the technology was developed. Google also runs a very similar system called “Borg” – which it developed independently long before Mesos came into existence – to keep its data centers humming.
Mesosphere, an early-stage startup founded by ex-Twitter and Airbnb engineer Florian Leibert and former Airbnb engineer Tobias Knaup, is currently spearheading the development of Mesos. Just like other startups that focus on open source, Mesosphere also hopes to sell additional services in the long run, but as Leibert told me, the focus right now is on building out the Mesos ecosystem and evangelizing the concept.
Mesos, which is now hosted by the Apache Foundation, was born of a research project by Berkeley Ph.D. student Benjamin Hindman who then introduced it to Twitter (where he is still a technical leader).
The idea behind Mesos turns out to be pretty straightforward, though it went out of fashion some time ago: instead of setting up numerous server clusters for different parts of an application, Mesos allows you to share a pool of servers that can all run different parts of your application (say your Hadoop big-data cluster, your web server and other services) without them interfering with each other and with the ability to dynamically allocate resources across the cluster as needed. This means you could easily switch resources away from your big-data analysis and allocate them to your web server if you’re getting hit with a large amount of traffic, for example.
The high-level idea here, as Leibert explained to me, is that Mesos is essentially the kernel of an operating system that looks at the data center as a multi-core CPU. That operating system needs more than just a kernel, though, and all the components around it is what Mesosphere is now focusing on. At AirBnB, Leiber also built the Chronos scheduler, a cron replacement for automatically starting and stopping services (and handling failures) that runs on top of Mesos. The other part of the Mesos puzzle so far is Marathon, which Leibert described to me as the equivalent of init.d – the first process that’s started on a Unix system – for the Mesos ecosystem. Marathon provides the Mesos framework with an API for starting, stopping and scaling services (and Chronos could be one of those services).
Using Chronos and Marathon together, the team announced today, Mesos can now also run and manage Docker containers. Docker allows developers to automate the deployment of their applications by packaging them into lightweight, portable containers that can then be deployed on new machines. In the context of Mesos, that means it’s now much easier to deploy applications on the cluster. “Delivering applications with Docker on Mesos,” the team writes in a blog post today, “promises a truly elastic, efficient and consistent platform for delivering a range of applications on premises or in the cloud.” Docker support, Leibert told me, will also make service discovery across the cluster easier. After all, it’s not enough to just set up the cluster, but the different servers also need to be able to find each other.
Adding Docker support to Mesos, Leibert told me, is something others have tried before, but it’s actually very hard to do. Thanks to its intimate knowledge of Mesos, however, the six-person team at Mesosphere managed to integrate the system.
With these building blocks in place, Leibert said, the team now also plans to add some user-friendly interfaces to its framework to make it a bit more accessible to new users. Mostly, however, Mesosphere plans to listen to the Mesos community and then add the components its users are asking for.
No comments:
Post a Comment