Our hosting stack from the bottom up - Part 3
Language
English
Our hosting stack from the bottom up - Part 3
Once you clear the hardware level, your choice of operating system is the base of everything else. It is essential your OS of choice is robust and secure, you don't want to build your stack on sand.
Thu, 2015-11-05 15:54By greg
We talked in previous posts about our infrastructure and our virtualisation. This time it's the turn of our operating system.
The first point to state is that we use Linux as the base for our software stack. This may seem obvious to the seasoned PHP developer, but it's by no means a hard requirement these days. We know plenty of organisations that are almost entirely Windows and run so-called LAMP applications (Linux Apache MySQL PHP) on a WAMP (Windows Apache MySQL PHP) stack without really encountering any great issues.
But we like Linux for a number of reasons. Firstly, like Drupal, it's free. No license fees, no costs at all beyond your hardware and your time.
Secondly, again like Drupal, there are many, many ways to solve the same problem with the toolset available. There are a huge number of available applications, almost all free (though ocassionally non-free). The amount of choice is just amazing, I remember as a Windows user getting my head around the concept of having a repository of 20,000 absolutely free programs to install and try. No going to different websites to find things, no trial periods, no locked features. Just. There.
Finally, while Drupal can run on a WAMP stack, the reality is PHP and MySQL, the critical applications behind Drupal, are written in C, which is native to *nix like systems, of which Linux is the most popular when it comes to running servers. Windows might still have around half of the desktop market, and OS X might be the most popular *nix like desktop system by a mile, but nothing touches Linux in serverland. So if you follow the logic through, Linux is the sensible choice.
(Interestingly, while it might not be relevant to this post, nothing touches Linux in mobile device land either, where Android is absolutely top dog - and yup, you guess it - it's Linux.)
But the problem with Linux is there are dozens (if not hundreds) of different versions of Linux out there. (This mind-blowing family tree gives you some idea of the scale of Linux.) They all have the same core, the Linux kernel, at their heart, but the way they organise themselves, the packages they include, their popularity, their policies, their financial support structures, are all totally different. So choosing a distribution is tough.
At the base there are three main branches of Linux distributions, and they are Red Hat, Slackware and Debian. All three are almost as old as Linus Torvald's original Linux kernel itself, which was released in 1991. Debian started in September 1993, Slackware began a little earlier in April 1993, and Red Hat put out their first release just a few short years later, in 1995. Philosophically Debian and Slackware are more similar, focussing on being purist, free, secure and stable. Red Hat have a different model - they decided to monetise the Linux kernel and their operating system, Red Hat Enterprise Linux (RHEL), is a private, paid-for software product like Windows. However, unlike Windows, all the source code for RHEL is freely available online, if you can be bothered to compile it, which is why the many Red Hat based distributions out there like the CentOS project - which is basically RHEL for free - can exist.
(By way of an interesting aside, Apple's OS X is a fork of the BSD branch of *nix like operating systems and they open source their kernel and base system, in the Darwin project, and use it as an incubator in much the same way as Fedora and Red Hat maintain a close relationship. The key difference being that while Red Hat open source all their code, Apple keep the UI and many libraries that are integral to OS X very much under wraps.)
Generally speaking it's smart to stay on or close to one of these popular main lines, because everything else is derivative, so the main lines tend to be the most stable. Fedora, the distribution I use for my laptop, is derived from Red Hat (RHEL), for example, but it is only one step removed from the main line, and the two have a "special relationship", as mentioned above. Two very popular distributions on the Debian side (in fact, I think the two most popular Linux distributions) are Ubuntu and Mint. Ubuntu is a Debian derivative, one step from the main line like Fedora (though with no real connection to Debian) whereas Mint is derived from Ubuntu - further from the trunk and, consequently, potentially slower to receive updates and likely to be using less mature additional packages. I'm less familiar with the Slackware line of the family, but you get the point.
At Code Enigma, we chose to stay close to the source. We chose Debian.
I mentioned Miguel Jacq in my last post, the Australian systems administrator who has been instrumental in setting up our systems and continues to guide large parts of our service development. Well, hand on heart, five years ago at the start of all this if someone asked me "Why Debian?" I'd say "Because Mig said so." But as time has gone by I've come to appreciate some of the features and philosophies of Debian, and why they mean tangible benefits when it comes to looking after hundreds of servers, which helps me to answer the question "Why Debian?" with far more verbosity. I also asked Mig to give me some headlines of his own, here's what he said:
Philosophy
Debian is about proper 'free as in freedom' opensource philosophy and so avoids big binary blobs where possible. Even just this week, Elasticsearch was pulled from Debian Jessie because the upstream maintainers stopped shipping details on specific security issues, meaning individual security patches can't be backported. This doesn't suit the Debian philosophy of being forced to upgrade, so support was pulled. This is a hardline approach but it speaks of gung-ho and commitment which matches Code Enigma's philosophy to building tools and supporting its customers.
Quite so! And indeed, I find Debian philosophically very similar to Drupal in that regard. In Drupal we also have strict packaging rules and we police each other to make sure everyone toes the line and our values are not eroded over time. So that feels like a familiar thing to us Drupal developers.
Of course, while the software included in Debian is free and released under the same license, there is a flip-side to that. As the Tech Republic article "Why aren't more people using Debian?" points out, things like proprietary driver support are quite deliberately excluded from Debian, for purely philosophical reasons. But that's absolutely fine with the Debian community. It's not like you can't get this software for Debian. If you need it, compile it yourself. Or find someone else who has a software repository who has compiled it for you (see DotDeb, for example). Or use Ubuntu or Mint instead, who are less picky about what they include in their mainline repositories (and whether it works for everyone or not).
Plus in our case there's an element of "whatever". We're running servers here - who cares about Intel's wifi drivers?
It's also worth noting that Debian has a strong "social contract" to go alongside the technical policy (which we talk about next). This is important, because it provides the mirror Debian developers and users can hold up to themselves, and the code they're working on. It is the consitution of Debian. Everything comes back to this. Every good project needs a good constitution.
Stability
A guiding principle of Debian is never to be on the 'latest' version of software just for the sake of being hyperactive about features. Tried and tested tools are the priority, even if it means they lack features. 'Old and Boring' wins where possible. We have enough skill ourselves to build custom Debian packages in our own APT repository if we need special sauce. I have been bitten in the past with regressions in Ubuntu for example (I have in Debian too, but far less often). The end result is a happy customer with a nice boring stable platform.
We can delve into that even further. At the heart of everything great about Debian is their policy. They've put a LOT of thought into their policy and it's the most comprehensive one I've seen. I doubt any other Linux distro has given it as much thought as the good people of Debian have. Packaging software for the Debian project is strictly controlled, but in a good way. Anyone can do it, and the rules are clear, but they must also be strictly followed. Policy dictates QA, licensing decisions, code style, file system structure, config locations, everything has a rule and there's a rule for everything. Which makes it much easier to ensure you continue to produce a very stable Linux distribution.
They also adopt a "ship when it's ready" approach to release management, again, something we in Drupal can appreciate. While projects like Ubuntu deliberately ship "almost working" software and do a Long Term Stable (LTS) release every four years or so that is more robust, Debian only make releases they feel they can hang their hat on. It's ready when it's ready. It's ready when we can roll it out to every single Debian user on the planet, they can upgrade, and we're sure it'll work for them.
Maturity
Debian is one of the oldest and most widely used distributions around, even by implication, since it is the base for so many other distributions (including Ubuntu) that have come since. A product can only last this long in tech if it improves on itself enough to stay relevant and popular. 10 million people can't be wrong etc etc.
Quite so, every time you Google the "Why Debian?" question you come up with threads like this one. Debian is bedrock in the Linux world.
And it's growing fast! 10 years ago Red Hat ruled the roost and Debian were scraping the bottom of the popularity charts. But nowadays, according to DistroWatch, they are the third most popular distro and have the second most popular website. And if you look at web server statistics, it's an even better story, with Debian topping the charts in some surveys.
And just because...
I simply hate Red Hat based distributions and Yum/RPM package management.
Fair enough.
Aside from all that, I'd like to add a few bullets of my own, which I believe are important:
Security
Like Drupal, Debian has a dedicated security team and a dedicated bug tracking service. The project is highly organised, critical bugs are fixed in a matter of days, often faster.
Our own Debian experiences, with major vulnerabilities like Heartbleed and POODLE breaking in recent years, and using a range of different distributions in the business, tell us Debian is usually a day or so ahead of "the competition" when it comes to patching major vulnerabilities. They've always beaten Red Hat to the punch for as long as we can remember, though I couldn't find any actual data to support that claim, as a Fedora user I know I always have to wait longer for a fix via the package manager than our Debian servers do.
Community
The Debian community is really great. There are over 2,000 registered developers (and many more contributors besides, including our very own Adrian Alves, our man in Argentina). Over 350 registered using organisations from across the globe (including Code Enigma) are proud to back this distro, and that's just the ones who have registered. There's a big annual conference too, with all the trimmings and over 500 people this year.
It also has truly global coverage. I've heard said it is "the most cosmopolitan Linux distro", and having a round-the-world presence means having a round-the-clock, community-driven support structure. Things are fixed rapidly, mailing lists are buzzing 24/7, questions are answered quickly, it's an always on world.
There's also a very high developer-to-package ratio, there are far more active Debian developers than most other distros, thanks to the awesome community and inclusive policies.
In summary, I'm going to quote from a quote from an essay I read on this subject:
There is no other OS or distribution that I know of which has just this mix of properties (ease of maintenance, affordability, stability, size, customizability, strong support). For the most part, I do not want to tinker with and Debug my workstation, I want to get my job done, easily, safely, and with minimal concern about the infrastructure I use. Debian helps me accomplish that.
https://people.debian.org/~srivasta/talks/why_debian/talk.html
This. One thousand times this.
I think that's more than enough good reasons why Debian is our distribution of choice! Next time I'll be looking at the software we run on top of Debian, to make our Drupal servers as fast as possible.
Photo by Kai Hendry on Flickr.com, released under the Creative Commons Attribution 2.0 Generic license.
Our hosting stack from the bottom up - Part 1
Blog
Our hosting stack from the bottom up - Part 2
Blog
Drupal Hosting
Page
WordPress Hosting
Page