Matt Williams

Your First Hire Should be a Sysadmin

From the very beginning of Crowdtilt, we envisioned an architecture where the only limitation to how fast we deployed code was the speed at which we could actually develop. Our CTO saw the value in hiring a Sysadmin to focus on this objective from day one. That’s where I come in and here’s how I’ve helped us achieve the goal…

Configuration Management

The first order of business was managing our systems at a higher level. It took very little deliberation to decide on Chef as our configuration management (CM) tool. Our prior experience, the massive collection of available application “cookbooks”, and a very active community were enough to convince us that it was the right tool.

The nice thing about integrating with a CM framework from the beginning is being able to tie in each piece of a web-stack as it becomes necessary, instead of having the monolithic task of converting an entire architecture. The very start of our app involved a Postgres database, the web app, and Nginx out front – fairly typical. We were able to utilize community Chef code for Postgres and Nginx and all of our site specific modifications were implemented in our Crowdtilt “cookbook”. We rolled our own Chef resources for app deployment and wallah – running chef-client now deploys code from our git branches. Combined with great plugins like knife-ec2, Chef is now building our Cloud servers and then configuring our software on top of them!

Consistency From Development to Production

To ensure that the developers are developing in the same environment as our Linux servers in AWS, we introduced Vagrant very early on. For anyone unfamiliar to Vagrant, it’s a wrapper for Virtualbox (and others) that allows for integration with CM frameworks like Puppet and Chef. After provisioning a Vagrant VM, the developer can visit an instance of the site on their local workstation and see their code as it will appear in staging and production.

With Chef managing all of our servers and deploying code from our git application branches, we rely on merges to dev to deploy code to staging, and likewise, merging to master deploys our production environment. When combined with code reviews and Jenkins test-suite automations, it really allows the developers to focus on features without getting bogged down in implementation details or manual processes.

We utilize a very similar system for the Chef code itself. We test Chef code against the same Vagrant instance but use a local bare-bones Chef server called chef-zero. This tool has all of the same core functionality of the real Chef server with minimal setup cost of a full-blown Chef server. We write our infrastructure code locally using this chef-zero server and build our local Vagrant boxes against it. Once we’re satisfied with our Chef code we’ll merge to dev and a Jenkins job syncs the dev branch with the Staging Chef server. Rinse and repeat for production/master.

The Payoff

The end result of this system really speaks for itself as we have been able to move incredibly fast! A great example of this was when we decided to move from Ubuntu 10.04 to 12.04. I was able to run a simple set of Chef commands that provisioned the new servers, removed the old servers, and automatically populated the load-balancer with the new server data. All of a sudden we’d replaced 90% of our stack before lunch! This kind of flexibility and speed has allowed us to spend less time doing busy-work and more time innovating!

Lessons learned publishing my first CPAN module »