James Cooley i.e.

Test-first system administration

By James Cooley - June 25 2008 tags: erlang admin xp testing python java

As a software developer in a startup it's a common enough task to have to set-up a server requiring the following recipe

  • A firewall (apt-get install ufw)
  • A mail server (apt-get install postfix)
  • A web server (apt-get install apache2)
  • A database (apt-get install postgres)
  • Some development tools (apt-get install subversion git-core openjdk-6-jdk)
  • Some python packages (easy_install -U django soaplib mechanize pytz lxml beautifulsoup mercurial)
It's pretty easy to add the packages and but it's harder to add apache modules, or configure a database, or add users.

Configuring a Linux image and deploying it to a cloud is a popular solution but that convenience probably doesn't help when servers start to misbehave. Wouldn't it be nicer if I could

  • script this process
  • use the script to monitor the box
  • check if my machine has been compromised
  • rebuild services as necessary
  • Do backup over ssh (with Duplicity)
  • And most important not have to go back to notes to figure out what happened when something breaks
In the XP test-first tradition you could define a set of tests and then write your script to something like
  • check if my DNS servers are up
  • test if the box is up
  • check that only the expected ports are open
  • check my application specific configuration
  • log in and check activity
  • update packages
  • Send me a txt only if something was wrong.
The Expect toolkit lets you do just this - run a command and deal with a list of cases. The original Expect uses TCL but I like a Python implementation, Pexpect. To get a flavour of the Python version there's an example showing monitoring over ssh. There is an Erlang Expect implementation, Erlspect, that could be a very interesting when you consider how it is possible to create Domain Specific Languages in Erlang.

There are some pretty comprehensive deployment platforms out there like SmartFrog and Puppet but my deployment issues don't need scale up to that level of sophistication. I'd like to use my deployment tools to scale down to a MythTV box, a NETGEAR router or one of the new batch of ssh-capable mobile handsets so Expect seems closer to my needs. A Capistrano-like or ANT-based option might work here too but an Expect-based pattern would seem to be better for bespoke tasks.

The best pitch I have seen for a piece of software recently was for Feedparser

Parse RSS and Atom feeds in Python. 3000 unit tests. Open source.
Test-first system administration with Expect looks like a nice way to capture deployment logic for any kind of device. Any workflow system succeeds only if it can handle exceptional cases - Expect gives you a succinct way do just that.