Automate to survive

Everyone has a choice

Automate or die, That is pretty much it, you can automate everything or you can keep working with manual processes that slow you down. If you don’t think you have the time to automate, you’re wrong; you need to automate and do it quickly before you get even more busy and even further behind. Maybe you think that you can’t automate because you don’t have the time to do it justice? maybe you can’t automate because the task is to big? Too complicated? well it’s all rubbish.

Start small

This is a bit like eating an elephant, You have to start somewhere, you have to start small, by all means try and start big if you want, but smaller is better. Maybe you have a task to check for new packages from a site once a month, that is a good place to start, pull third party packages from vendors sites into your yum repo or maybe every time you build a server you need to do x, y & z. These sort of tasks are achievable for everyone even those without a good background of programming background which leads on to language choice, not all are equal but knowing two or three is better than just one. At a minimum some sort of terminal language, so Bash,ash,sh or ksh and a ‘Proper’ sort of language that is object orientated like, Ruby, Python or Perl. The terminal languages are good for re-producing what you do on a terminal into a reproducible and consistant format but are terrible for manipulating multiple data sources, mangling data, although with that said you can do some complicated things.

Once you start building up many smaller components of automation start looking at ways of linking it all together so that a series of tasks becomes one. It is this constant cycle of simplifying the process to automate the small chunks and then linking the small chunks together that make an automated system.

Grow large

Over a year ago we use to deploy our environments with Puppet and cloud formations and it use to take about 2.5 days to complete and get it working, that whole process is now down to 10 mins thanks to automation. It required many leaps of faith, many poor decisions and a lot of bug fixing but it got there though simplifying everything down and then automating each component. Other than building the servers and tagging them with appropriate keys in Amazon the whole process is controlled by bash to the point of a working system and is typically very robust. That is a massive time saving, but to get there we had to fail, we had to try and we had to persist.

As a result we now automate large portions of the architecture to a point where all of our time is split between incidents or project work to implement new features hardly any daily grind. Recently I have been working on our DR strategy to take it to the point of clicking a button to deploy a clean environment built from the ground up and automatically pulls the latest backups to restore to the environment but it is now done and saves hours of time building out a DR which makes the recovery time shorter and the process is easier. So larger projects are perfectly achievable with the right attitude!


Give it a go, start small and work up to it but be un-relenting and do what ever it takes, no matter how much you disagree with it, just do it to get it automated, once more is automated you’ll have some time to fix it up properly or you’ll need to extend it and you can make a small part better then.

Release consistency

It’s gotta be… perfect






  • Quick release
  • Quick accurate testing
  • Known Product build
  • Known configuration
  • No ad-hoc changes to systems
  • Ability to make ad-hoc to systems
  • Infallible release process


If you love it, let it go



  1. Removes ad-hoc changes
  2. Tests documentation / configuration management
  3. Enhances DR strategy and familiarisation with it
  4. Clean environment







Sentinel Update

No, really an actual update

So after feeling all bad for not committing any work to sentinel for a while I decided to brush off the dust and crack on with some simple bits.

As of right now Sentinel will:

  • Perform basic checks of *nix processes
    • Is there a process running?
    • Is it in a running or sleep state (or other healthy state)?
  • Basic check of system health
    • Check disk usage with DF
    • If the disk usage is high, Log the offending disk info to the system log
    • Check memory usage of system
  • Basic Application health
    • Perform basic URL grab / scrape for search string
    • Check the amount of memory the application is using of what it requested & of the system total
  • Basic actions
    • If process Zombied / Dead, Kill it
  • Run as a cron job
  • Log output to file and to screen in log4J style format
  • Take options from CLI where appropriate

This is not to bad, there’s a few things still missing from what I’d like it to be able to do before I start testing it out such as:

  • Identify service associated with process and restart if it kills it
  • Restart application if URL check fails X number of times
  • Tidy up disk space based on known “safe” files that can be deleted (like log files over X days)

That’s all I want to achieve. Once I have that I then need to start with what is a rather tiresome activity which will be the testing of that application to make sure it works in a way that is sensible. After the testing that’s it!

I wish, I will after I’ve done some testing then re-structure the code, although it works as it is it is not maintainable and it is becoming harder to write code for it without it being in the wrong place and not quite doing the right thing… so I want to split the code out so it is easier to see what each bit does. I will probably create a a few classes to put the code in and make it better, either way I’ll grab some advice from people that know how to program first so I can get it into a reasonable structure.

Other than code layout, theres the structure of the data I’m trying to store, at the moment I am trying to use a default constructor in Ruby to create a class for my scores, however this is not working out I’m pretty sure I’ll need to write my own constructor and methods to get and set the values. This way i’ll be able to create a hash to store the data in instead of a list of variables.

Then there’s writing a proper fix to my 50 line case statement which I imagine is no small task but hopefully that will be pleasantly challenging.

is it worth it?

I think so, even if no one ever uses this I am learning more about programming structures which I think will be rather useful in the long term as I imagine there may be a time when being able to code to a reasonable standard may be needed.

Hopefully when the re-structure of the code is done and the last of the features is there it will hopefully be a useful application to deploy and hopefully that will save me from being woken up at 3am to deal with a simple support issue.

So that is basically what I’ve been playing with for a majority of this last week and I will try to continue on with it for now to at least get the final proof of concept done and see how it works out, at that point I’ll start trying to design it a bit better so it works as a proper application rather than a dodgy script.

Sentinel – An open source start

An open source start




  1. More ruby programming experience
  4. I like making things work








A call for help





So please, please, please get involved with Sentinel

Self healing systems

An odd beginning

So I’m writing this having just spent the last 10 days on my back in pain and finally starting to feel better, it’s not come at a good time as another member of the same team as me decided they had a “better opportunity” This is the second person to have left this organisation without as much as a passing comment to myself that they were even leaving, how strage; but I digress.

Either way it opens up a void, a team of 2 and a manager now down to a team of one, with the one having back pain that could at any moment take me out of action. Unfortunately up to the day before I was not able to make it to work the system we look after has been surprisingly stable, rock like in-fact; as soon as I say “I’m not going to make it in” the system starts having application issues (JVM crashes).

Obviously the cause needs a bit of looking into and a proper fix etc etc, but in the mean time what do we do? I had an idea, A crazy idea which I don’t think is a fix to any problems but it at least a starting point.


I have spent a bit of time exploring Ruby a few weeks back so I started to look at ways of writing something that would do a simple check; is process X running? In the simple version I wrote it just checked that tomcat was running more than one instance (our application always runs 2) if it was 2, do nothing, if it was more than 2 do nothing (something else crazy has happened so it just logs to that affect) but if it was less than 2 it would try a graceful-ish restart of the service.

So this obviously works in the one specific case that we have, but isn’t extensible and it doesn’t do any better checks, which all got me thinking. Why isn’t there something to do this for us? I don’t know of anything that does this, if anyone does I’d appreciate knowing, there’s a number tools that could be muddled together to do the same sort of function.

Nagios would monitor the system, cucumber could monitor the application interactions, Swatch could monitor the logs, but in most cases these are monitoring, I’m sure there’s ways to get them to carry out actions based on certain outcomes but why use so many tools.

Yes, the individual tools probably do the job better than a single tool, but as a sysadmin, I’d rather have one tool to do everything but that isn’t practical either. So can we some how get the benefits of monitoring with nagios but have a tool that is specifically monitoring the application performance nagios is gathering information about and then making decisions based on that?

The big Idea

So I wonder if it’d be possible to write a simple ruby application that every now and then did a number of actions:

  1. Check the service on the box, right number of processes, not zombied etc, etc
  2. Check the disk capacities
  3. Check the CPU utilisation
  4. Check the memory utilisation
  5. Probe the application from the local box, a loopback test of sorts
  6. Integrate with nagios or another monitoring tool to validate the state it thinks the box is in compared witht he locally gathered stats
  7. Depending on the outcome of all the checks carry out a number of actions
  8. Hooks int ticketing systems

When I was thinking this through the other day, it seemed like a good idea, the biggest issue I have is not being a programmer, So I have a steep learning curve, it’s a complicated application, so requires some thought. I would also probably have to ignore everyone that thinks it is a waste of time, which isn’t too hard to do.

I guess what I’m thinking of is something like FBAR. As a system scales up to hundreds of servers the up time and reliability becomes more important, it is sometimes necessary to take a short term view to keep a system working. The most important thing is that those short term views are totaled up and then logged as tickets, 1% of your severs crashing and needing a restart isn’t an issue, but if that 1% becomes 5% and then 10% it’s panic stations!


I think my mind is made up, a sentinel is needed to keep watch over a solution, and what’s crazy is that the more I think of it the more useful it seems and the more complicated it seems to become. As such I think I’m going to need help!