Graham King Solvitas perambulum Mon, 02 Mar 2015 06:53:15 +0000 en-US hourly 1 Quotes from veteran software engineers Mon, 02 Mar 2015 06:53:15 +0000 The software industry is the most fashion-conscious industry I know of.
– Ivar Jacobson
If you take some of the programs that exist today, they are more complex that just about any artefact that humankind has build before.
– Bertand Meyer
The best path to high-quality software is talented experts who share a pretty clear sense of what they want to produce. I have no idea how to produce good software without talented programmers.
– Peter Weinberger

Masterminds of Programming is a fascinating book, where the authors get many of our industry’s most highly regarded veterans to speak out about basically anything software related. The hook is that they’re being interviewed about the programming language they created.

Here are some of the best quotes and most interesting historical anecdotes.

Common themes

Across all the interviews, several things keep coming up.

  • They don’t use debuggers. Instead they stop and think, inspect the mental model of the application in their head, and then add a few print statements to adjust that mental model.

  • Simplicity is an overarching goal. Ivar Jacobson: “Simplicity is the idea behind being smart.” James Rumbaugh: simplicity “requires a willingness to do too little rather than doing too much.” Only add features to the language if they are useful to multiple groups of users, and solve multiple problems. Keep APIs as simple as possible.

  • Concurrency is a problem we need to solve, and it will probably need language support. Functional languages may be part of the answer, as will frameworks.

  • Software engineering education is lacking, but there was no consensus on how to make it better.

  • Productivity differences between programmers vastly overshadow differences in programming languages.

  • You have to maintain backwards compatibility, even though it’s a pain. Anders Hejlsberg: “I’ve never been able to justify breakage.”

Bjarne Stroustrup (C++)

This was one of the most interesting chapters. It made me want to learn modern C++ (which is nothing short of a miracle), probably using his latest book, Programming: Principles and Practice Using C++.

He portrays C++ as having everything, from low-level direct memory access (“some language has to do it”), to “higher-level abstractions such as iostreams, containers and algorithms”. It it multi-paradigm, supporting both Object-Oriented programming, and Generic Programming, the goal being “direct expression of ideas, resulting in code that can be efficient in time and space”.

There has to be a place behind a barrier where code can be written simply, elegantly and efficiently without worrying about random pieces of code abusing random pieces of other code.
C and C++ have provided stability over decades. There is a price to pay for such stability, but languages that don’t provide it are simply unsuitable for large, long-lived projects.

Alfred Aho, Peter Weinberger, and Brian Kernighan (AWK)

Lots of interesting stuff in here. AWK was designed for people to use by copy-pasting other people’s programs and tweaking them. It was designed for 1-line programs (but Peter Weinberger tells a story of a Bell Labs colleague who “had written an assembler for some esoteric piece of hardware in AWK and it was 55 pages of code”). They see Perl as a natural successor to AWK. Unix command line is a prototypical functional language.

Peter Weinberger:

It’s useful to tease out what the “Unix concept” really was. It was a style in which many programs were useful with one input and one output, together with command-line syntax, and system support that made all input and output uniform (read and write system calls, no matter what the device) and system support (pipes) that avoided having to name and allocate temporary files.
Most comparative questions about languages, coding techniques, development methodologies, and software engineering in general, are appallingly unscientific. Software is still a craft, rather like furniture making. [..] so much of it is anecdotal and based on people’s judgements or even people’s aesthetic judgements.
There are two kinds of software projects: those that fail, and those that turn into legacy horrors.
Given a group of people and some project, it really doesn’t matter what the language is, but for individual programmers I suspect it does matter. I think people, for personality, or what they learn first, or whatever, find some kinds of languages easier to adapt to than others. This is where you get these funny debates.

James Gosling (Java)

Java is about strong fault-isolation mechanisms: “the way that memory pointers work and garbage collections work and exceptions work are really about fault isolation.”

He likes Scala a lot.

On the value of a virtual-machine and Just In Time compiler:

You get two advantages when the compiler is running right at the last moment:
  • One is you know exactly what chipset you’re running on. [..] It knows exactly how the cache works. It knows exactly how the memory hierarchy works. It knows what instruction set extensions this chip has got.

Tom Love and Brad Cox (Objective-C)

Tom Love:

100,000 lines of code is a box of printout. It cost $3 million to develop. It takes two people to maintain it. The number of test cases to fully test that box of code is another two or three boxes of code.
You do a really good job of architecture and design of some new system, and then you turn it over to the organization that paid you to do the work, it’s astonishing how fast that orderliness gets lost in the application. People that don’t know what they’re doing go in and start whacking, and they can do a lot of damage in a short period of time.

Brad Cox:

Objective-C started out as C with Smalltalk (by contrast, C++ started as C with Simula). If he started again today, he might not have inheritance at all: Inheritance just isn’t all that important. Encapsulation is OOP’s lasting contribution.”

He is interested in the economics of software components, as it might enable re-usable components. There’s no economical way to “sell” a component, you can’t do DRM on a class. He’s enthusiastic about SOA, because now you can own and rent the components, the economics line up, so we might finally see the dream of re-usable software components (as re-usable services).

Don Chamberlin (SQL)

SQL started life in 1974 as SEQUEL, a query language for IBM’s System R, which was an industrial-strength prototype of Ted Codd‘s new relational model of databases.

However the first commercial product based on SQL was released in 1979, not by IBM but by a small company called Relational Software Inc. The product was called Oracle, a name that was later adopted by the company, which is no longer small.

SQL is still very widely used. He puts that down to a few things such as being an ISO standard, SQL manages persistent data, which has a long lifetime, but also the fact that SQL is declarative, not procedural. This gives a much greater opportunity for query optimizers to work, so performance has kept up with data growth, and the language remains relevant.

Anders Hejlsberg (C#)

He is very excited about LINQ, mentions it a lot. Languages have to evolve, otherwise they die. Functional ideas are entering imperative languages. Many languages today, the learning curve is dominated by the ecosystem, not the language itself.

A lot of these new innovative solutions that we see in programming languages are great at the 10% new, but then they are terrible at the 90% that every language must do in order for you to really be able to write programs. It’s very, very important to understand that there’s a bunch of boring standard stuff that has to be in every programming language. If you don’t get that right, you will fail.
I’ve seen lots of Java programmers defect to Ruby just because they’re dying in frameworks and Struts and Spring and Hibernate and what have you.

Paul Hudak, John Hughes, Philip Wadler and Simon Peyton Jones (Haskell)

This is the other chapter that made me want to learn the language. It sounds like a very different approach. (I’ve done Miranda in the past). This is easily the most academic interview, with talk of publishing papers, theory, and formal proofs.

The main benefit of functional languages is lack of side effects. Functional languages have influenced mainstream languages, through garbage collection, generics (Philip Wadler worked on Java Generics), iterators, and anonymous functions.

Simon Peyton Jones:

It’s not out of the question that some recognizably functional language might eventually make it into mainstream. (But my guess is that even if it does it will be called Java3).
Where OO folks draw UML diagrams, Haskell folks write type definitions. This is much, much better, because it’s precise and machine checkable.

James Rumbaugh, Grady Booch and Ivar Jacobson (UML)

They all agree that UML 2.x is bloated, most users only need a simple subset of it. Ivar Jacobson’s training company teaches a subset called Essential UML.

Ivar Jacobson’s interest is process, how software is made. Software is built by teams, not organizations. He advocates a team in the sense of a soccer team, complete in of itself (functional teams), self-organizing, and led by a coach.

He has seen software process change drastically every second or third year, more frequently than the whims of fashion. The old process is thrown out entirely, good with the bad, so there is no incremental improvement. Later they realize the new process had some bad too, so it is again replaced entirely. The result of this is a lot of wasted effort as old truths are rediscovered but cloaked in apparent new clothing..

To fix this Ivar Jacobson and Bertand Meyer are working on SEMAT, aiming to find the kernel of practices that underlie all processes. They goal is to find that core of things which we always have or which we always do when we build software.” and extract it, so processes can be compared and evaluated more rigorously.

Simplicity is the core idea behind being smart.
Process in the past was just a soup of ideas. We make the process a bag of practices. [..] Instead of working on a big methodology or a language, focus on working with on practice at a time.
We really don’t have a theory for software engineering. To most people software engineering is just a soup of ad hoc ideas. This is one of the most important problems to fix.
The “agile” movement has reminded us that people matter first and foremost when developing software. This is not really new – this theme resurfaces every decade or so as naive managers try to mechanize and commodotize what is basically an exercise in creative problem solving.

James Rumbaugh:

The main problem facing the computing field [is that] most of its practitioners have no understanding of computing history, and so they are condemned to repeat the same mistakes. Unlike scientists and engineers who built upon past discoveries, too many computing practitioners treat each system or language as a new thing, unaware that similar things have been done before.

Grady Booch:

Consider that our industry has transformed the world in, essentially, one generation. To me that’s fast, not slow.
Abstraction is largely a problem of classification.

Larry Wall (Perl)

He is interested in linguistics, and that influences Perl more than mathematics. Larry Wall is pretty much unique in this. The other designers often mention that programming languages are like mathematics, never like human languages.

John Warnock and Charles Geschke (PostScript)

They made PostScript a language, rather than a data-format, because they didn’t know what kind of devices it would be used on in the future, and because they knew there would be bugs: a full programming language would allow us to program our way out of things we forgot or did not anticipate”

Interpress, the pre-cursor to PostScript developed at Xerox PARC around 1982, was nearly all done remotely. Remote working isn’t new, and doesn’t need fancy tools:

Except for John and myself, the six of us who worked on that project were never in the same physical location until we completed the project. We did it all by email and using the ARPAnet to transfer information around during the entire design project.

Bertand Meyer (Eiffel)

Designing Eiffel didn’t take any design time: “it was the absolutely obvious thing to do”. Eiffel had generics / templates as early as 1986.

Object-Oriented Programming was part of structured methods right from the beginning. Structured Programming was for programming-in-the-small aspects, and Object-Oriented Programming was for programming-in-the-large, but there was no gap between the two.
One of the characteristics of software is that any software element you look at is the specification of something that is more concrete and the implementation of something that is more abstract.

Eiffel has pre and post-conditions (contracts). One of the most interesting things this allows is automatic test case generation.

The tool creates objects almost randomly, and then it calls all the routines, all the methods of the corresponding classes, with mostly random arguments. Then we just wait. We call this “Test While You Lunch”. We start the push-button test, and come back from lunch an hour later, and see the postcondition violations. But you can only do this with a language that has contracts built in.

And more

There’s also interviews with Guido Van Rossum (Python), Tom Kurtz (BASIC), the Lua designers, and others. A very interesting book indeed.

]]> 0
Raw sockets in Go: Link layer Sat, 31 Jan 2015 20:55:57 +0000 Continuing our dive into the Internet Protocol Suite from Go (See part 1 Raw sockets in Go: IP layer), we are going to the link layer, so we can see the IP headers. This will also allow us to craft our own IP headers, or handle address families outside IP. We’ll send ping packets (ICMP echo request) and watch the kernel’s response.




Other useful address families:

  • AF_INET6 for IPv6.










There’s an IP header type in the sub-repository, with Marshall and ParseHeader methods. That file doesn’t depend on anything else in there, so grab it wget, change the package statement to main, and we’ll play with it.







syscall.SetsockoptInt(fd, syscall.IPPROTO_IP, syscall.IP_HDRINCL, 1)




]]> 0
Continuous Delivery: my notes Sun, 25 Jan 2015 04:44:31 +0000 unit test -> acceptance test -> … -> deploy -> release Automate everything. DevOps. Project team should be mix of development, operations and quality assurance / test. Involve […]]]> Continuous Delivery, by Jez Humble and David Farley is about three big ideas to get your code into production more reliably:

  • Make a deployment pipeline: commit -> unit test -> acceptance test -> … -> deploy -> release
  • Automate everything.
  • DevOps. Project team should be mix of development, operations and quality assurance / test. Involve operations (sysadmins) from the start.

Stages of the deployment pipeline:

Stage 1: Commit Tests

Trigger off a version control push. Usually happens in Continuous Integration server.

  1. Static analysis (lint, code metrics like cyclomatic complexity & coupling)

  2. Compile

  3. Unit test (output code coverage):

    • Check that a single part of the app does what the programmer intended.
    • Should be very fast.
    • Do not touch the database, filesystem, frameworks, system time or external systems. Mock or stub these, or use in-memory db.
    • Avoid the UI.
    • Try to avoid testing async code. Should never need to sleep in unit tests.
    • Include one or two end-to-end tests to prove app basically runs
  4. Package a release candidate. Bake in version number.

    Use OS’s packaging tools (deb, rpm). Operations team will be familiar with it, all the tools support it.

  5. Push release candidate to artifact store (a file system, or full fledged artifact repository)

Stage 2. Automated Acceptance Tests

All later stages use the same steps.

  1. A human user (for production) or automated system (for testing) selects a release candidate to deploy, from the available ones in artifact store.

  2. Prepare the environment. Provision machines, install libraries, middleware, etc. A tool like Ansible or Puppet is essential here.

  3. Deploy the release candidate binaries, pulling from the artifact store.

  4. Configure the application. During acceptance testing this means stub/mock out interface to third-party systems.

  5. Run Data migrations

    Have a SQL script to create the database, and load reference data if necessary, to setup new environments.

    Have automated database migration scripts for each step, with a forward and a back migration. Version the database (ID in a table). To get to a specific version you run the scripts forwards/backwards from version the database is at new to desired version.

    Try to do non-destructive or additive work on the database; copy data to an archive table instead of deleting it, rename old columns instead of dropping, etc. Cleanup a few versions later once sure that rollback not needed. Interim version of app should maintain both schemas (maybe write to two tables, etc). This allows de-coupling of database migrations and app deploys.

    Use Expand / Contract pattern. Add fields / tables in one release (Expand), remove them in a later release (Contract).

    Rails / Rake database migrations seem most popular tool for all this, but there’s also Django migrations, Flyway, and others.

    Application binaries should know which version of the database they work with, or keep an external mapping between application version and database version.

    Try very hard not to share the database with other apps. Use service-oriented architecture instead.

  6. Environment tests / Smoke tests

    Did everything deploy and start correctly?

    Is the environment as expected? Are all our dependencies available? Measure how long app takes to start (including cache priming). Ideally these tests are part of the application’s own startup code.

  7. Run the Tests, or Release to end-users

    Test whole stories at a time against a running version of the application in a production-like environment. Can be functional (does feature x do what is expected) or non-functional (security, resilience, etc). Check that the app does what the customer meant it to: “Given some initial context, When event occurs, Then there are some outcomes.”

    Developed in partnership with the customer. Good acceptance tests need good requirements, and a shared understanding of what value the software delivers. Pushed to the edge, acceptance tests are an executable specification (aka Behaviour Driven Design).

    Tests should use business language to talk to a driver layer. The driver will either talk to the same API the UI uses (best, UI should be thin layer anyway), or fill in UI fields (brittle, painful).

    Make each test atomic; create data it needs and tidy up after. Use different user / naming for each test so you can run them in parallel. Be tolerant of initial conditions – if test creates three items don’t check that 3 exist, check that num_before + 3 exist. Can use database transaction isolation for this, each test in own transaction.

    Acceptance tests use real versions of everything we control (database, cache, etc) but still stub out external / third-party systems. We need to be able to read and control their state. This replacement would happen at configuration time.

Stage 3. Automated Capacity Tests

Capacity testing needs realistic scenarios. Replay live traffic, or analyse it to define the scenarios. Choose a handful of key acceptance tests that match common production scenarios and improve them into capacity tests (ideally keeping them as acceptance tests too). They should be able to run in parallel and sequentially with each other.

Define minimum acceptable performance for each one and fail the test if below that. Optimizing code beyond that level is not in customers best interest.

We want to know the maximum load we can handle, and how that changes between versions.

The bottleneck in capacity tests is often the test itself. Have a no-op version of the application to see how fast the tests themselves can run.

Longevity testing is capacity testing but with less load, over a much longer period of time.

Stage 4. Integration Tests

During acceptance testing we stub out third party systems so we can direct their behaviour. Need to check external systems behave as expected.

We can’t read or control their state, and operator usually doesn’t welcome our scripts. Integration testing must be basic, low intensity, and probably run less often than acceptance testing.

In a service oriented architecture setup, this stage might be much bigger, and the Acceptance Tests stage correspondingly smaller.

Stage 5. Manual tests

The QA team do their thing. If successful, they mark that on the build / release candidate in the artifact store, so we know this build is ready for the next stage.

Stage 6. Deploying and Releasing

Continuous Delivery enables incremental regular deployments, which are far less risky than occasional, big-bang deployments. Deploy small units, often.

Use a single way to build and deploy to all your environments. This way the process gets a lot of testing. The process may involve several tools, such as a Continuous Integration server, a configuration management tool (Ansible, Puppet), probably some bash.

Ensure deployment process is idempotent. Tools like Ansible do this. Can also build a new (virtual) cluster from scratch each time (see blue-green deployment).

Don’t login to beta and production machines, all access is via automation tools.

Roll back is deploying the last previous-known-good version.

Backup data assets (db, files) before releasing. Use Expand/Contract (see Stage 2 / Run Data Migrations) so that you never have to roll back data.

Decouple deployment from release. Aim for Zero-downtime / Hot deployment. Run new versions side-by-side with old ones. Several ways to do this:

  • Blue / Green deployments Don’t necessarily need two sets of machines (although that’s best). Can run two copies of the app on same boxes, using different directories and ports. Releasing or rollback is just a load balancer / router config change.
  • Canary Release: Release to a small sub-set of servers. Switch a small sub-set of users to those. If all is well switch more servers, more users. To rollback just switch the users off those servers.
  • Dark Launch: Release non-user visible parts first, so that you get real production traffic, but failures don’t impact the users.
  • Feature Toggles Have a runtime way of switching behavior back to previous version.

However you do it, the shared resources (database, cache, message bus, external services) need to work with both versions of the application at the same time.

Continuous Deployment: The end-goal of the deployment pipeline is fully automated releases to production. Even if you can’t actually fully release, go as far as possible. Maybe auto-deploy to the ‘blue’ cluster, but don’t flip the switch. Or auto-deploy to your canary users (maybe internal company users).



  1. Collect data. This comes from:
    • Hardware: temperatures, fan speed, etc.
    • OS: CPU, memory and swap usage, disk I/O, net I/O, etc.
    • Middleware: Num DB connections, slow queries, redis latency, etc.
    • App: Version, garbage collection, external systems info (liveness, connection pool), and lots of business level metrics.
    • Logs: Treat writing good logs as a first-class requirement. Must be human readable and machine parseable.
  2. Display it: Dashboards for dev / ops and business. Show only the most important.

Making changes without branching (because branching breaks continuous integration):

  • Hide changes until they are ready. Different namespace (URL), config switch.
  • Break big changes into small incremental changes.
  • Branch by abstraction:
    • Create abstraction over part of the system that needs to change
    • Refactor rest of the system to use the abstraction layer
    • Create new implementation
    • Switch abstraction layer to use new implementation
    • Remove old implementation, and maybe abstraction layer

Treat build and deploy tools as seriously as the main project.

The continuous integration server the authors use is open source: Go.CD

]]> 0
Release It: Write software for production Mon, 10 Nov 2014 05:12:08 +0000

We need to design software to run in production. That’s the main lesson of Michael T. Nygard’s Release It. We often think of shipping the system as the end of the project, when in practice it is just the start.

Release It is an enjoyable book with some excellent production war stories. It suffers from being a little too broad in concepts, and a little too narrow in examples (all enterprise J2EE webapps). Despite this I’d recommend spending some time with it because it advocates a very important and easily overlooked idea: don’t code to pass the QA tests, code to avoid the 3am support call.

What follows are my notes from the book, which are a mixture of what the book says and what I think, grouped in categories that make sense to me.


Think about failure modes as you build. Expect and handle failure. A router between your application and the database wil get upgraded, and start closing your TCP connections afer 30 seconds idle time. Your app is only idle that long on Sunday mornings. You can’t test for everything, but you can write your code to recover from failure.

Think about making the inevitable breaks less bad. It’s much easier to do this at development time, rather than try and retrofit.

Decouple parts of the system. Which ones can you survive without? For how long? Split them apart using middleware such as a message queue, or store-and-forward (the email model). Decoupling components means one failing won’t take out the others.

For every external call (disk I/O, socket I/O, sub-process, pipe) ask:

  • What if I can’t connect? What if it takes 10 minutes to connect? Remote side may be gone, or overloaded.
  • What it the connection drops? What if it hangs, or takes 2 minutes to respond? Some middle-ware (or remote) maybe have dropped our connection without telling us. TCP will re-transmit into the void for several minutes.
  • What if 10k requests arrive at once? What if remote is using the connection, but is very slow?

Always use timeouts. Don’t wait forever for anything.

For every incoming piece of data ask:

  • What’s the maximum size it could be?
  • What’s the maximum size we can handle?

Add a LIMIT to database queries. Stop reading incoming data beyond a certain size.

What is the maximum allowed size for any in-memory cache?

Connection pools are a common bottleneck. Make sure they are big enough and don’t block (unless number of connections db can handle is our capacity bound). We use connection pools to avoid the connection setup cost, so if you block on the pool all the gain is lost. What do we do when all connections in the pool are in use? Make some more? Wait a limited amount of time? (then what?).

Pay double attention to any external / resource requests that happen inside a locked region, because a hung external resource will cause deadlocks in your system.

Retry things (db might have done hot failover, virtual IP might have moved, etc), but only a limited amount of times. We don’t want to hammer an already struggling external system. Most failures are not intermittent, they mean something is actually wrong the other end (full disk, remote server down, etc), so don’t waste time and resources with too many retries.

Circuit Breaker pattern helps manager retries. A circuit breaker is a wrapper for dangerous operations (things that can fail, e.g. remote system call). Normally does the operation (‘closed’). If too many operations fail (e.g. timeout) it stops trying and returns errors immediately (‘open’), and of course raise an alert to monitoring system. When in fail state regularly try a single operation to probe if service has returned, if so go back to normal operation.

Avoid config file bloat. Keep system properties (db password, IP addresses) separate from application settings (feature flags, essential plumbing, etc). The first is often handled by the operations team and needs to be secure, the second the developers handle and isn’t sensitive information. 12-factor puts the first type of setting in environment variables, Google puts them in command line parameters.

Fail early: Don’t accept any work (typically connections) until the app is ready. Do a basic self check, add some connections to various connection pools (and ping them), and report status to monitoring system, then start working. You might be bringing up a server during peak load, so it needs to be ready before it opens the doors.

Fail fast: Don’t accept a unit of work unless you expect to complete it. Start by checking all the necessary resources (can get a db conn from pool, remote system has not tripped Circuit Breaker, not at capacity, etc). If we won’t be able to complete the request, return an error immediately.

Shutdown cleanly. Stop accepting new work but let current work finish (with a timeout). We’ll need this to stop the old version when we release new verions.

Design for transparency. Add monitoring from the start.

Anything useful here? Recovery-oriented computing


Do longevity testing. Not extreme load, but some high and some low. System should survive as long as you have between releases (a week? a month?). This will expose memory / resource leaks. The high / low variation will expose idle connection timeouts (such as connection in a pool).

Do integration testing, write tests that use the full system. Run the test suite against your beta / QA environment, run it during the longevity test.

Be perverse in your testing. Maybe TCP connections hang, be very slow, read or write large amounts of random data, repeat the exact same request / message 1000 times, etc.

Test with realistic (large) data volumes. Thing like poor SQL queries, absence of indexes, or slow algorithms, are often only revealed with larger data sets.

QA can’t always have the same number of nodes as production, but make sure it matches as zero, one, or many. If production has 8 app servers (‘many’), make sure QA has at least two. Virtual machines can help lower hardware costs, whilst keeping node counts closer to production. Get QA as close to production as possible, ideally a perfect equivalent. Spend the money, it’s nearly always a big saving compared to production downtime. See Blue / Green deployments


Do load testing. How much load can the system handle before errors or slowdowns? This is tricky as often the load testing client fails first. It’s not something you can do in an hour between other tasks, take it seriously.

When the load test charts show an inflexion (a ‘knee’), that’s usually a bounding constraint has been reached. Figure out what it is: cpu, memory, disk i/o, num threads, num connections, etc.

Reaching capacity usually triggers a stability problem, so how do we limit the load to what we can handle? Options:

  • handshaking: Use protocol to tell upstream to back off. TCP does this.
  • healthcheck: Make an endpoint which says No if we are at capacity. The healthcheck page should also do a general healthcheck, more than just respond OK. Most load balancers can check this and remove back-end server from pool if healthcheck says no.
  • limit connections at the load balancer level to per

In a webapp number of sessions is a capacity limit. Each session uses RAM, maybe CPU and net bandwidth (if it is shared between servers), and persists until it times out (often 30 mins). Clients that don’t have cookies (crawlers, bots), may create lots of sessions. If you can make the session be just an in-memory cache of on-disk data (last search, shopping cart contents, etc) then you both get the benefit of infinite sessions (users hate session expiry), and you can be very aggressive in purging idle sessions.

All the usual performance work applies to increasing capacity too.


Vertical: Bigger box, go big. Db servers (at least the master / writes) usually have to scale vertically.

Horizontal: Add more boxes, go wide. This is preferred. App servers can often scale horizontally.

CPU cycles are only cheap in financial terms. They cost in clock time. Disk space is cheap, but if it’s local you have to multiply extra usage by number of servers, plus the RAID overhead.


Prepare (and practice using) information gathering scripts for production. When something in production breaks you want to a) gather data to isolate the cause, but b) get it going again as fast a possible. The scripts allow you to gather the data you need for a) very fast, so as not to delay b).


Monitoring is about transparency. It is needed for immediate and historical reasons.

Make a dashboard for current (recent past) state, with red / green signals (red bad, green good). It allows seeing at-a-glane what the system is doing. Maybe have a business and an operations dashboard, as those user groups typically care about different indicators.

Historical trends (daily, weekly, monthly) are your best guess of the future; next Saturday’s work load will be most like last Saturday’s. They help answer “what happened before that server crashed“, and predict when you’re going to need to buy more disk space or another server.

Have an interface that makes correlating metrics easy. A single metric across all servers, and multiple metrics on a single server. When traffic goes up, what other metrics go up?

Monitor as a real user, in a remote location. Have a bot doing an as-realistic-as-possible interaction. There are many services to do this for web apps. Most metrics of interest come from inside the app (in the same process), but if you can monitor it externally use that instead.

Store monitoring data off-system.

Things to monitor:

  • System: CPU, Load, Memory, Net I/O, Disk I/O
  • App: Garbage collection times, num threads, num external connections, data about each connection pool (size, wait times, usage), data about caches (size, hit rate), circuit breaker info.
  • Work load: Num work units (typically net requests) processed, avg time, errors / work units aborted, time of last request. Traffic levels anonymous vs registered users.
  • Integration points: Avg response time from remote systems, num active, num errors, time of last expected events (e.g. in / out data feeds, backups).
  • Business metrics: Whatever they care about.

Monitor expected events as well as unexpected. e.g. You need to know if the batch jobs didn’t run, the incoming feed didn’t arrive.

If possible define an acceptable range for each metric and alert or info when out of it. For many metrics that’s very difficult; thanksgiving traffic is ‘normal’ for thanksgiving, not for any other part of the year.

All the alerting, thresholds, and display decisions should happen outside of the app, in the monitoring system.


Aggressively archive off data to remote systems. Do all reporting and ad-hoc queries from the remote system (or read-only slave db). “Ad-hoc” typically means less or not tested, keep them away from the critical system.

Multi-home the production servers (two IP addresses, one or more private). Production traffic uses one interface, admin traffic such as backups, data archiving, replication, monitoring, etc uses the other. The prevents admin traffic flooding the production network, and keeps monitoring going during e.g. a DDOS attack. For security only bind the admin features to the admin interface, it’s a separate listener.

Allow settings to be changed live. Things like which features are enabled, size of different connection pools, length of timeouts, etc. Don’t force a server restart to reload config. If you have to change config, it’s usually because something went wrong, load is high, and you can’t afford to reboot.

Write log file messages for operations staff (which might be you, at 2am). An error should be something actionable. Write log files that are easy to scan ( align fields, keep each log to one line). Log a work unit id (transaction id, user id, etc) so you can group all messages for a given work unit.

Don’t login to production servers (unless something really bad has happened). Automate and stay away. Don’t fiddle. Automate deployment, data purging (db tables, log files, any data that grows), anything else you might need production server access for.

Consider a separate set of servers for very important traffic – critical services, a sub group of users, certain third-party systems, etc. If the main service fails the critical ones are not impacted.


Releasing needs to be easy, so you can do it often. Getting good at releasing is a goal in itself, focus on it, from very early on.

Version everything: files, endpoints, protocols, etc. Use a specific version. This allows multiple versions of the code to co-exist.

Release when the software is ready, not on arbitrary date set months ago (except if you are a product vendor with PR machine).

Release in phases to avoid downtime:

  • Expansion: Add new features, new db columns, new files, etc. Current site not affected. Don’t add db constraints yet, because old code is still running. Write backwards-compatible code. Upgrade data on-the-fly but make sure it stays compatible with old. During this stage there is extra code to bridge the two versions.
  • Rollout: Push new code to servers one by one, gradually. The backwards-compatible code can co-exist happily. Ideally handover socket connections, or hot-reload code, but at least shut down cleanly (see Design section).
  • Cleanup: Once all servers are on new version and it’s stable, drop old columns, add referential integrity, remove the bridging code.
]]> 0
What a visit from Reddit looks like Tue, 04 Nov 2014 04:58:30 +0000 My credit card generator was recently on the front page of reddit. Here’s what that looks like.

Here’s htop during peak traffic (click for larger view):

100% htop

This site is Apache (WordPress) behind nginx, running on a Linode. The top banner is served by netshare, a small C program I wrote using the magic of epoll and sendfile.

The credit card generator is a real generator, every visitor gets new random numbers. It’s PHP (github) and very evidently CPU bound. Here’s Linode’s CPU graph:


I have to pause and point out how awesome Linode is. I’m paying for 1 core, and there’s 8 cores at 100%. That’s 8 times what I’m paying for. No extra charge, no angry email. During the peak of reddit traffic, the site was always responsive.

Today, what it takes to survive being on the front page of the Internet is $20 / month and a handful of basic open-source tools (WordPress, Apache, Nginx, MySQL). Thanks Linode and thanks free software!

It is often said that traffic from sites like Reddit is ‘low quality’ traffic. A brief spike of traffic, with no returning users or long-term visitors. That’s quite true in this case – here’s the WordPress site stats:

Wordpress stats

Finally, here’s the network traffic for IPv4 vs IPv6. Look at the numbers on the left. We’re not there yet.



]]> 1
Learning assembler on Linux Sat, 23 Aug 2014 03:49:14 +0000 For entertainment, I’m learning assembler on Linux. Jotting down some things I learn here.

There are two syntaxes, AT&T and Intel (Go uses it’s own, because Plan 9). They look very different, but once you get over that the differences are minimal. Linux tradition is mostly AT&T syntax, MS Windows mostly Intel.

There’s no standardisation, so each assembler can do things it’s own way. as, the GNU Assembler is the most common one on Linux (and what gcc emits by default), but nasm, the Net wide Assembler is very popular too. Code written for as will not assemble in nasm.

Talking to the Linux kernel is different depending whether you have a 32-bit (x86) or 64-bit (x86-64) processor:

  • The registers to use change
  • The instruction to call changes (int 80h vs syscall)
  • The syscall numbers change

So before you even get started, you need to pick a syntax, an assembler, and a target. I’m using as, with AT&T syntax, on Linux x86-64.

To learn I’m reading Assembly Language Step-by-Step. It’s definitely helpful, but it’s targeted at a CS 101 class which makes it slow going. It also uses Intel syntax, with nasm, on 32-bit, which takes a bit of mental translating.

Here is the first program from that book, translated, in case you want to play too:


    .ascii "Eat at Joe's!\n"
    eatlen = . - eatmsg


.global _start

    mov $1, %eax        # 'write' syscall
    mov $1, %edi        # write to stdout (fd 1)
    mov $eatmsg, %rsi   # address of string to write
    mov $eatlen, %edx   # length of string to write

    mov $60, %eax       # 'exit' syscall
    mov $0, %edi        # return code 0

Save as eatsyscall.s and build with:

as -gstabs -o eatsyscall.o eatsyscall.s
ld -o eatsyscall eatsyscall.o

Other bookmarks I keep open:

I’ve already learnt two interesting things, about starting and stopping programs.

Programs don’t start at main, they start at _start. When you build a C program, _start is put in for you, does some setup, then calls main. _start is the symbol the linker ld looks up to know what address to put in the ELF header as the entry point address. For a different example, the Go start symbol (on x86-64 linux) is _rt0_amd64_linux.

Programs have to explicitly exit. If you don’t call the exit (or exit_group) system call, your program keeps on running, tries to get it’s next instruction from whatever comes right after it in memory, and crashes.

You can call all of the C stdlib functions from assembler, by using gcc to link, or passing the right arguments to ld. Or you can not be so lazy, and do everything yourself!

That’s the part I’m most excited about. How am I going to allocate memory, without malloc? No, don’t answer that. The fun is in figuring it out.

]]> 0
Dump Go Abstract Syntax Tree Sat, 28 Jun 2014 19:09:44 +0000 Go has good support for examining and modifying Go source code. This is a huge help in writing refactoring and code analysis tools. The first step is usually to parse a source file into it’s Abstract Syntax Tree representation. Here’s a complete program to display the AST for a given Go file:

package main

import (

func main() {
    fset := new(token.FileSet)
    ast.Print(fset, f)


  • Save that as goast.go
  • Build it: go build goast.go
  • Run it: ./goast <myfile.go>
]]> 0
Sync, a Unix way Sat, 24 May 2014 04:49:59 +0000 Ever since Dropbox, I’ve been searching for a self-hosted, secure (and now Condi-free) way of keeping my machines synchronised and backed up. There are lots. I tried many, wrote a couple myself, but none were exactly what I wanted.

My problem was thinking Windows, looking for a single program. Once I started thinking Unix, looking for modular components, the answers were obvious.


First we need a remote master storage to sync against, somewhere to backup our files. And we want that exposed as a local filesystem. I use the most obvious answer, sshfs:

sudo apt-get install sshfs
mkdir -p /home/graham/.backup/crypt  # Why 'crypt'? Read on.

sshfs /home/graham/.backup/crypt

You can use any storage that can appear as a filesytem, such as FTP (via curlftpfs), NTFS, and many others.


There’s two kinds of data: public data, and encrypted data. We want the second kind. Just layer encfs:

sudo apt-get install encfs
mkdir -p /home/graham/.backup/clear

encfs /home/graham/.backup/crypt/ /home/graham/.backup/clear/

The first time you run encfs, answer ‘p’ for paranoia mode.

At this point anything you copy into .backup/clear will be encrypted and copied to your remote server. Not bad for two lines!


If you’re offline, .backup/crypt and .backup/clear will be empty. We want a local copy of the files. If you’re just doing backup (i.e. all changes come from a single local machine), use rsync. I want true two-way sync (desktop<->server and laptop<->server). The tool of choice for that is Unison:

sudo apt-get install unison
mkdir /home/graham/backup   # Local copy

unison -auto /home/graham/backup/ /home/graham/.backup/clear/

And that’s it! Three lines, an encrypted, remote, synchronising file system. Thanks Unix!

Bash glue

You need to re-mount the sshfs and encfs filesystems whenever you reboot your machine, and you probably don’t want to type that unison command line all that often. Let’s make a little bash script. I called mine sy, in my ~/bin/ directory.


sync_mount () {
    if [[ $(mount | grep .backup/crypt) ]];then
        echo ".backup/crypt is mounted"
        sshfs /home/graham/.backup/crypt
    if [[ $(mount | grep .backup/clear) ]];then
        echo ".backup/clear is mounted"
sync_umount () {
    fusermount -u /home/graham/.backup/clear
    fusermount -u /home/graham/.backup/crypt
sync_install () {
    sudo apt-get install sshfs encfs unison
    mkdir -p /home/graham/.backup/crypt
    mkdir -p /home/graham/.backup/clear
    mkdir /home/graham/backup
sync_full () {

case "$1" in
    sync_umount ;;
    sync_install ;;
    sync_full ;;

Further options

I’ve been using this exact script and setup for a month now, and I’m very happy with it. It feels so much simpler than everything else I tried.

If you have files in your backup folder that change rarely, Unison has a powerful -ignore syntax, for example -ignore 'Path Documents'. The fewer files Unison has to check, the faster it runs. Also -batch will stop Unison asking you for input.

A next stage could be to trigger sy automatically whenever you change something. That would use something based on inotify, maybe lsyncd or incrontab. I thought I would want this when I started, but so far it hasn’t bothered me. I just type sy once in a while. Let me know if you go that route.

]]> 1
GopherCon 2014 favorite talks, notes Sun, 04 May 2014 18:39:00 +0000 My favorite talks at GopherCon 2014:

  • Peter Bourgon: Best Practices for Production Environments Soundcloud were an early Go adopter, and this talk is their distilled learnings from two years of Go: Repo structure, config, logging, testing, deployment, and lots more. The one talk you need if you’re starting (or running) a significant Go project, and you want to do it right.

  • Petar Maymounkov: The Go Circuit: Towards Elastic Computation with No Failures Stick with this one. It starts off quite academic, but gets fascinating very fast. He models whole companies as a distributed system (based on CSP), then builds a language-agnostic cluster programming library where the API is a filesystem The Circuit. One of the highlights of the conference for me was building a filesystem with Petar in the hallway.

  • John Graham-Cumming: A Channel Compendium John is the author of those great in-depth Cloudflare blog posts. Solid talk about Go channels. nil channels always block, so you can ‘disable’ a select clause by setting a channel to nil. Closed channels never block. Heartbeat is just time.Tick, timeout is time.After. Go programs are small sequential pieces joined by channels.

Those are the three talks I enjoyed most. Here are my general notes on the conference and a few of the other talks.

GopherCon was 700 people, an unexpectedly large turnout for a first conference and a recent language. Go is definitely “the language of servers”; lots of talks by sysadmins and ops people, lots of sponsors from infrastructure businesses (Digital Ocean, Ubuntu / juju,, Apcera, etc).

Denver is a very enjoyable city. Great beer (craft breweries all over), a bike share scheme and many bike lanes, warm dry weather, friendly locals, great food. It put me in mind of Portland without the rain. I urge you to go if you get the chance.

Rob Pike’s keynote presented the history of Go, all the way from ‘hello world’ in B in 1972, through K&R C, ANSI C, to Go. Initial discussions about Go started in 2007, first draft spec 2008, open source Nov 2009. He talked about why he thinks the language is such a success. He dissected a Go ‘hello world’ program, token by token.

For those following the “Go has no debugger” saga, Rob said that modern operating systems make it very difficult to write a debugger, because the debug interface came before multi-threaded programming. He also said ptrace is not very good (not his exact words – swearing was involved). A deep learning there maybe.

Derek Collison talked about optimizations he’d done on a message queue (there were three message queue talks at GopherCon – Go is definitely the language of software infrastructure):

  • Be garbage collector friendly: Don’t allocate short-lived memory on the heap. Do as few allocations as possible. Strive for ‘zero allocation’ in critical sections of code. Converting string to []byte, and vice-versa, is an allocate and copy, so don’t do it. This advice came up in several other talks.
  • Use -race in your automated test suite.
  • A fast parser is a non-destructive parser (again, to avoid memory allocation, copy, and garbage collection). Fast parsers tend to look like giant switch / if statements. Lots of array matches.

Josh Bleecher-Synder talked about Embedded Go and Bluetooth Low Energy. The number of memory allocations is as important is the size, both influence the length of the garbage collection pause.

Bluetooth Low Energy sounds really interesting. It’s so low power it can run on a button battery, and your device can be ‘always on’. It’s cheap and soon to be ubiquitous. Most new smartphones support it. Josh recommended this book for more details.

He also talked about building “crash only” programs: Assume you can crash at any point. Check everything at startup, make it all sane. No cleanup, recover or shutdown code, it’s all done on startup because if the battery runs out, your shutdown code isn’t going to run anyway.

Rob Miller of Mozilla talked about Heka. The embed Lua so they can do dynamic loading of plugins. Go starts Lua via cgo. Lua is very fast, and has Lua Parsing Expression Generators which sound interesting. He said in many cases they could pretty much copy the BNF grammar from the RFC into Lua, which makes writing parsers easy. They use protocol buffers to interact between Lua and Go. He also said don’t convert between string and []byte (or vice-versa) because it’s an allocate, copy and free operation.

Ben Johnson talked about high performance databases. Query execution is what everyone cares about, so focus on that. Protocol Buffers is much faster than JSON (60 times, for him). Like several other speakers, he said in performance critical sections limit memory allocations and memory copies. One of my favorite parts of his talk was his “Hierarchy of SPEED”: Network I/O > Disk I/O > Memory Allocation > Mutexes > Memory Access. Worry about them in that order.

Victor Vieux talked about Docker. Sadly I missed much of his talk (because I was learning to build a filesystem!). He mentioned that Go test files run in alphabetical order. He also showed the mflag package, a replacement for Go’s flag package which does both long/short options (GNU style) and option grouping. It sounds like a better flag. He showed which will run code coverage on any public Go project (in a Docker container of course).

Ron Evans and Adrian Zankich had a very cool talk about Go powered robots with They showed Go on an Arduino, a Beaglebone Black, a Sphero, and then an AR Drone. Highlight was Go running on an AR Drone, using Open CV to identify faces, flown via a PS3 controller. You really need to see the video to enjoy this one, I’ll try and link when it’s online.

Blake Mizerany ran an entertaining panel of core devs:

  • Rob Pike shared an office with Bjarne Stroustroup at Bell Labs.
  • Rob advocated programs that write programs. You can have generics, just generate the type-specific version from a template at build time. Interestingly Ward Cunningham elsewhere has also mentioned that programs-that-write-programs are underused.
  • Interface’s are pure behavioral typing, structural typing. It’s important that you don’t declare what interfaces you implement, because it’s up to the user of you code to decide what interfaces they depend on.

Baron Schwartz, co-author of High Performance MySQL talked about the database/sql package:

  • sql.DB is not a connection, it manages a connection pool.
  • Call rows.Err() after your rows.Next() loop is done, that’s where you would get any errors that interrupted rows.Next.
  • Call rows.Close() as soon as you can, because a row set holds a database connection.
  • MySQL prepared statements are tied to a connection, whereas Go prepared statements are not. That means if between calling db.Prepare and using the prepared statement someone else took your connection, the statement may get prepared twice. He didn’t mention any problem with this, just a curiosity.
  • db.Ping() is how you open and validate a connection.
  • Because we’re in a connection pool, it makes no sense to issue connection-specific commands (SET NAMES UTF-8, USE <mydb>, etc). If you really need to bind a connection make a transaction: db.Begin() and db.Commit().
  • The database/sql package will auto-retry for you up to 10 times.
  • Make custom types by implementing Scanner and Valuer interfaces. Baron uses this to transparently encrypt sensitive information. Suggests gzip too.
  • If you really need database specific functionality, import the driver directly.

Alan Shreve talked about “Living outside the cloud”. He works on a command-line app which must run without network. Embed assets with go-bindata. Use termbox to abstract away curses.

Andrew Gerrand closed the conference:

Code is communication. Strive for good names, simple interfaces, precise docs. Don’t be too clever. Less is exponentially more.

Don’t build a thing, instead compose simple tasks.

Simplicity is hard. It takes time.

]]> 0
Raw sockets in Go: IP layer Sat, 01 Mar 2014 23:41:48 +0000 In the Internet protocol suite we usually work at the transport layer, with TCP or UDP. Go (golang) has good support for working with lower layers. This post is about working one layer down, at the IP layer.



Let’s read the first ICMP packet on localhost:

package main

import (

func main() {
    protocol := "icmp"

    buf := make([]byte, 1024)
    numRead, _, _ := conn.ReadFrom(buf)

Build it go build test.go and run it as root sudo ./test. You need to be root to do raw sockets, because (from man 7 raw):


In a different window, ping localhost. You should see the raw bytes of your ICMP packets. Next, change protocol to tcp, and wget localhost.

We usually work at the TCP / UDP layer, which gives us the content of TCP or UDP message. Here, because we are at the IP layer, we get the contents of the IP message, so we can see the ICMP or TCP header.

In the ICMP (ping) case the first byte should be 8, an Echo Request.

In the TCP (wget) case, the first four bytes are the source and destination port. The source port will be a high number, an ephemeral port the OS chose for you. The destination port (bytes 3 and 4) will be 00 50, which is hex for 80, the HTTP port. Byte 14 will be 02, telling you this is a SYN packet, the opening message of the TCP three-way handshake. This handshake is normally hidden from you, when you work at the TCP layer.


To send packets, you need to make them. There’s a Go TCP header and example usage in

A full program would be too long to show here, but if you get tcp.go from the link above, you could send a TCP packet like this:

packet := TCPHeader{
    Source: 0xaa47, // Random ephemeral port
    Destination: 80,
    SeqNum: rand.Uint32(),
    AckNum: 0,
    DataOffset: 5, // 4 bits
    Reserved: 0, // 3 bits
    ECN: 0, // 3 bits
    Ctrl: 2, // 6 bits (000010, SYN bit set)
    Window: 0xaaaa, // size of your receive window
    Checksum: 0, // Kernel will set this if it's 0
    Urgent: 0,
    Options: []TCPOption{},

data := packet.Marshal()
packet.Checksum = csum(data, to4byte(laddr), to4byte(raddr))
data = packet.Marshal()

conn, err := net.Dial("ip4:tcp", raddr)
if err != nil {
    log.Fatalf("Dial: %s\n", err)


Making your own packets means working in binary, and several things help when working with binary in Go:

  • The encoding/binary package.
  • Conversions truncate without complaining. If you have a := 0xFFFF and you only need the lower 8 bits, byte(a) does it.
  • The usual shift <<, >> operators, and boolean logic | and &.

In part 2, we go to the link layer to read and write IP packets: Raw sockets in Go: Link layer

]]> 2