Thursday, 6 February 2014


This past weekend, I and several others made our way to ULB for FOSDEM 2014. FOSDEM is an annual conference aimed at the free and open source software communities. In addition to talks on a wide variety of topics, it doubles as a convenient place and time for developers on projects who are spread all over the world to meet up in person.

I attended a bunch of talks over the weekend; below is a brief list and some impressions based on my recollections and notes. Almost all the talks were recorded; once they’ve been published, I’ll come back to this blog and link them all.


Reproducible Builds in Debian

Binary packages in Debian are currently the opposite of reproducible. Two consecutive builds of the same source package on the same machine will produce different binaries. Most differences come from timestamps (in archive formats, documentation, etc) but there are some other sources. There is an effort on going at the moment to modify the tooling used to create binary packages in Debian to eliminate these sources of variation. An experiment using these modified build tools resulted in more than 60% of packages being identical. That’s insanely impressive. It seems very possible that we could get to a situation where 100% of packages are verifiable. If we could then create an audited build image from first principles (or ideally two or three), it would become significantly harder for an attacker to sneak undocumented behaviour into these packages. Definitely one to watch.

Autoscaling Best Practices

A talk about autoscaling; what it is in general, the various types of autoscaling, what it’s good at, what it’s not so good at, and some of the implementations available in the world. A solid high level overview which gives good advice. Worth watching if you’re new to the concept of autoscaling infrastructure. (TODO: video).

Network Function Virtualisation and Network Service Insertion and Chaining

This was a somewhat theoretical talk about how the world changes when you virtualise network services, using the various components of OpenStack to provide examples.

Briefly: a network service is some capability which is provided by the network to one or more application/host/hypervisor. The most trivial network service is that of basic connectivity; the ability to send packets elsewhere, although obviously this isn’t that interesting.

Consider instead something a little more complex; say a network firewall, or a load balancer. Traditionally these capabilities/services have been provided by big centralised pieces of Magic Vendor Hardware; having a pair of firewalls and a pair of load balancers for an entire datacenter is a fairly common (anti?) pattern. This leads to a bunch of complexity and scaling issues.

For example: if you need to firewall flows between two applications in your cluster, you need to drag those flows all the way up to your firewalls and then back down again, even if the applications are running on two VMs/containers scheduled on the same hypervisor. This then has implications for how much bandwidth you need to provision from your top of rack switches to your cluster edge and back. Your firewall policy also grows alarmingly; it is essentially the union of all permitted flows to/from every node in your cluster. Such a policy is going to become difficult to maintain and reason about, and quite possibly is going to blow out the capacity of whatever BigCorp firewall appliance you’re using, meaning that you soon have to start scaling it out.

A key point this talk makes, which bears emphasising: you don’t need SDN for virtualised network services, but the two do complement each other nicely.

Now, let’s pretend that this is a world where network functions are virtualised, and that I’m spinning up a new SSL web service.

I need a new public IP, because SSL; no problem, my SDN control plane starts advertising the new IP via BGP to the outside world, and my SDN controller installs a flow on my edge routers which sends any traffic to this new IP to some newly provisioned software load balancer instance which is running inside the cluster. It also updates the iptables rules on the hypervisor that the loadbalancer is running on, permitting inbound port 443 traffic to the new public IP.

In the mean time, my cluster control infrastructure has spun up a reverse proxy VM as well as a webserver on which I can run the actual website. Right now, because no-one knows about this service, the load balancer, the reverse proxy, and the web server VMs are all running on the same hypervisor. However, 6 months from now, when people actually care about this service, scaling it out is a simple case of spinning up more instances of whatever the bottleneck component is, combined with possibly updating some flow rules so that you can have more than one load balancer.

Compare this to a world where you need to submit tickets for new hardware, new firewall rules, some load balancer changes, some reverse proxy changes… a new order is coming, folks. Adapt or die.

Dovecot’s way of scaling to millions of users

This was an interesting talk by Dovecot’s maintainer about the various implementation optimisations that have been made in order to get over performance limitations in Dovecot. There was some interesting conceptual/philosophical stuff here (e.g. everything should be asynchronous), as well as good discussion about the architecture of a scaled Dovecot installation. (Leaf nodes, proxies, directors). I’m reasonably certain that I’m going to have to get back to self-hosting my own mail in the next few years, as Gmail becomes more and more of a consumer product, so this was a useful refresher on how real email works.

SPARK 2014: Hybrid Verification using Proofs and Tests

An interesting talk on how to extend compiler and testing infrastructure. A lot of it went over my head, since I don’t spend enough time coding, but the main thrust of this seemed to be that contracts (decorators/hints to the compiler which express preconditions/postconditions/invariants) allow you to move many runtime assertion/check failures to compile time. (Trivial example; bounds checking on array access). In addition, contracts allow you to skip a vast swathes of trivial unit tests; e.g. the contract for an “increment” function provides sufficient information to allow the compiler to auto-generate test cases; we shouldn’t need to write them ourselves.

SPARK appears to include a bunch of interesting language constructs which aid in producing safer, less fragile code. It will be interesting to see if contracts or something like them make their way into more widely used languages.

Foreman Not a talk, rather a project I found out about.

Foreman is an opensource project for managing the lifecycle of physical and virtual servers, from provisioning to operating to monitoring to configuring to decommissioning. It looks like a mature project, with significant support for both Puppet and Chef, as well as a bunch of physical and cloud compute providers. Definitely a project to keep an eye on.

Who ate my battery?

Why free and open systems are solving the problem of excessive energy consumption.

This was a very thought provoking talk about the seeming disconnect between the hardware and software worlds when it comes to power optimisation. The hardware world has long included “power consumption” as one of the variables in the optimisation problems that are being solved. In the “old” days, when embedded hardware ran custom, purpose written software, such hardware features were made use of as a matter of course. Now, more and more embedded devices run general purpose operating systems and applications written in high level languages which do not expose developers to the realities of embedded development. This has a direct impact on the power efficiency of such devices.

This talk discussed various approaches to change this; starting with the profiling of software combined with measuring power consumption, in order to gather power efficiency data for various combinations of hardware, compiler, compiler version, compiler optimisations… and so on. Once this profiling data has been gathered, it can then be used to feed a machine learning system which then generates a model with which to simulate upper and lower bounds for power consumption of various algorithms/implementations of those algorithms, given a set of the above constraints. Think of these as akin to back of the napkin calculations based on latency numbers every progammer should know.

The end goal is to extend general purpose compilers like GCC and CLANG to provide -o power optimisation modes, to produce binaries that run efficiently in power constrained environments. The project as a whole is called MAGEEC, and it looks very promising.

The FOSDEM Network

FOSDEM is a unique environment; thousands of geeks descend on the ULB campus every year to talk about all manner of interesting topics. There are no fees, and no guest registration. Running a wireless network in such an environment might be described as challenging. Only a crazy person would make it harder by making the wireless network IPv6 only by default.

Running a v6 only network by default isn’t something that’s really been done at scale before, as far as I’m aware. Networking events like IETF and NANOG have had an IPv6 only SSID for a while, but those events self-select for people who know about IPv6. FOSDEM, in contrast, is an event aimed at the open source community, most of whom don’t care about IPv6. Setting the default SSID to be IPv6 only, with transition mechanisms in place to facilitate access to the legacy IPv4 internet was a bold move; the good news is that it seems to have gone extremely well. (Modulo a bug which meant every Android user had to switch to the fallback FOSDEM-dualstack SSID. Boo!). Also, it broke anyone validating DNSSEC; which Faux, tom and I spent far too much time discussing.

In addition to talking about how this was done, there were some interesting stats about the number of unique devices, how much traffic FOSDEM sees, etc. All in all very interesting (but then again, I am a network geek).

Preventing craziness: a deep dive into OpenStack testing automation

For those who aren’t aware, OpenStack is an “open source cloud operating system”; it’s a collection of subprojects which, when taken together, allow you to spin up your own private cloud on top of computer, networking and storage resources.

This was a really impressive talk. Openstack is 2 million source lines of code spread across 95+ code repositories, with 400 contributors, all of whom can commit changes. In order to keep control over the chaos that results, the people responsible for OpenStack’s project infrastructure have put together an impressive set of processes and controls which allows them to cut a release every 6 months and to have a master branch which is always usable despite dozens of merges a day. I have reams of notes but the TL;DR is:

  • Nothing gets merged without being code reviewed by two other people.
  • Nothing gets to code review without tests passing.
  • If something is not tested automatically, it’s not tested.
  • If something is not tested, it doesn’t work.

Though I have reams of notes from this talk, I’m not going to replicate them here. Instead, I highly recommend watching the talk when it’s published; this is how to do rapid software development at scale, and the OpenStack team have either integrated or created a bunch of open source tools to make it work really well. This project has always intrigued me; now that I’ve seen how they run things, I’m even more convinced that they’re going to end up being the default open source cloud management solution.

ANSTE: Advanced Network Service Testing Environment

Testing Network Services in Multimachine Scenarios

This was a lightning talk on a framework for testing network services using a bunch of virtual machines. I don’t have reams of notes, but it did seem to be fairly straightforward to use, and provided a simple way for developers to spin up and configure interconnected VMs for the purposes of testing network applications.

Adventures with CloudStack and OpenDaylight

A talk about getting support for OpenDaylight into the CloudStack project. Some background:

OpenDaylight is an open source software defined networking (SDN) platform. What is SDN? It’s a topic for another blog post, but the short version is: a network which is configured by software applications written by your own developers, rather than by software provided by your network hardware vendor.

CloudStack is similar to OpenStack; it’s an open source cloud management platform.

This talk chronicled the experience of one of the CloudStack developers in getting support for provisioning networking for VMs OpenDaylight into OpenStack. It was quite interesting, but did serve to demonstrate that OpenDaylight still has quite a ways to go before it’s production ready.

Tunnels as a Connectivity and Segregation Solution for Virtualized Networks

This was an excellent talk, in my opinion. It’s targeted at developers who don’t have much/any understanding of networking below the sockets API.

It sought to explain the importance of tunnelling in network solutions for VM-VM communication in a virtualised environment. It started with some historical background about how Ethernet works, and why VLANs are the hammer of choice for many network engineers when it comes to segregation. It then went on to explain that in a world where networking is moving up into the hypervisor, VLANs aren’t necessarily the best choice for this anymore.

Overall an excellent explanation of how networking works in today’s virtualised datacenters; stuff that every developer should know. I’ve given variants of this talk at employers; it’s good to see this stuff being discussed in the wider open source community.

No more IPv4

A talk by IPv6 evangelist Eric Vyncke about the implication of IPv4 exhaustion; the nature of an internet with ever-increasing amounts of NAT, and how applications can be extended with relatively little work to support IPv6. Specifically, why should open source developers care about IPv6? Because sooner or later, some of their customers will only have IPv6 connectivity, or horribly broken, multiply NATted IPv4 connectivity.

As an aside: I was pleasantly surprised at the focus that IPv6 received at FOSDEM this year; both this and the earlier talk about the FOSDEM network were in large rooms and pretty full. Hopefully this reflects a growing interest in IPv6 from the wider open source community.

Using RIPE Atlas API for measuring IPv6 Reachability

RIPE-NCC is the organisation responsible for distributing number resource (IPv4, IPv6 addresses, AS numbers) to the European internet community. In addition to this primary responsibility, they also provide a bunch of data and tools to the internet community, to assist service providers with monitoring and analysing the internet. One of those tools is ATLAS, a distributed system managed by RIPE NCC used for running large scale measurement across the internet.

ATLAS is made up of a bunch of probes which run a stripped down, hardened Linux distribution which are installed in host networks around the world. These probes are then used by RIPE NCC (and members who sign up to use the API) to run tests about internet reachability from devices all over the world. The data from such tests is then made freely available to the internet community. The source code for the probes is freely available.

This talk provided some detail about what RIPE NCC and ATLAS are, and how to use ATLAS to get information about IPv6 reachability on the internet. It was very interesting; ATLAS itself seems like a very cool project; I’ve requested a probe for myself, and will probably sponsor a few more once I get one.

NSA operation ORCHESTRA: Annual Status Report

I can’t do this one justice; once the video is published, you just have to go and watch it. :)


Wow. That’s the longest blog I’ve written in a while. I may expand on some of the talks/topics raised in future blog posts; I’m definitely going to continue attending FOSDEM. The fact that such an event can be set up for free and attract 5000+ developers from all over the world continues to amaze me. See everyone next year!

(To come: my write up of Configuration Management Camp, which occurred down the road in Hogent immediately following FOSDEM).

Monday, 20 January 2014

OpenJDK Hackday

Since I am currently possessed of an excess of free time, I’m always looking
for events to participate in which will be entertaining and/or educational.
Yesterday, this led me to attend a hackday run by the London Java

These hackdays are based around OpenJDK, the reference
for Java Standard Edition. People generally work on
whatever they like (it is a hackday, after all), but one of the main goals of
these regular hackdays is to provide an environment where neophytes to OpenJDK
development can try and start making contributions. There are a handful of
active contributors to the OpenJDK codebase around to provide support and
guidance to people getting familiar with the language.

For myself, the last time I wrote Java code was in my final year of university,
which was a Very Long Time Ago. I also have no burning desire to pick up Java
development again. This leads to an obvious question: what exactly am I doing
spending time at a hackday for the reference implementation of Java?

The short version: it’s all Richard’s fault.

The slightly longer version: Richard had mentioned the OpenJDK hackday at the
pub a few weeks ago, and tried to dragoon several of us into coming along. When
I objected that I wasn’t a Java developer (arguably I’m not any kind of
developer, but that’s a larger problem), he promptly shut me down:

“It’s fine! We need help sorting out build tools and stuff. Chef! Vagrant!
Devops! You know about that kind of low level rubbish!”

(I may be taking some liberties with the details, but I maintain that this
captures the core of his argument.)

And so it was with some trepidation that I made the long trip to Shoreditch on
Saturday morning; not entirely sure what to expect, and somewhat doubtful that
I would be of any use. I needn’t have worried. Everyone at the event was very
welcoming, and it turns out that one of the main problems that needs solving
doesn’t require any Java knowledge at all. (Namely: the problem of providing a
reproducible working build environment so that new contributors can quickly get
stuck in to OpenJDK develpment.)

We very quickly formed an “infrastructure” room, where between 5 and 10 of us
over the course of the day worked to try and make progress on producing a
deterministic base virtual machine and a build environment to
go on top of it. I learned a great deal, and found myself actually able to make
some meaningful contributions.

There’s still lots of work to be done, and I suspect I’ll be working on making
a better OpenJDK development environment for some time to come.

Sunday, 15 December 2013

Global Day of Code Retreat 2013

Have you heard of Coderetreats? I hadn’t until a couple of weeks ago, when a friend of mine prompted me to try one. From the About page:

Coderetreat is a day-long, intensive practice event, focusing on the fundamentals of software development and design. By providing developers the opportunity to take part in focused practice, away from the pressures of ‘getting things done’, the coderetreat format has proven itself to be a highly effective means of skill improvement. Practising the basic principles of modular and object-oriented design, developers can improve their ability to write code that minimizes the cost of change over time.

I signed up for the Global Day of Coderetreat session and then promptly forgot about it until yesterday, when I realised that I was going to regret my 0600-1200 sleeping pattern today. (Other than making it difficult to interact with normal people, however, it’s awesome.)

Despite my lack of sleep I dragged myself to the depths of Shoreditch where Unruly Media, our hosts for the day, make their home.

To summarise the format, the day is broken up into multiple 45 minute sessions. All sessions revolve around pair programming a solution to Conway’s Game of Life. There is a preference for using test driven development, and a hard rule that all code must be deleted at the end of each 45 minute session.

We rotated pairing partners throughout the day, and there were no language restrictions other than the basic sanity check of “at least one of the pair should be comfortable with the language you’re using”. I only wrote in Python and Go, since my recent programming experience is limited to those languages, but people worked in Java, Javascript, Haskell and (I think) a couple of other languages as well.

In addition to the basic rules, some of the iterations had constraints placed on them, in order to stimulate thought about the problem domain or just for the entertainment of the facilitators. (Mute immutability springs to mind. :P)

Was it worth it? I certainly think so. A day of practice with no pressure to deliver useful code to a customer was very refreshing; I felt it enabled me to think about the actual problem being solved as well as the various approaches that could be taken. The enforced deletion of code as well as changing pairing partners during the day lead me down several different tracks of how to solve this problem.

The day was concluded with a wrap up, where everyone was asked three questions:

  1. What, if anything, surprised you today?
  2. What, if anything, did you learn today?
  3. What, if anything, will you do differently in the future?

Speaking for myself:

I was surprised at how effective pair programming is. This was my first experience with it, and I was expecting it to be stifling (having to wait for other people, not using my own development environment). It transpired that these worries were largely unfounded (Alex’s shocking vim skills not withstanding; he has been directed to the appropriate learning resources). Having someone to bounce ideas off meant that at no point did I ever get stuck; if I got to a point where I wasn’t sure what to code next, my partner generally had a concrete suggestion. If neither of us were quite sure how to progress, 30 to 60 seconds of chat usually resolved this and we were back to making forward progress.

In terms of learning: I think for me, the biggest lessons were simply in collaborative coding. As a network engineer, most of my coding since leaving university has been knocking together code that just about works to solve specific problems, with a couple of exceptions where I’ve worked with full time developers on specific projects.

Things I’ll do differently: I will try and pair program more, when given the opportunity; today has definitely given me an appreciation for its benefits. It also brought home the real benefits of test driven development; one of the exercises had the stipulation that one partner wrote the test code, and the other partner the implementation. This turned out to be a good way of forcing clarity of requirements, and having the tests in place gave me much greater confidence when making changes, which was particularly important given the time pressure.

Overall a very enjoyable and productive day; for those looking to practice the art of programming, you could certainly do worse than attending a code retreat. Many thanks to Rachel Davies for organising the event and keeping us fed and watered, and to Steve Tooke for facilitating. I look forward to next year.

Written with StackEdit.

Tuesday, 3 December 2013

tmux and vim for Scheme development

I’m currently working my way through Structure and Interpretation of Computer Programs, as a lot of people I know speak very highly of it as a text on the craft of programming.

It uses the Scheme programming language, a dialect of Lisp. Most tools for developing in Lisp and Scheme (e.g. SLIME) are based around the emacs text editor. Without going down the rabbit hole, suffice it to say that I’m a vim user, and wanted an alternative.

My first thought was to use slimv.vim, which apparently provides vim users with similar functionality to SLIME. Unfortunately, my attempts to get slimv.vim to play nice with mit-scheme on OSX lead to frustration; I may revisit that at a later date, but in the mean time, I needed to get something working.

Enter tmux. tmux is a terminal multiplexer in the vein of GNU screen. One of its main benefits over screen is that almost every action in tmux is scriptable via the command line, which lends itself nicely to automation and integration with other programs.

For example, we can load the contents of a file into a tmux paste buffer…

$ tmux loadb <filename>

… and then paste the contents of a paste buffer into a specific pane in tmux.

$ tmux pasteb [-t [<session>:]window[.<pane>]]

If a target isn’t specified, the currently active pane is used. Windows can be specified by name or by number.

To create a new pane:

$ tmux splitw [-h] [cmd args...]

(The -h creates a side-by-side split; by default, tmux splits are stacked vertically.)

Finally, we can create a mapping in vim, which saves the current file and then shells out to tmux, reads in the contents of the current file, and pastes it into a target pane.

:map ,t :w \| !tmux loadb % && tmux pasteb -t 0.1

I can now have a scheme REPL side-by-side with my editor, and whenever I hit ,t my code will be saved and run through the REPL.

A quick screencast to demonstrate:

Written with StackEdit.

Network notes

As mentioned previously, my good intentions to blog as I learned interesting stuff didn't survive contact with my new employer.

Now that I'm unemployed, I'm going to be writing more frequently here; the main intention is for this to be a memory aid, but if other people find this stuff interesting, so be it.

First up: in preparation for my Google interviews, I put together some revision notes on some fundamentals of networking, which may be of more general use. You can find them on my Github page:

Saturday, 12 October 2013

Changing plans

[Originally posted on Google+ on 2013/10/12.]

I'm writing this from the quayside having just watched my crew sail away. Best of luck guys - sorry I'm not with you.

It's a little strange being sat here - as most people know, I was planning to go the whole way round the world on this year's Clipper race. However, between the sailing, cooking, cleaning and everything else, the last 31 days at sea have given me a lot of time to reflect. For me, the races to Brest and Rio were nice - but only nice. I've discovered that I enjoy sailing, but not enough to spend a year of my life crossing oceans for a month at a time with only a week's downtime in between. Had I stayed on, I'm pretty sure that 'this is nice' would have faded to apathy and possibly worse. Having crossed the Atlantic and the equator, the time has come for me to step off with a bunch of great memories and lifelong friends.

What now? I'm going to head back to the UK and spend a couple of months with my family before looking at getting back to work; if you're in London, drop me a line. We'll be going out to Cape Town to welcome the fleet in, too.

Good luck, Team Mission. See you in Cape Town.

Saturday, 8 June 2013

Blogging is hard

Alternatively, it's entirely possible that I'm just lazy and/or easily distracted.

I started this blog intending to record all the various things I was studying, due to a lack of technical challenge at work. Things didn't really go as planned. A lot has happened in the almost 3 years(!) since I've last published a post here; however two things in particular stand out.

Firstly: new job. In May 2010, I joined Google Dublin as a network engineer. The first couple of months were equal parts exhilarating:
"Wow, that's cool!" 
and terrifying:
"Wait, what? I'm on-call for that?!"
Both of those reactions still occur on a semi-regular basis.

A vastly more stimulating work environment meant that I was spending more time at work, and I was learning enough that I didn't feel the need to push my self study as hard, and hence didn't have a reason to be writing blog posts. The last 3 years at Google have been an absolutely fantastic experience.

The second thing... well. The second thing essentially involves me taking leave of my senses.

Before being hired by Google I was pondering taking a couple of months of leave from the old job to do something different, having never done the gap year/travelling thing. One of the things that I looked at was sailing, since after spending time in the Royal Naval Reserve while at university, I missed spending time at sea. During my pondering, I found out about the Clipper Round the World yacht race; those of you in London have probably seen posters in tube/train stations at some point. I sent out for an application form, thinking about doing a leg or two in the 2011-2012 race. Then Google came along, and I promptly forgot all about it.

Fast forward to two years later, and Clipper sent me an email about the 2013-2014 race. Fair enough, says I. After a bit of thinking, I applied and was interviewed (more on that in another post hopefully), and after setting foot on one of the Clipper 68s, I decided to go all in and do the full circumnavigation in 2013/2014.

As I said. Taking leave of my senses.

A lot more has happened since then; there has been training, theory courses, recreational sailing in Dublin, finally meeting my crew last month, preparing to move back to the UK... those will all (hopefully) be covered separately.

I've been incredibly lax about journalling what's been going on, which I'm sure I'll regret later. Better late than never, I suppose. The plan is to record everything that's going on in the run up to the race here. I'll try and intersperse some stuff about the earlier stages of this adventure, but my memory being what it is, those posts will probably be somewhat lacking on detail. Bah.

85 days and counting...