This is just a little anecdote. Don't expect it to be deep or meaningful. I don't know, maybe there's something in here about feedback loops or focussing development effort in the right place, whatever.

Way back when, when I was just about fresh out of university and in my first programming job (1996), I got a bit of a reputation at being quite good at making our database and its applications go fast, including SQL query optimisation. This was using Ingres databases, hosted on VMS.

One day I was asked to investigate this particular stock report (I even remember the report id, “AGR48”, though I confess I have no idea now what the report was actually for). The report was too slow, I was told.

So I dug.

The report consisted essentially of a sequence of 8 giant SQL statements, each of which was quite slow. Lots of creation of temporary tables and the like. I think I rewrote the report and did make it faster, but that's not the point.

The point is: the report took about 4 hours; but due to a bug in the eighth and final SQL statement, the report output was always empty.

This report was effectively a four-hour no-op.

Unsurprisingly, the users didn't use that report.

Last week I attended QCon London, a “Conference for Professional Software Developers” run by InfoQ. Three days of keynotes, presentations, facilitated discussion, and general open mingling with other delegates and and speakers.

It's the first time I've been to this conference. I went because last year my colleague Stephen went, and I could tell from his experience there, and from the online videos of the talks published later, that this was worth going to. So I signed up for this year at the earliest opportunity.

It's well known that the technology sector has a diversity problem. Go to almost any technology event in Europe / North America, and you'll see overwhelmingly white male faces.

Today is International Women's Day 2015: so what have I learnt of the diversity aspect of QCon, and about the inclusion of women in particular?

Do I expect I'll go again next year?

The speakers

For those used to technical conferences, you might be interested in taking a quick look at the diversity on show amongst the speakers at QCon London this week.

When I checked and did a rough count, there were 123 speakers listed, and quite a range of faces.

Or, so it might appear at first glance.

I very roughly counted males and females, white and non-white. By my count, this year's QCon speaker line-up included:

• 3 (2.4%) non-white women;
• 5 (4.1%) non-white men;
• 24 (19.5%) white women;
• 91 (74.0%) white men.

Only 74% white men? For the technology sector, that's actually pretty good! So, that's like, 26% of “doing diversity”!

Make no mistake: it's a lot better than most conferences, by which I mean that it's closer to being more representative of the population as a whole.

But that's still 78% male, and the world simply isn't like that. (In case it needs saying: it is, of course, 50% female, 50% male, give or take). And this lineup, while being more diverse than we've come to expect, is still 93.5% white, compared to 87% in the UK population as a whole.

Which doesn't sound so bad, but does mean that if you're attending QCon, and you're non-white, and hoping to see non-white speakers, then instead of the 16-ish speakers you should expect to see, you in fact see only 8; only half of what it should be.

78% male. 93.5% white.

In this sector, however, 22% female counts as so far above the norm that it won't have happened by accident, which implies that the organisers took deliberate steps to include more women, which implies that they care about diversity — which is great. But at the same time we all need to recognise that it's still not enough, and we should demand more.

The audience

The organisers don't collect the diversity profile of the audience, so that information is harder to assess quantitatively.

I tried two different (but similar) ways of measuring the audience diversity, and I admit, both methods are highly unscientific. Firstly, as I sat waiting for the keynote, I just looked around at the people sat nearest to me. Secondly, I took a photo of a larger area of the audience, and counted faces later.

Doing a quick sample of the 50 people sat around me while we waited for Friday's keynote, it looks like about 16% female, 84% male. Which again, is a long way away from 50/50, and it's notably even more skewed than the speaker line-up.

As for white / non-white: again, based on the people sat around me: roughly 88% white, 12% non-white. So actually, on this highly unscientific sample, this one's pretty much on-target, matching the figures for the UK population as a whole.

The count of faces in the photo came to 82 white male, 5 non-white male, 2 white female, 3 non-white female. That is, 95% male, 91% white.

On inclusion

I noticed before the conference started that QCon publishes a code of conduct, and it's nice and clear, concise, and good to see. And with one exception, I neither witnessed nor heard of any harassment or anything else that would violate the code of conduct.

We'll come to the exception later.

People with special mobility requirements were pretty well catered for (but it was hard to work out how to get to the 6th floor). Several people noted how the speakers would be using “she” or “her” more, to refer to actors in their stories (such as: “Your CTO says <x> and she knows what she's talking about…”), instead of just thoughtlessly going with male as the default. Speakers' slides would include little stick figures to represent people — and those stick figures were often women.

All in all, a huge step (compared to tech industry norms) in the right direction. Sadly, the free tee-shirts being given away by the sponsors were obviously not made for women.

And, then… the exception.

On-stage transphobia and its effects

Just before the keynote on the last day, one of the track hosts, John T Davies, made transphobic remarks whilst on stage. It only took a few seconds, but in that moment, much of the good work that QCon had done was very quickly undone: I no longer felt completely welcome, or safe, or included. I felt threatened. At risk.

During the next presentation, I saw the official twitter account post an apology.

In the mid-morning break, I went to speak to the organisers on a different matter, but ended up talking to them about the on-stage comment. I spoke to Silke D'Alessandro, and we were joined by Floyd and Roxanne, founders of InfoQ, and Nitin Bharti; and it was very reassuring to see their obvious concern over the incident, and to see their commitment to diversity and inclusion. They said that they'd spoken to Mr Davies about the incident, and that ahead of his own talk this afternoon, he'd make an on-stage apology.

This conversation was great to have, but at the same time, it's not why I came to QCon: I was here for the conference. So I felt frustrated that, because of what was said on stage, I felt unable to participate in this part of the conference, because of this unwanted distraction.

I joined the next session late (having missed the first half); had lunch, ate alone, my head still full of these distractions. I went for a walk outside, just to forget this, to be an anonymous tourist for half an hour.

Back at the conference for the afternoon, and the next two talks went well. For the third afternoon slot, I popped along to see Mr Davies give his on-stage apology (before I would then quickly switch rooms to go to the talk I actually wanted to hear). Unfortunately, for whatever reason, his apology came across as diminishing and insincere.

I went to talk to the organisers again, to point out that the apology was not good enough. In fact, the apology itself was harmful. And all credit to them: once again, Floyd, Roxanne and Silke said all the right things, completely understood the problem, and said that they'd be taking further action.

I thanked them, and left. But by this point I was furious. I went to hide, to vent, to calm down. As a result of just a few seconds of offensive content on stage this morning, I was missing a significant proportion of the conference.

I managed to catch to the last talk of the day, and then, there was just one more item on the schedule: “Meet the speakers”, where the conference hosts and speakers are encouraged to mingle with the other delegates, and chat, share ideas, be creative. And it occurs to me: one of those other speakers is Mr Davies. Am I ready for that?

I seriously consider just going home. After all, there are no more talks. I could just slip away: many other people are doing so. I could too.

But I shouldn't have to. I came for the conference. This is why I'm here.

Meeting the speakers — and more

So I went. I grabbed a beer. I mingled.

Unlike the other “networking opportunities” — the coffee breaks, lunch, and so forth — where the mingling seems quite random, here it seemed far from random.

A lovely lady named Vanessa came up to me, to talk about this morning's incident. We compared notes about this conference, and past conferences, and how women are treated in the industry.

Then Roy Rapoport (of Netflix; this morning's keynote speaker) found me, and again, we're talking about the transphobia, comparing notes. About how he heard the comments, just before he was due to go on stage, so has to choose: make reference to it on stage, or act as if nothing happened? (He chose the latter, and I don't blame him). About he felt offended by the remarks too.

Then Floyd Marinescu (InfoQ) again, this time asking if I'm prepared to meet with Mr Davies. I know it could well be constructive to do so; and I know it's also fine if I say no. Even as I weigh up the decision, right there and then, I can feel the emotion, the anger, the frustration, the fear welling up in me: I'm still far too emotional about it, and so I decline.

Then I do some proper mingling: back to what the conference is meant to be about. At last.

As things are thinning out, I met with someone — who I shan't name — involved with organising the conference, who offered an opinion on Mr Davies' prospects of being invited back again. Enough said.

Then, finally, just as I'm leaving, another lady (whose name I didn't get) comes up to me, and again, it's to ask about the incident this morning. So we chat, I do my best to explain what happened, and about some of the following chain of events.

And then, the conference is over. Finally, it's time to go home.

Fallout

InfoQ's commitment to diversity at QCon is clear, and is to be congratulated. I wish that more conferences and events were like this.

But, transphobia was on stage. On show.

Make no mistake: incidents like this are exactly why diversity struggles in STEM fields. It drives people away.

What frustrates me so much about this is the uneven effect that such incidents have: I'm assuming that of the 1400 or so people attending, approximately 1399 of them didn't then spend a significant proportion of the day not focusing on the conference because of this.

Straight White Male is the lowest difficulty setting there is.

The closer you are to being a cis straight white male — the more the existing biases of the technology sector pander to you — then the less likely you are to find something that offends you, the more likely you can just get on with your job.

So the transphobia, on the whole, won't have upset non-trans people, and they'd have been able to get on with their day as normal. In fact, they might not have even noticed that there was a problem.

In contrast, as a direct result of the incident on stage, I missed a huge chunk of two talks, was greatly distracted from the others, missed several of the breaks and other opportunities to network, and even the “meet the speakers” mostly consisted of people wanting to talk to me about the transphobia — not about technology, which is nominally what we're all there for.

What next?

The diversity at QCon, compared to the industry as a whole, was good, so it's clear that InfoQ are trying. At the same time, I believe they can try harder: 78% male is still too high, and they are in a position to change that.

I'll admit, I was disappointed to see so few people speak up about the transphobia. If you see it, call it out. Speak up. Take action. Even if the offence isn't directly at you, it affects you, because it negatively affects diversity, and we all know that diversity — that is, having the make-up of the people in the industry reflect the make-up of the population as a whole — is a good thing.

Will I go back to QCon in future? Probably. The conference was well-run, with good content, and InfoQ's commitment to diversity is clear to see, not only from the speaker line-up, but also from their reaction to the unfortunate event of Friday morning.

But in future, can we get women's tee-shirts too? That'd be just great :-)

Update, Monday 9th March: On Friday, Floyd sent me Mr Davies' written apology, asking for my thoughts. I'm sad to say that this written apology was deeply troubling in its tone — "completely unacceptable" is I think also accurate.

March 8th is International Women's Day, celebrating the achievements of all women and calling for greater equality. Together we can make it happen.

Source for the speaker list: the QCon web site

Recently I've been repaying some tech debt that I had built up at home. I've been sorting out old files, arranging automated off-site backups, committing uncommitted files to git (including finally getting myself some private git hosting), simplifying config, upgrading Debian, upgrading my Internet connection, and adding IPv6 support to my home network. It's been a fun few weeks, actually.

The last one is interesting: adding IPv6. What does “add IPv6” really mean, and why should we do it?

Say “AAAA”

Assuming you've already got a working IPv4 setup, then adding IPv6, at its simplest level, is:

• ensure each host or service has an IPv6 address;
• ensure those IPv6 addresses are discoverable via DNS, by adding “AAAA” records.

Back in January 2011, there was “World IPv6 Day”: some big names (Google, Facebook, and quite a few others) temporarily added IPv6 addresses to their systems, and added “AAAA” records to DNS: just for a day, to see what happened. In essence, it was successful: some of their visitor traffic was carried over IPv6, and nothing broke. Then, in June 2012, there was “World IPv6 Launch Day”: basically the same again, but with more participants (e.g. Wikipedia), and this time the configuration wasn't removed afterwards: it was left in for good.

Because I've recently been adding IPv6 to my little part of the Internet, I've been repeating the exercise myself. Once I'd got myself an IPv6 allocation, and my home network was handing out IPv6 addresses, I could “ping6 google.com”, for example. I'd solved the first part: adding addresses. Now for the second: discoverability in DNS.

Servers and Nameservers

First, the easy part: each of my servers has an “A” (IPv4) record in my DNS records, so that the server can be found without having to remember IPv4 addresses. So all I had to do was add corresponding “AAAA” (IPv6) records. One quick visit to my provider's DNS control panel later, and that's that job done: I can now resolve my server name to an IPv6 address, and thus connect to it. So, for example, “ssh -6 myserver.rachelevans.org.uk.” now works.

This is where things start to go downhill a bit.

What did we just do here? I allowed my server to be addressed via IPv6, and I added its IPv6 address to DNS.

Does that mean that, if had an IPv6 Internet connection and didn't have an IPv4 connection, I could connect to my server? No, it doesn't. At this point, I'm still dependent on IPv4.

To understand why, we need to understand a little more about how DNS works.

Zones and Referrals

To resolve a computer name to an IP address, we use DNS. It's a distributed system, whereby lookups start at the “root” zone, and each zone can delegate authority to sub-zones. For example, “.com” is a sub-zone of the root, and “example.com” is a sub-zone of “.com”. Each zone has a set of nameservers.

Fundamentally, the problem is this: for DNS lookups over IPv6 to be successful, each zone has to provide at least one nameserver with an IPv6 address, and those IPv6 addresses have to be in DNS — otherwise that zone will be inaccessible to the IPv6 Internet.

(Exactly the same thing is true of IPv4, by the way: if none of a zone's nameservers have IPv4 addresses, that zone won't be accessible to the IPv4 Internet. However, so far everyone seems to get it right for IPv4, so it's less interesting).

So for our web site to be reachable to clients purely via IPv6, what do we have to do?

Say our web site is www.example.com, and our DNS zone is example.com. So we have to:

• give our server an IPv6 address;
• add an entry mapping its name to its IPv6 address as an “AAAA” record, in our DNS zone;
• ensure our nameservers are reachable via IPv6.

How exactly do we do the last step? Simple: we repeat the whole process again. Say for example the nameservers of the example.com zone are ns1.big.hosting.co and ns2.big.hosting.co — we repeat the process for those two servers (or, more typically, we have to talk to the bighosting.co in question to check that they have already done this).

And then, in turn, bighosting.co itself has nameservers, and the whole process repeats.

An example

Back to what I was doing over Christmas: I was adding IPv6 support to my network, and then adding the IPv6 “AAAA” records to DNS. But, there's no point doing that unless the nameservers are also reachable via IPv6, which got me to thinking about this whole area.

So I wrote a tool I called “dns-dependency-walker”: it analyses the DNS hosting of a given domain, and finds its dependencies, and analyses, them, and so forth. It then draws a directed acyclic graph illustrating the relationships between the various zones and nameservers involved.

For example, here's the picture for rachelevans.org.uk:

(Nameservers or zones that are not reachable by IPv6 are shaded in grey. See my note to the side of the diagram for a detailed explanation.)

My domain is rachelevans.org.uk, so I depend upon the parent zones, org.uk, uk, and the root. The nameservers of my zone are in the domain buddyns.com, so I'm dependent upon that zone too. In turn, buddyns.com is dependent upon the .com zone. And so forth.

The moral of the story is: the success of your IPv6 support could depend on more than you might expect.

Let's do this!

So back to World IPv6 Launch Day. Their web site lists the companies that participated, adding support for IPv6.

Let's pick a few of the big names and see how they're doing.

Akamai

Not all of akamai.com's nameservers support IPv6, but overall, it's reachable. So far so good.

Microsoft

Slightly better for microsoft.com: all of that zone's nameservers support IPv6. It doesn't matter that some of the zones that they depend on have some IPv4-only nameservers, because they also have IPv6-capable nameservers. So again, the result is: it's reachable.

Facebook

Here's where it starts to go wrong. Although www.facebook.com does indeed have an IPv6 address, the facebook.com zone's nameservers don't. So if you, or more specifically your DNS resolver, don't have an an IPv4 address, then you can't find out what www.facebook.com's IPv6 address is — therefore you can't connect.

Google

Exactly the same situation at Google: www.google.com does indeed have an IPv6 address, but you can't find it without an IPv4 address, because none of the google.com nameservers have IPv6 addresses.

Cisco

At first glance, Cisco appears to have the same problem as Facebook and Google — lack of IPv6 addresses on the nameservers. But actually, it's a bit more complicated.

The nameservers do have IPv6 addresses, as we can see from this query:

rachel@dudley ~$ dig +short aaaa ns1.cisco.com
2001:420:1101:6::a
rachel@dudley ~$ dig +short aaaa ns2.cisco.com
2001:420:2041:5000::a

So what have Cisco got wrong, exactly?

The answer lies in DNS “glue” records: whenever the name of a DNS server (“ns1.cisco.com”) lies within the zone that that same server is trying to serve (“cisco.com”), you have to add the nameserver's IP addresses — the “A” and “AAAA” records — to the parent zone. These are called “glue” records.

Here's microsoft, getting it right (for the domain where their nameservers live, msft.net):

rachel@dudley ~$ dig @a.gtld-servers.net. a www.msft.net.

;; QUESTION SECTION:
;www.msft.net. IN A

;; AUTHORITY SECTION:
msft.net. 172800 IN NS ns3.msft.net.
msft.net. 172800 IN NS ns1.msft.net.
msft.net. 172800 IN NS ns2.msft.net.
msft.net. 172800 IN NS ns4.msft.net.

;; ADDITIONAL SECTION:
ns3.msft.net. 172800 IN A 193.221.113.53
ns3.msft.net. 172800 IN AAAA 2620:0:34::53
ns1.msft.net. 172800 IN A 208.84.0.53
ns1.msft.net. 172800 IN AAAA 2620:0:30::53
ns2.msft.net. 172800 IN A 208.84.2.53
ns2.msft.net. 172800 IN AAAA 2620:0:32::53
ns4.msft.net. 172800 IN A 208.76.45.53
ns4.msft.net. 172800 IN AAAA 2620:0:37::53

The response tells us which servers handle the zone msft.net, and then also tells us their IPv4 and IPv6 addresses.

By way of contrast, here's Cisco getting it wrong:

rachel@dudley ~$ dig @a.gtld-servers.net. a www.cisco.com.

;; QUESTION SECTION:
;www.cisco.com. IN A

;; AUTHORITY SECTION:
cisco.com. 172800 IN NS ns1.cisco.com.
cisco.com. 172800 IN NS ns2.cisco.com.

;; ADDITIONAL SECTION:
ns1.cisco.com. 172800 IN A 72.163.5.201
ns2.cisco.com. 172800 IN A 64.102.255.44

The response includes the IPv4 addresses of the nameservers, but not the IPv6 addresses. (The cisco.com zone does contain the nameservers' IPv6 addresses, but that's no use unless you can find the nameservers in the first place).

Why have these big companies not got it right yet?

So with all of this — multiple dependencies between zones and nameservers, referrals, glue records — does that mean that adding IPv6 support in DNS is somehow hard? That is, harder than IPv4?

Not at all —when it comes to DNS, the rules for IPv6 are exactly the same as they are for IPv4.

So why do so many companies — even the ones who promoted World IPv6 Launch Day, back in 2012 — get it wrong? What exactly was the point of it all?

To answer that, let's consider the following scenarios:

If everyone on the Internet has an IPv4 address already, is there any benefit to me if I add IPv6? Well, a little: addressability without NAT, for example.

But, what if not everyone on the Internet has an IPv4 address — what if some only have an IPv6 address? Is the result just that they can't find a lot of sites (including Google, Facebook, etc) because they can't get the DNS lookups to work?

Maybe. But there is a mitigating factor.

Often, if your computer (PC/laptop/phone) needs to do a DNS lookup, it won't do the full lookup itself — rather, it'll ask another computer to do it. Internet Service Providers will often provide such a service, technically called a Recursive Resolver or Full Resolver (but often lazily, and confusingly, called just a DNS Server). As long as your computer can reach that one using IPv6, and as long the Resolver does have an IPv4 address, then that also works. In other words: your computer might be able to manage without an IPv4 address, as long as it can reach a Resolver that does have one. But that's a hack.

But what if even the DNS Resolver doesn't have an IPv4 address? Well, then you're out of luck: you'll only be able to find those servers whose DNS zones are reachable via IPv6; and additionally, you'll only be able to see those web sites where the web site itself also has IPv6.

If either piece of the puzzle is missing, you miss out.

Why IPv6?

So back to my original questions: What does “add IPv6” really mean, and why should we do it?

Adding IPv6 isn't just assigning an IPv6 address, then bunging that it DNS.

It is ensuring that you support IPv6 at least to parity with IPv4, and ensuring that all the services that you depend on (usually provided by other companies and vendors) do the same.

Adding IPv6 in parallel with IPv4 does add some value in and of itself; but as you'll probably have heard, one of the big selling points behind IPv6 is that IPv4 addresses have run out (or “will soon run out”, depending on your definition), and when we're out of IPv4 address, newcomers to the Internet will only get IPv6.

The key reason therefore to add IPv6 support is for the benefit of people in the future. Increasingly, the IPv6 Internet will be the Internet, and as more and more of the people that you want to talk to are on IPv6, you'll be cutting yourself out of the picture if you haven't joined the party yet.

Some of the efforts so far to encourage IPv6 adoption have, perhaps necessarily, omitted some of the technical details. Companies have declared “We're on IPv6,” without really telling us what they mean — perhaps because they themselves didn't know.

Testing, Testing, I P 6

As you add IPv6 support to your Internet presence — your web site, email systems, and so forth — it's highly advisable to check that it's reachable even to people without IPv4.

Set yourself up a test area, on its own Internet connection, only with IPv6. Make sure you don't use your ISP's (or corporate) DNS Resolver service – because if you do, you might accidentally be depending on IPv4 without realising it. Reboot all the devices in the test area, to clear any DNS caches.

Congratulations: you've weaned yourself off of IPv4. Now, can you still access your web site? Can you still communicate with your company via email?

If you can, you're way ahead of the curve. Welcome to the future.

Last month at the AWSUKUG meetup I talked about Video Factory, and there was a little section there where I spoke aboutthe tooling that we use to manage all of our components. One of the tools, “stack-fetcher”, generated quite a bit of interest from the audience, and there was interest in open-sourcing it. I definitely want to do this — but we're not quite there yet.

For now, though, I can talk about where stack-fetcher is right now, and what direction I want to take it in.

The problem space

“AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion,” says the documentation. As a developer, you do this by creating a template (JSON which defines one or more desired resources), then submitting that template to CloudFormation — either via the API, or via something which wraps the API (e.g. the web console). Then CloudFormation goes and creates or updates your stack to match your template.

As a developer who loves automation and consistency, this leaves you with several problems:

• How do I generate the template JSON?
• How do I generate the other JSON required by the stack (e.g. parameter values)?
• If I was to push that JSON to CloudFormation — i.e. apply the change — how do I know what changes I'm actually pushing?
• Can I push some changes but not others?
• Once I know what I want to push, how do I do so?

A little BBC Media Services history

To put all of the above into a specific story: in BBC Media Services, we found during the development of Video Factory that we were managing more and more stacks, and by the start of this year we had something like 100 stacks to manage in each of our three environments.

By January 2014, we had a system for generating the JSON, but different people ran the relevant tools in different ways, therefore sometimes yielding differing results. And once the JSON had been generated, we had no way of knowing in what way it was different from the stack's existing template, so we didn't know what we were actually changing. And finally, we had no consistent approach for actually updating the stacks with the new template — mostly we were using the web console, but not always in the same way. And even then: it's a web console, so that's just awful from a productivity and automation point of view.

Thus, stack-fetcher was created, to address all of the above problems.

The workflow

Once you've updated your source files, the workflow to update a stack consists of three steps:

• Run “stack-fetcher”. This generates a set of three files: current, generated, and next.
• Use your favourite diff/merge tool to compare the current, generated and next files, making whatever changes you wish to next.
• Run “stack-updater” to push next into CloudFormation.

The workflow in action

Here's a demo of a simple change, illustrating the basic workflow, and some of stack-fetcher's strengths.

Before running stack-fetcher, We have two stacks, “resource” and “component”. The first diff has already been applied: a queue was added to the resource stack. These screenshots show the second diff being applied: to modify the IAM policy defined in the “component” stack, such that access is granted to the queue in the resource stack.

We then run stack-fetcher (in this example, “int” is the environment in question — integration). stack-fetcher retrieves the existing stack, generates the desired template, and compares the two. The summary shows “resource: same” (all in sync), and “component: DIFFERENT (20 lines)” (there are 20 lines of differences).

stack-fetcher has generated three template files per stack: current, generated, and next. Here we see the three files compared, using vimdiff:

and the bottom half of the same files:

You can see that “generated” (in the middle column) has some sections that “current” doesn't — these is for the policy change we're trying to make. But you can also see that “current” has some lines that “generated” doesn't. This is because in this example, the stack in CloudFormation started off not in sync with our local copy (for example, maybe someone applied a change but neglected to commit the corresponding source).

So now we modify “next” (the right-hand file) to match whatever changes we want to apply. In this example we choose to pull in the new lines, but elect not to remove the extra, unexpected ones:

After saving these changes (remember, we didn't modify “current” or “generated” — only “next”), we run stack-updater:

stack-updater now warns us that it has detected a new parameter on the template (“MattressFailQueueArn” in this example): it adds this parameter, with the default value from the template, to the description file; then invites us to check this and edit the description file if we wish.

In this case the default is fine, so we just run stack-updater again:

Now stack-updater very clearly shows us the diffs between current and next: that is, if we elect to proceed, these are the changes that we're actually about to make.

After confirming that we're OK with this, stack-updater applies these changes, using the CloudFormation UpdateStack API:

stack-updater polls the stack's status, waiting for it to reach a terminal state (i.e. not “in progress”). The stack events are displayed as they occur.

In this case the stack update completes successfully, and stack-updater's work is done.

In more detail

stack-fetcher is a name given to a collection of scripts, one of which is itself called “stack-fetcher”. The other script that is intended to be manually invoked is “stack-updater”. There are other scripts, but one of the goals of stack-fetcher is to invoke and orchestrate those other scripts so that the user doesn't generally have to think about them.

stack-fetcher

stack-fetcher's job is to generate a set of three outputs:

• current is the existing stack, fetched from CloudFormation
• generated is the stack that you want, generated from your codebase
• next is what you're going to push back to CloudFormation using “stack-updater”

When stack-fetcher runs, next is generated simply as a copy of current — that is, if you don't edit the next file, then you won't push any changes.

To make generated, stack-fetcher runs a series of scripts. Currently, this step is rather BBC-specific: we invoke ./generate-templates with PYTHON_LIB set to point to part of the stack-fetcher codebase; if there's a transform script, then the json is then filtered through this; then there's a cosmos-cloudformation-postproc script which post-processes the json in various ways — primarily, providing defaults for the stack's parameters.

To make current, stack-fetcher needs to know what stack name it should work with — and again, currently calculating this stack name is fairly BBC-specific. Once entered, the stack name is remembered via the ./stack_names.json file, so you don't have to calculate or enter it again. Once the stack name is known, the existing stack template and descriptor are fetched, and saved as current.

After this, stack-fetcher normalises both current and generated. The purpose of the normalisation is partly to make the files more readable, but also to get rid of differences that are meaningless. As well as whitespace reformatting and sorting object keys, the normalisation also includes CloudFormation-specific elements, such as sorting parameters, tags and outputs; removing empty arrays, if that would mean the same thing; and even re-ordering statements within IAM Policies.

next always starts off as a copy of current, so that by default no changes are pushed.

Finally, stack-fetcher compares current and generated and shows a simple summary: they're either the “SAME” or “DIFFERENT” (or, if the stack doesn't exist yet, “NEW”); then shows some help text describing what to do next.

diff/merge

The help text displayed by stack-fetcher suggests using vimdiff to compare and edit the files, but of course you can use whatever tools you wish. The goal of this step is to update next to reflect what you want pushed back into CloudFormation (whilst leaving the current and generated files unchanged).

You may wish to simply review that generated is exactly what you want, then copy generated over next (this is probably what you want, ideally); or, you can cherry-pick, and perform more complex merges.

stack-updater

Once you've updated next to be as desired, you invoke stack-updater, with exactly the same arguments as you did for stack-fetcher.

If there are any differences between the set of parameters declared in the stack template, and the set of parameters passed in the stack descriptor, then stack-updater shows those differences (e.g. “You're passing a parameter called X but it doesn't exist”), automatically applies corrections (e.g. removing the no-longer-existent parameter), then stops, so that you can check its changes before re-running stack-updater.

Assuming the stack already exists, then stack-updater now diffs current against next — that is, it shows you the changes you're about to push. It also displays the differences between the stack's parameter defaults, and the actual parameter values you're passing, so you can check which ones you're overriding. (If the stack doesn't currently exist, then this step is skipped, and the confirmation step up next reminds you that you're about to create the stack).

It then asks for confirmation to proceed, and if you say yes, then the change is pushed using the CloudFormation “update stack” (or “create stack”) API, and then stack-updater polls the stack status, waiting for completion.

Finally there's another BBC-specific step, wherein the stack can be registered in Cosmos, our deployment manager.

Dependencies

stack-fetcher is written in ruby, and uses the aws-sdk gem.

Benefits

By using this tool, we have realised several benefits:

• speed: Using this tool is much quicker than using the other (several) tools that we used before. There are fewer commands to type, with fewer options to remember. And probably most importantly, you never have to leave your terminal.
• consistency: By automating more of the process, and by normalising the output, we now achieve more consistency: by which I mean between developers, between environments, and between components.
• understanding: This tool makes it very obvious what changes you're about to apply to live (or whatever environment you're updating) — no more blind pasting of a load of json and hoping for the best — which means fewer mistakes.

All of which means: this tool has helped us to be more productive.

Next steps

We need to separate out the BBC-specific parts from the rest, so that we can offer this tool out to a wider audience.

I'd like to make the “generation” phase more uniform: run a series of executables (bash, ruby, whatever — the tool should not care), where the first executable receives null input, and each subsequent tool filters the output of the previous one. So for example you might have filters which do: make the basic template; customise it for this environment; fill in parameter defaults.

I don't have any news yet of when this might happen, but I certainly want it to happen. Please drop me a line via a comment or on twitter if you have thoughts on this — I'd love to hear your feedback.

Yesterday I went to the Amazon Web Services Enterprise Summit, in London. I listened to this man first, then this man; then this guy, this one, and then after lunch these six men all spoke too. And throughout all of it, another man acted as compère.

Eleven. White. Males. (I couldn't find a picture of the other two, but trust me: the other two were white and male too). Not exactly a shining example of diversity, is it? And I fear that the speaker line-up at AWS re:Invent won't exactly be that much better.

This is is no way a criticism of the eleven men in question: it's a criticism of the industry in general, and Amazon Web Services in particular.

Diversity helps us all. It encourages everyone to participate. For example, having women on stage will help other women to feel like they're represented, welcome, and included. And with that inclusion comes a greater array of ideas and perspectives. Surely you want the best people, whatever their gender, race, sexuality? Lack of diversity means that you're risking excluding some of the best people who would otherwise feel included.

So, AWS: just how hard did you try to bring in non-white-male speakers? I find it hard to believe that you tried, but were unable, to find any speakers who were female and/or non-white. Call me cynical, but what I find much easier to believe is that either you couldn't be bothered, or that the concept of ensuring diversity in your speakers just didn't even occur to you (and straight white non-disabled male is the default state for humans, amirite?).

Come on Amazon. I love your products, but on the diversity front you're setting a bad example. A company of your size, influence and reach is in a fantastic position to lead by example: just as your services help your customers (and, some would say, lead the way), please use your influence to help the industry improve in diversity too, so we can all feel included.

Thank you.

Headshot images from their various public profiles.