Their current flagship project, the hippo roller, though not much more than a tough plastic barrel and pulling handle, is beginning to have an immense impact on the role of women in developing societies.

In many small villages a large proportion of a woman’s day is spent fetching water. This design allows a single person to transport a much larger amount of water much more quickly with much less effort (I might be able to push a car, but I could never carry one on my head). The beauty of this invention isn’t just in the immediate benefit of more water, but in the benefits that flow downstream (all puns intended) of freeing women and children up from this back-breaking work and giving them time to focus on education and self-improvement. This then has a massive cumulative beneficial impact on the rest of the village. All this just from applying a very old solution (the wheel) to a very old problem. Social change through re-inventing the wheel.

In a similar vein Engineers Without Borders have been applying advanced design to alleviate a growing problem in Dafur refugee camps. The problem is that collecting firewood is becoming an increasingly dangerous activity. The area immediately surrounding the camps has been denuded of tinder so the women have to go further and further to gather the wood required to feed their families. This has the double-whammy impact of both increasing malnutrition as less meals are cooked and increasing the chances of being mugged or raped by the marauding gangs in the outlying areas.

The solution the clever kids from Berkeley designed to meet this problem is beautifully simple - make a more efficient stove. The more efficient the stove, the less fuel you need to cook the same amount of food. The less fuel you need, the less time needs to be spent scavenging. The less wood that is removed from around the camps, the less the resources get depleted. The less the resources get depleted, the more people the area can sustain. That’s what I call a virtuous circle.

Another not-for-profit I’ve been lucky enough to be involved with, Inveneo, have a more high-tech solution, but a similar understanding of how to trigger these positive feedback loops by introducing efficiency gains early on in the process. Even though you may pay more for the hardware, if you draw down less power the downstream benefits are magnified to dwarf the initial outlay. By improving the power consumption of their hardware and software Inveneo manage to make computing infrastructure affordable enough for poor societies to sustain the ongoing costs themselves.

What ties these all together is maximizing the efficiency of resource utilization - getting more results from less input. But what makes it even better is that the beneficial results seem to be exponential. The savings fan out as the compounding effects of the initial efficiencies multiply.

The eureka moment for me was realizing that changing the world needn’t involve a lot of hard work - it’s all about leveraging small efficiencies. Archimedes knew this when he said “give me a fulcrum on which to rest and I will move the world.” By increasing efficiency at an early stage in a process the benefits compound along the way, giving us the lever required to shift the direction of our resource depleting way of life.

The robot maker’s argument was that robots don’t have to be super intelligent to carry out some pretty useful, robotic, tasks. He was suggesting that there’s a blind-alley in robotic research premised on being able to map and track everything in your environment in order to move and interact effectively. His argument was that instead you just needed to be able to identify a goal to move towards and be able to avoid collisions. That is a much simpler goal than understanding your whole environment.

This leads to a different attitude to creating robots: have multiple robots, each good at doing one thing and then find a way to co-ordinate them. This is very much the Unix way as opposed to the Windows way.

From a feasibility and an affordability point of view this simplistic approach is much more achievable than the all singing all dancing uber computer often depicted in futuristic films. But his favored (and more philosophical) argument for this simplistic approach was that it would verge on the unethical to build a quasi-sentient being and then just ask it to fold your laundry every day. This is the Marvin-esque argument that the more human-like you make a robot the more you have to start affording it human-like rights.

Despite being interesting this line of argument instinctively felt wrong to most of us. Someone suggested the comparison with dogs, but it soon became clear that we feel animals have some level of rights too, whereas you don’t expect your computer or phone to complain when you keep asking it to check your email every few minutes. While there is clearly a quantitative difference between the rights of a human and a dog, there seems to be a qualitative difference between those of an animal and the rights we’d afford to a robot (however intelligent).

So I interjected and asked the simple question:

Does my hand have rights?

I ask my hand and my feet to do things all day (just like my computer) and I don’t expect them to complain or tell me they’re busy. This seemed to clarify the distinction for me that I’d been struggling to find.

The difference in my mind is that humans and other animals have their own goals that they are responsible for selecting (within some nature and nurture constrained boundaries). Robots (and hands and feet) on the other hand (API) are mere tools and don’t have separate goals or aspirations. This lack of goals and aspirations is the key differentiating factor that makes me feel comfortable not giving a robot freedom of speech / assembly / religion etc. I retain the absolute right to reformat my computer’s hard drive or swap it’s RAM without feeling any pangs of guilt.

A user’s interaction with a piece of software or web site is as much emotional as it is functional. Compare the soft, warm, fuzzy feeling you get when first interacting with a product from 37 Signals, say, to the stomach churning reaction you get when booting up Lotus Notes, for example.

This immediate emotional response will pervade the whole of a user’s long-term impression of a product, imbuing their relationship with whatever feeling was conjured up in those preliminary interactions. They say that in most job interviews the interviewer makes up their mind within the first 5 minutes. The same is equally true for software.

That’s the reason that I’m often accused of prematurely optimizing the UI. I’ve been round the block enough times to have been bitten by the following type of interaction:

Me: “This is just a wire-frame, we’re only trying to prove out the functional flows of the application. Don’t worry about the layout or the look and feel - that will be dealt with later.”

Client: “Ooh, I don’t like that font … and I think that’s our old logo. Maybe we should give it a drop-shadow or something. I’m disappointed, I expected more from you guys …”

After a host of similar experiences I now make sure that I give whatever product I’m working on an appropriate level of design and polish for the current stage, while still trying to evoke positive emotional responses. In my mind it’s always worth spending an hour or two tweaking a CSS file to tighten the layout, soften the lines and add a little sparkle - a little :hover goes a long way. Then even if the application isn’t functionally complete at least it looks professional and imbues the client with a sense of confidence and pleasure. This will put them into a much more positive state of mind, which can only be a good thing for the next stages of the project.

Call me shallow if you want, but when I arrive at a (studiously) un-styled site, like our guru Martin Fowler’s, I have a negative knee-jerk reaction making me resist reading the wise words he’s actually written. When, however, I land on a site where they care about the first bite I feel compelled to carry on reading all day.

If someone in a meeting has a bit of chopped herb stuck in their teeth it’s really hard to focus on what they’re saying - you get distracted and have a negative emotional response. In the same way drizzling some coulis and sprinkling a bit of chopped herb onto a plate before serving can provoke a positive emotional response which fools the diner into enjoying their food more.

When we came onboard manual intervention was needed for any of their builds or deployments to work and so it was rare for more than a couple of builds or deployments to be completed successfully in a day. Now we often have up to 1,000 builds running every day - what’s more the majority of them now pass!

This article looks at a few of the techniques we’ve had to put in place to enable this transformation and what we’ve learnt along the way.

Dividing up builds - separation of responsibilities

Initially we had two versions of the build - one that would run on commit and a second that would only run at night. The two versions did pretty much the same things, except the nightly builds would deploy to a shared testing environment. Deployments only happened at night to avoid disruption to the multiple teams working in this shared environment (another smell to be dealt with later). The on-commit build did compilation, unit testing and packaging - the nightly build did the same and then handled environment preparation and deployment.
There were a few problems with this:

• The on-commit builds ran slowly because they were doing a lot of work
• The nightly builds were very brittle since responsibility for different parts of the build (compilation, packaging, deployment etc) was split among multiple teams - a failure at any step would mean QA didn’t have a new build to test that day
• It wasn’t DRY

Our solution was to divide the monolithic build up into several smaller steps and tie the steps together into a “build pipeline”, with each step triggered by a successfully completed “upstream” build. Our pipeline was divided into quick, assemble, package, deployment and regression builds. “Downstream” builds would pick up the artifacts prepared at the previous step in the build.

This lead to various benefits:

• Each build ran more quickly
• We didn’t repeat steps
• It was easier to divide up responsibilities for keeping the different builds green:
  • Developers :: quick build (compilation + unit tests)
  • Build team :: assemble build (jar, war, ear creation etc)
  • Deployment team :: package + deployment (preparing RPMs / configuration / deploying)
  • QA team :: regression suites
• Because we used the “last known good” upstream artifact, deployments weren’t blocked by a last minute problem upstream in the pipeline.
• We were now in a position to split the builds up over more boxes to achieve greater throughput through parallelism

This concept of dividing the responsibilities (at least for first and second line build support) had a major impact on improving success rates and response times to failures through an increased sense of ownership of the process.

Tying the builds together - HTTP publishing

In an ideal world the input for a downstream build should be the output from an upstream build - the appearance of a new artifact from one build would trigger the next build down the chain. In our situation that wasn’t quite possible (the server topology and optimal dividing lines between builds, from a speed point of view, didn’t favor communicating via artifacts) so we took a different approach.

Our solution was for builds to communicate with each other via small properties files, published locally and accessed over HTTP. On successful completion each build publishes a properties file (to the local Cruise web-server) containing the build number and SVN revision of the build. We built a custom Cruise publisher to handle this. We also built a custom Cruise HTTP modification-set to watch for upstream builds and an HTTP label-incrementer to keep all the build numbers in the pipeline in sync. The same build number is used through each of the steps of the build, making it easier to check whether changes have flowed through to the QA environment – more on this later.

Watching the river flow - visualizing the build pipeline

Once we had divided up the builds and added further applications, we ended up with close to 100 builds that needed to be managed and monitored. Since they were spread across multiple boxes to increase throughput this wasn’t as easy as just watching one large cruise page. So we built a centralized page for aggregating together the build results.

The dashboard page was comprised of a few separate components:

• Current build status grid
• Graph of passing v failing builds over time
• Graph of build times per type / step in the pipeline
• Quick list of which builds are currently broken
• Some basic code metrics

We also provided similar drill-down pages, filtered just to show data on a single application or step in the pipeline. These then link through to the actual Cruise pages for log files, failure reasons and test results.

Current build status grid

The current build status grid was probably the greatest breakthrough. The basic idea was to show the current status of all our builds in a simply structured manner. Over a few iterations we evolved a grid structure (a bit like battleships!). The columns in the grid represented steps in the build pipeline and the rows the different applications we were building. The columns were repeated for each active branch. Cells were then green if the build was good, red if it was bad and yellow if the last build was good, but was more than 24 hours old (stale). Mouseovers give more details about the selected build, such as when it ran and what its build number was.

From this view informative patterns were clearly distinguishable with a cursory glance. It became obvious if problems were localized or spread across a specific application or step in the pipeline.

Initially we emailed a snapshot of the grid first thing in the morning and at the end of the day to management and the teams. This had the wonderful effect of halting all the annoying questions and sparking the interesting ones. Instead of people asking whether a build was successful or what build number was last deployed to QA we got questions like:

• Why are all the assembly builds broken?
• Why is all of application X broken?
• Why hasn’t application Y built for 2 days?
• Why is there a build on the prod support branch?
• Wow – everything’s green – shall we go to the pub?

This definitely shone the spotlight in a few uncomfortable places, but the visible improvements this lead to made it worthwhile. It became an invaluable tool for educating people about how the pipeline works. Once people understood and were reliant on the data we replaced the emails with a dynamic, self-service, application hosting the data.

Graphing build metrics over time

We also had great success with some of our time-based visualizations. Simply graphing the times of each step in the build pipeline over the last month quickly highlighted problems as they developed. In one case we realized that all the packaging builds were getting geometrically slower and slower – a quick bit of research showed that each build was adding new artifacts to SVN and the ensuing checkouts were suffering. We fixed this problematic practice and had a massive positive impact on the total throughput of the pipeline.

The historic view of successful vs. failing builds over time was also a particularly useful tool while we were focusing on improving the stability of our deployments to QA / UAT. This gave us a simple report on how we were doing – 50% success last week, 75% this week – woo hoo!

Sparklines

We found a simple yet effective way of communicating a large amount of information about how a build has been performing by using “sparklines”.

For a given build these graphs communicate the number, duration and success of a large quantity of builds. You can quickly get an idea of how things are trending. Again mouseovers provide more details, context and drill down.

Under the hood

Initially the grid was generated and emailed by a scheduled round-robin script hitting all the build servers. But this was slow, required us to configure all the build server addresses in one place, wasn’t fault tolerant and stored no build history. So we moved the build telemetry dashboard to a simple Rails webapp employing CSS wizardry for the visualizations. The data was populated via a basic RESTful API. Each build has a custom HTTP publisher (again we built this) that fired off a simple set of information about the build in a POST request:

• Project name
• Build label
• Date / time
• Build duration
• Build successful
• Hostname of the Cruise server

Build naming conventions

The one extra key to making this all work was standardizing the format of the Cruise project name so we could parse out the application, branch and pipeline step without adding any extra complicated meta-data. We chose “<branch>-<application>-<step>” as our format, e.g. “trunk-bigapp-quick” or “3.20-littleapp-assemble”.

Where’s my build?

Using our custom label incrementer meant that each build in the pipeline for a given SVN revision would have the same build number (e.g. “trunk-bigapp-quick.123”, “trunk-bigapp-assemble.123”, “trunk-bigapp-deploy.123”). This made it much easier for developers and QA to work out what builds and features had made it through to the QA environments. We built on this by providing a customized version of the currents status grid to answer exactly that question – where’s my build?

The grid displays the latest build number at each step – clearly showing how far changes have progressed and also highlighting how a broken step breaks the pipe. This has become a vital self-service tool for the dev, QA and deployment teams, and has saved the build team from a lot of very dull questions.

A growing awareness of the importance of the pipeline has allowed us to devote efforts towards optimizing the total throughput. All the good lean principles definitely apply at this point – looking at optimizing the whole system and working out where we’re queuing or waiting too much nearly always deliver the best results.

And the rest …

These tools freed us up to use the build team as a beach-head for spreading general coding, development and automation best practices throughout the organization. Some of these included:

• Untangling spaghetti dependencies with Ivy
• Spawning multiple virtual machines for regression testing
• Automating Cruise configuration and version updates
• A whole host of black-belt Ant Fu
• Correlating multiple code metrics to drive code quality improvements

But those are stories for another occasion …

[This article was originally written for an internal ThoughtWorks innovation newsletter - I've made some minor edits for this blog]

The mailing lists at ThoughtWorks have been running hot since Jay Fields stoked the flames of the dynamic versus static language schism. His basic point is summarized in the opening line:

“Given a good test suite the return on investment simply does not justify the use of static typing”

Basically he’s saying that statically typed languages have benefits, but there is an associated cost - anyone who’s seen object definitions disappear off the screen with Java 5 generics will understand this. The benefits definitely exist - modern IDEs (auto-complete, refactoring, etc), compiler optimizations, basic verification and clarity in communicating integration requirements - but there is also a cost incurred in terms of complexity and inflexibility. Neal Ford has started calling this the “static type tax”. Jay’s solution is, rather than relying on static typing for partial protection, rely on tests for full protection.

Work to rule

This pain or tax associated with static types reminds me of the parody of union rules gone bad - “I can’t plug that in, I’m a carpenter, you need an electrician”. These rules start out having important uses (protecting workers against abuse and ensuring quality results), but can end up just causing pain and making the jobs they’re trying to protect less secure by making the industry less responsive. There is an obvious correlation with the trade-offs between dynamic and statically typed languages: it’s nice to know from an integration and tooling perspective exactly what type of object a method requires; it’s also nice just to be able to give that same method any object that can respond to the messages it will be sent.

In strong and statically typed languages, like Java, the method signature is a shorthand explicitly stating what type of object is required for the method to work. In a dynamic or weakly typed language, like Ruby or JavaScript, you have more flexibility about what you can pass into the method, but less visibility into whether it will work or not - you have to check (visually or through tests) that the object you pass in can respond sensibly to the instructions it’s given.

One of the pains of the statically typed method signature is that various different requirements tend to get bundled up together. A method may well require a parameter to be of type (implement interface) X, where interface X defines 10 methods, even though the method will only call one of those declared methods. Ideally you would define a new interface for each combination of methods you want to call so you can keep the contract as simple to fulfill as possible, but this in itself would be a burden as the amount of interfaces would explode.

Risk assessments

The arrival of complexity and restriction in the name of preventing accidents (passing an object that can’t react to the messages sent to it) reminds me of how school trips have become harder and harder to organize because of the ever more stringent “risk assessments” required:

“Endless lengthy risk assessments and fears of being blamed if anything goes wrong have put teachers off taking children away on trips and led to youngsters being cooped in classrooms instead.”

I remember loving some fairly wild school or youth club trips as a kid. Rock scrambling was one of my favorites, which involved climbing, jumping, swimming, hopping and whooping round sections of coastline like the beautiful Lulworth Cove. Sure there were risks involved, but learning to handle yourself in those risky situations lead to really important lessons and personal growth. Risk breeds responsibility.

I have similar feelings about using dynamic languages - with great power comes great responsibility. Programming itself is inherently risky (when was your last zero-defect release?) so programmers have to make decisions about how to act responsibly. One way is to impose risk assessment forms on every method call through static typing; another option is to force objects to act responsibly and be good citizens. This is the route that interests me more. Testing definitely plays an important role in acting responsibly, but it’s not the end of the story.

Team collaboration : object collaboration

Coming back to the analogy of objects as workers on a job, rather than using strict role definitions in an attempt to control quality, I would much prefer to use the example of self-organizing Agile teams. On an Agile development team there is an expectation that people can work in a cross-functional manner with common ownership of the code-base. This means that team members should be interchangeable and comfortable in a variety of areas - just like in dynamic languages there is no type checking before picking up tasks (”sorry I only do JavaScript / Java / SQL”). This leads to highly flexible and productive teams, but there are a few techniques needed to make it work. Some of these techniques are conceptually transferable to working responsibly with dynamic languages:

• communication
• simplicity
• responsibility

Explicit and clear communication is the biggest of these - the others can be derived from this principle. The importance of communication in teams is fairly obvious, but it can also be important at the code level in areas like explicit tests, clearly named methods and transparent intentionality.

Simplicity aids communication and also lowers risk. The simpler the routine is, the easier it is to understand, the easier it is to see what is required of collaborating objects and the less likely it is that bugs will creep in.

Responsibility underpins this all. In teams it means talk to people if you’re not sure, or tell people if you’re going to do something strange; in objects it means testing your interactions, handling errors appropriately and checking compatibility where necessary.

Logicalis have put together some advice on how to reduce the impact of IT on the environment. The suggestions range from reducing hardware requirements through virtualization and other consolidation techniques to old favorites like double-sided printing, video-conferencing, electronic forms and turning off your desktop at night.

One idea I hadn’t really thought of was the impact of data archiving strategies (or Information Life-Cycle Management as they grandly title it) on energy usage. Data needs hard drives need power. Choosing what to compress, archive and delete can have a major impact.

If you don’t think that powering down your PC at night can make a difference Global Action Plan’s recent report might open your eyes as it shows that the environmental impact of IT is almost identical to that of the much-maligned aviation industry.

Ultimately, as with so many environmental issues, the best solution is to make sure that the people who make the decisions are the same as (or at least aligned with) the people who are impacted. I’m not saying sys admins should go and live in the semi-submerged Bangladeshi delta. My advice is much simpler - make sure IT departments are responsible for their electricity bills. The most striking statistic in the “An Inefficient Truth” report (PDF 1.3MB) was that not only do 68% of IT departments not pay for the energy their systems consume, 56% of them don’t even see the bills.

Making environmentally sound decisions isn’t about warm fuzzy feelings - it’s about hard-nosed, joined-up, economic thinking.

Managing Ant target dependencies

“depends” are great, until your build file gets bigger than a couple of screenfuls. You can end up with a crazy spaghetti monster of dependencies very quickly. On a few builds I’ve worked on we’ve had a basic rule:

Targets can either have depends or a body, but not both.

So you end up having two types of targets:

• “functional targets” that actually do the work, carrying out small individual tasks
• “flow targets” that tie the functional targets together

<target name=”the.lot” depends=”compile,test,jar,report” />
<target name=”compile”>
…
</target>
<target name=”test”>
…
</target>

Public vs Private targets

I’ve built build files using the public and private target idea (affix a “-” to private targets so they can’t be called externally).

It can work in some situations and makes it reasonably clear what’s going on. It can also work in conjunction with the “functional and flow” targets described above. (Functional targets are private; flow targets are public).

My main issue with this approach is that as a build engineer you often want to call internal / private targets to run just a small step in the build (particularly while developing new features) - you can’t do this if the targets are private. so you end up creating a second, public, target that calls the private target … and you end up doubling the size of your build files.

I prefer just to have the main entry point targets (your flow targets) at the top of the build file so it’s obvious to anyone who opens the build file what they can use.

You can also differentiate your public (API) targets by only allowing them to have description attributes. These will be the only targets listed when your run “ant -p”.

Macrodefs with closures

Macrodefs are great and should be used to reduce duplication wherever possible.

An extra-powerful feature of macrodefs, which I only came to understand recently, is the ability to pass arbitrary ant elements into macrodefs - think closures / ruby blocks.

We now have expressive parts of our build scripts looking a bit like the following:

<for-each-module>
<compile module=”@{module}” />
</for-each-module>

<for-each-module>
<test module=”@{module}” />
</for-each-module>

<macrodef name=”for-each-module”>
<element name=”action” />
<sequential>
…. do some stuff …
<action />
…. do some more stuff …
</sequential>
</macrodef>

I’d be interested to hear if anyone has other suggestions for keeping Ant from spiraling out of control.

The good people at MIT have put together scratch a visual tool allowing kids to do drag-n-drop coding. The help screens give a good idea of how it works. Basically differently shaped blocks are put together (lego style) to form programs: a loop looks like a capital C and holds all the nested statements; booleans are pointy ended and only fit in pointy slots - likewise numbers are round and only fit in round slots … a nice simple introduction to strongly-typed languages. It actually fits pretty closely with how I visualise blocks of code, so it looks like a great way to introduce children to the coding mind-set. The welcoming colourful blocks are non-threatening and simple to understand.

If that all seems too point-n-clicky then there are also efforts to get kids using Ruby. It’s a simple hosted and sandboxed service giving children the chance to write real programs - like a blog in just 6 lines.

This block caught my attention:

WHITHER ART THOU, BASIC??

In the 1980s, a language called BASIC swept the countryside. It was a language beginners could use to make their computer speak, play music. You could easily draw a big smiley face or a panda or whatever you like!

I’m interested in their choice of Ruby as the simple “gateway” language (the first one’s free, tell your mates), since there have been various discussions about whether Ruby is too complex or powerful for many inexperienced QA and dev teams to be *trusted* with.

It’ll be interesting to see how these kinds of tools are adopted. Either way I think some people at my current client could use a more pointy-clicky programming language …

I’ve seen some demos of Mingle and it looks great, but of equal interest to me is the choice to release it on JRuby. This seems to be another step along the road of Java moving down the stack. Java was selected as the delivery platform because corporate IT understands how to deploy, integrate, support and optimize it; Ruby was chosen as the development language because of the productivity and expressiveness of the language.

I don’t want to knock Java - it’s provided my bread and beer for 10 years, but there are many strong arguments for using more concise and expressive languages to encode business logic.

So it looks as though Java is moving more into the space of an OS rather than a language for building business logic. Our illustrious founder, Roy, suggested that Java’s legacy will be the JVM - years have been spent optimizing it and making it run on many platforms. I would add to that the ecosystem of libraries integrating with a panoply of databases, messaging formats and middleware.

JRuby allows Ruby to leverage this whole ecosystem. Operations teams are completely comfortable managing Java installations - they don’t tend to care too much what’s in your WAR or EAR as long as they can monitor and configure the JVM, datasources, queues and other resources.

One question that springs to mind is whether this is just a temporary stage while the Ruby deployment stack matures or whether it will be a lasting situation. It would seem odd to need all the extra infrastructure, but the current situation is that even though many apps are deployed on highly expensive Java containers (think IBM and BEA) most of them will still sit behind Apache. They do this because Apache is more trusted to manage security, redirection, static content delivery and various other web fundamentals.

So maybe the new stack will be:

• Apache handling the HTTP piece
• A Java container handling the Integration piece (with or without a captial E in front)
• Ruby (or other languages leveraging JSR 223) to handle business logic

It will be interesting to see where this goes, but I can’t help feeling there’s one too many pairs of hands in that stack.

At the client I’m currently working for the complexity of the build scripts was getting out of hand. I wanted to goad the customer into prioritising some simplification work. So I turned to GraphViz to depict how complex the build was. The build we’re using is a large, centralised, Ant script that builds about 10 different applications. It manages everything through the process of compile, test, package and deploy.

I found the handy ant2dot.xsl tool that uses XSL to transform an Ant build file into a DOT format graph representing the flow and dependencies between the various build targets.

Before

The picture that emerged was pretty shocking:

Even printed out on the largest paper the client’s printer could handle you still couldn’t even read the target names. Images of the Flying Spaghetti Monster kept jumping into my mind. But the tactic worked and when I showed the client the picture in our Iteration Planning Meeting they were quick to prioritise some work to simplify the build process.

After

After a month or so of gradual re-structuring and simplification I repeated the visualisation exercise to see how we were doing. The results were pretty striking:

Needless to say the pictures told more than a thousand words could: the client was happy, we had a neater picture to help developers understand how the build scripts worked AND the builds were simpler.