Much of my work over the last few years has involved helping organizations “adopt” Agile. It is, after all, a poor, unloved orphan and needs to find a good home. The key to whether the new approach sticks doesn’t seem to be affected by how many checklists, process maps or charts of roles and responsibilities we provide; what matters is whether an organization can adjust their collective and individual attitudes.

There’s a great quote from the 14th Dalai Lama that says:

“If you don’t like what’s happening in your life, change your mind.”

Beyond the double meaning of “if you don’t like it, decide to like it” is the more important idea that to change your behaviors you need to change your thoughts.

So that may just seem like good, Gandhi-esque, “be the change you want to see in the world” platitudes. The real question is how can you help an organization shift its attitude from a Theory X to a Theory Y approach to management? Two things that I’ve found to help are words and actions …

Words

The language we use is a direct pathway into our minds – shift your language and your mind will follow. A change in words can facilitate a change in attitude. One clear example of this I noticed was around the use of the word “requirements”. I agree with Steve Yegge’s insight that gathering requirements is wrong-headed to start with (and I love his take on the elevator pitch), but what I’m interested with here is how the word itself is used and the connotations attached to it.

“Requirements” is a weapon word if ever I’ve heard one, particularly when used with the phrase “sign off”. Requirements are all about ordering people to do things: “I demand that you do this. Exactly this. Nothing more and nothing less. I require it!” Requirements brook no discussion and allow no room for invention. More importantly requirements stem from a combative view of relations between development teams and their customers. Signing off on requirements is all about taking a legalistic, defensive approach to covering your behind should things go awry further down the line, which they almost certainly will if you take a combative and legalistic approach to interacting with a development team.

If, however, you shift an organization’s language to talk, not about signing off on requirements, but rather about agreeing on goals, then you begin to shift from a combative to a collaborative mode of engaging. The change in mindset is two-fold: there’s a flip from saying “I” require this of “you” to saying “we” share these goals; and there’s also an expansion from the narrowing, closed, nature of requirements to the broadening and open nature of goals. A simple shift in words helps drive the evolution from defensive and hostile relationships, premised on a lack of trust, to open and collaborative interactions rooted in a sense of common purpose.

Just look at the language in the phrase “requirements capture“. What does that tell you about how requirements are handled? Once captured are they covered by the Geneva Convention or are they still considered enemy combatants?  So when it comes to planning a project, decide whether you prefer white-boarding or water-boarding sessions.

You can witness similar shifts by focusing less on “sign-off” and more on “feedback“. If you get the right feedback in place early on, then final sign-off becomes a mere formality, but if you miss out on the feedback then sign-off becomes a painful and costly hurdle.

So that’s an example of how words can help shift attitudes, what about actions?

Actions

I know of nothing more soul-destroying than sitting in a darkened meeting room while a group of people watch someone update a spreadsheet or a webapp on an overhead projector. You can feel the life and energy vacate the room as people’s attention shifts from the task at hand to suggesting keyboard-shortcuts to make the process easier or just focusing on ways to keep their eyelids open.

I’ve found that a different approach can really help foster a sense of team as you’re making a plan. It’s all about physically engaging the whole team. Here’s how it goes:

• Print or write out your stories / tasks / goals onto index cards, post-its, whatever you have at hand,
• Lay them out on a large table,
• Get the whole team around the table,
• Ask them to move the cards into the order in which they want to tackle them,
• Stand back.

It takes some teams a little while to get going – some people might be waiting for the boss to tell them the order, other people might be reticent or sceptical and someone else will be muttering about Gantt charts and resource leveling – but once someone takes the lead and starts moving cards around, a debate immediately ensues. Hands start leaping in, cards get grouped and moved and the volume rises. Then the table gets crowded, new cards get created, existing cards get rewritten, fierce debates erupt over moving a card a few millimeters to the left or right and a plan starts to take shape.

And it’s not just any plan, it’s a plan that the whole team has collaborated over, buys into and understands. It’s also, crucially, a living plan. Not an untouchable plan that is set in stone and can only be updated by the project manager with multiple sign-off from all stakeholder committees. This is important because we all know that no plan survives first contact with reality.

There are various subtleties to this style of group interaction for a planning session. The subtle movement and grouping of tasks that you can achieve is much more nuanced and multi-layered than in a gantt chart or spreadsheet. Also since no one is “driving” and controlling the mouse the plan that emerges is truly collaborative; the whole team has had an unmediated impact on the outcome. To me, though, the real key is what we’ve learned from the Wii and the group-hug: we are physical, social, creatures and we are most engaged and effective when we embrace that physicality and stand up, roll up our sleeves and get stuck in. If we are just using our eyes and our mouse finger we’re literally not firing on all cylinders.

This is the very reason that approaches like Agile and XP favor stand-ups and huddles, white-board sessions and team walks – start acting like a team and you might actually become a team.

“But what about distributed teams?” I hear you cry. Good question, we’re all out-sourced and off-shored now, so how can we leverage this full-contact planning style?

Have you ever noticed how much teamwork sucks across distributed teams? I think that’s largely because it’s nearly impossible to engender the same sense of team as you can when people sit next to each other, eat together and go out together after work. This will need to be the subject of another post, but the bottom line is that, even though telepresence technologies and collaboration tools can help, your teams will be much more effective if they take into account geographic realities. Enable each group in each location to own their own plan, where possible allow them to form their own teams.

In fact the key is to set goals rather than requirements, focus on early feedback over final sign-off and above all persuade the team to STAND UP!

The big fat lie

I understand how queuing theory can help identify and reduce bottlenecks in processes and have used finger-charts and kanban-boards to do this for a while, but I still find calling this a “pull system” to be slightly disingenuous. All that’s happening is that more “stuff” is being pushed based on a trigger when certain buckets get too low. This reminds me of my annoyance with early technologies on the web that were touted as being “push” but were really just “repetitive-pull” (but not in a good way). I’ve never seen a software organization where the developers have said to the business or product people “we’ve got nothing to do, can you think up some new projects or features for us please?”.

My objection was always that even though looking at a software delivery process as a pull system was vaguely useful for debugging some process bottlenecks, it was a fundamentally incorrect way of characterizing what was really going on. It’s just not true that features or stories are pulled down through the analyze-develop-test-build-deploy-not-necessarily-in-that-order pipeline by the customer or even by a release manager; they are pushed down that pipe by a product or project manager and the pushing often requires a healthy dose of lubricant.

Pushing string

My moment of realization came in a discussion about process over-runs in the definition or analysis phase of a project. The team recognized that requirements weren’t being communicated clearly to the developers and were planning to spend longer on the analysis and definition phase to make the requirement documents longer and clearer. Their diagnosis was correct but their prescription for how to fix the problem would just exacerbate the pain. If people aren’t finding a document useful the answer isn’t to make the document longer!

This situation reminded me of the idea of pushing string. Attempting to push a single-page document is hard (it bends and wobbles), so adding more pages should increase the weight, or “thud-factor“, of the document making it easier to push … surely there’s nothing wrong with this logic!

Empowered to pull

The solution I suggested is to invert the process, empower the developers, and get them to pull the information they need. This fits with the general concept of “people-centered processes” that attracts me to lean and agile approaches (remember a bad process will beat a good person every time). If you allow your development teams to take responsibility for the product and features they’re building they will then pull the information they need at the times they need.

So it may still be a push system for features, but the flow of information is now driven by a pull system. This flip in direction allows all the benefits of just-in-time production and saves the problems associated with maintaining a large inventory. If you write all the details of a feature too far in advance it is certain highly likely that much of that work will be wasted since the details will change or the context will shift. Documents have a shelf-life just like any other organic product and that shelf-life is much shorter than you’d like to think.

When we came onboard manual intervention was needed for any of their builds or deployments to work and so it was rare for more than a couple of builds or deployments to be completed successfully in a day. Now we often have up to 1,000 builds running every day – what’s more the majority of them now pass!

This article looks at a few of the techniques we’ve had to put in place to enable this transformation and what we’ve learnt along the way.

Dividing up builds – separation of responsibilities

Initially we had two versions of the build – one that would run on commit and a second that would only run at night. The two versions did pretty much the same things, except the nightly builds would deploy to a shared testing environment. Deployments only happened at night to avoid disruption to the multiple teams working in this shared environment (another smell to be dealt with later). The on-commit build did compilation, unit testing and packaging – the nightly build did the same and then handled environment preparation and deployment.
There were a few problems with this:

• The on-commit builds ran slowly because they were doing a lot of work
• The nightly builds were very brittle since responsibility for different parts of the build (compilation, packaging, deployment etc) was split among multiple teams – a failure at any step would mean QA didn’t have a new build to test that day
• It wasn’t DRY

Our solution was to divide the monolithic build up into several smaller steps and tie the steps together into a “build pipeline”, with each step triggered by a successfully completed “upstream” build. Our pipeline was divided into quick, assemble, package, deployment and regression builds. “Downstream” builds would pick up the artifacts prepared at the previous step in the build.

This lead to various benefits:

• Each build ran more quickly
• We didn’t repeat steps
• It was easier to divide up responsibilities for keeping the different builds green:
  • Developers :: quick build (compilation + unit tests)
  • Build team :: assemble build (jar, war, ear creation etc)
  • Deployment team :: package + deployment (preparing RPMs / configuration / deploying)
  • QA team :: regression suites
• Because we used the “last known good” upstream artifact, deployments weren’t blocked by a last minute problem upstream in the pipeline.
• We were now in a position to split the builds up over more boxes to achieve greater throughput through parallelism

This concept of dividing the responsibilities (at least for first and second line build support) had a major impact on improving success rates and response times to failures through an increased sense of ownership of the process.

Tying the builds together – HTTP publishing

In an ideal world the input for a downstream build should be the output from an upstream build – the appearance of a new artifact from one build would trigger the next build down the chain. In our situation that wasn’t quite possible (the server topology and optimal dividing lines between builds, from a speed point of view, didn’t favor communicating via artifacts) so we took a different approach.

Our solution was for builds to communicate with each other via small properties files, published locally and accessed over HTTP. On successful completion each build publishes a properties file (to the local Cruise web-server) containing the build number and SVN revision of the build. We built a custom Cruise publisher to handle this. We also built a custom Cruise HTTP modification-set to watch for upstream builds and an HTTP label-incrementer to keep all the build numbers in the pipeline in sync. The same build number is used through each of the steps of the build, making it easier to check whether changes have flowed through to the QA environment – more on this later.

Watching the river flow – visualizing the build pipeline

Once we had divided up the builds and added further applications, we ended up with close to 100 builds that needed to be managed and monitored. Since they were spread across multiple boxes to increase throughput this wasn’t as easy as just watching one large cruise page. So we built a centralized page for aggregating together the build results.

The dashboard page was comprised of a few separate components:

• Current build status grid
• Graph of passing v failing builds over time
• Graph of build times per type / step in the pipeline
• Quick list of which builds are currently broken
• Some basic code metrics

We also provided similar drill-down pages, filtered just to show data on a single application or step in the pipeline. These then link through to the actual Cruise pages for log files, failure reasons and test results.

Current build status grid

The current build status grid was probably the greatest breakthrough. The basic idea was to show the current status of all our builds in a simply structured manner. Over a few iterations we evolved a grid structure (a bit like battleships!). The columns in the grid represented steps in the build pipeline and the rows the different applications we were building. The columns were repeated for each active branch. Cells were then green if the build was good, red if it was bad and yellow if the last build was good, but was more than 24 hours old (stale). Mouseovers give more details about the selected build, such as when it ran and what its build number was.

From this view informative patterns were clearly distinguishable with a cursory glance. It became obvious if problems were localized or spread across a specific application or step in the pipeline.

Initially we emailed a snapshot of the grid first thing in the morning and at the end of the day to management and the teams. This had the wonderful effect of halting all the annoying questions and sparking the interesting ones. Instead of people asking whether a build was successful or what build number was last deployed to QA we got questions like:

• Why are all the assembly builds broken?
• Why is all of application X broken?
• Why hasn’t application Y built for 2 days?
• Why is there a build on the prod support branch?
• Wow – everything’s green – shall we go to the pub?

This definitely shone the spotlight in a few uncomfortable places, but the visible improvements this lead to made it worthwhile. It became an invaluable tool for educating people about how the pipeline works. Once people understood and were reliant on the data we replaced the emails with a dynamic, self-service, application hosting the data.

Graphing build metrics over time

We also had great success with some of our time-based visualizations. Simply graphing the times of each step in the build pipeline over the last month quickly highlighted problems as they developed. In one case we realized that all the packaging builds were getting geometrically slower and slower – a quick bit of research showed that each build was adding new artifacts to SVN and the ensuing checkouts were suffering. We fixed this problematic practice and had a massive positive impact on the total throughput of the pipeline.

The historic view of successful vs. failing builds over time was also a particularly useful tool while we were focusing on improving the stability of our deployments to QA / UAT. This gave us a simple report on how we were doing – 50% success last week, 75% this week – woo hoo!

Sparklines

We found a simple yet effective way of communicating a large amount of information about how a build has been performing by using “sparklines”.

For a given build these graphs communicate the number, duration and success of a large quantity of builds. You can quickly get an idea of how things are trending. Again mouseovers provide more details, context and drill down.

Under the hood

Initially the grid was generated and emailed by a scheduled round-robin script hitting all the build servers. But this was slow, required us to configure all the build server addresses in one place, wasn’t fault tolerant and stored no build history. So we moved the build telemetry dashboard to a simple Rails webapp employing CSS wizardry for the visualizations. The data was populated via a basic RESTful API. Each build has a custom HTTP publisher (again we built this) that fired off a simple set of information about the build in a POST request:

• Project name
• Build label
• Date / time
• Build duration
• Build successful
• Hostname of the Cruise server

Build naming conventions

The one extra key to making this all work was standardizing the format of the Cruise project name so we could parse out the application, branch and pipeline step without adding any extra complicated meta-data. We chose “<branch>-<application>-<step>” as our format, e.g. “trunk-bigapp-quick” or “3.20-littleapp-assemble”.

Where’s my build?

Using our custom label incrementer meant that each build in the pipeline for a given SVN revision would have the same build number (e.g. “trunk-bigapp-quick.123”, “trunk-bigapp-assemble.123”, “trunk-bigapp-deploy.123”). This made it much easier for developers and QA to work out what builds and features had made it through to the QA environments. We built on this by providing a customized version of the currents status grid to answer exactly that question – where’s my build?

The grid displays the latest build number at each step – clearly showing how far changes have progressed and also highlighting how a broken step breaks the pipe. This has become a vital self-service tool for the dev, QA and deployment teams, and has saved the build team from a lot of very dull questions.

A growing awareness of the importance of the pipeline has allowed us to devote efforts towards optimizing the total throughput. All the good lean principles definitely apply at this point – looking at optimizing the whole system and working out where we’re queuing or waiting too much nearly always deliver the best results.

And the rest …

These tools freed us up to use the build team as a beach-head for spreading general coding, development and automation best practices throughout the organization. Some of these included:

• Untangling spaghetti dependencies with Ivy
• Spawning multiple virtual machines for regression testing
• Automating Cruise configuration and version updates
• A whole host of black-belt Ant Fu
• Correlating multiple code metrics to drive code quality improvements

But those are stories for another occasion …

[This article was originally written for an internal ThoughtWorks innovation newsletter - I've made some minor edits for this blog]

At the client I’m currently working for the complexity of the build scripts was getting out of hand. I wanted to goad the customer into prioritising some simplification work. So I turned to GraphViz to depict how complex the build was. The build we’re using is a large, centralised, Ant script that builds about 10 different applications. It manages everything through the process of compile, test, package and deploy.

I found the handy ant2dot.xsl tool that uses XSL to transform an Ant build file into a DOT format graph representing the flow and dependencies between the various build targets.

Before

The picture that emerged was pretty shocking:

Even printed out on the largest paper the client’s printer could handle you still couldn’t even read the target names. Images of the Flying Spaghetti Monster kept jumping into my mind. But the tactic worked and when I showed the client the picture in our Iteration Planning Meeting they were quick to prioritise some work to simplify the build process.

After

After a month or so of gradual re-structuring and simplification I repeated the visualisation exercise to see how we were doing. The results were pretty striking:

Needless to say the pictures told more than a thousand words could: the client was happy, we had a neater picture to help developers understand how the build scripts worked AND the builds were simpler.