The landscape is changing

Since the dawn of the software era, systems have generally followed a lifecycle of develop/operate/replace. For the type of systems our company, ThoughtWorks, specializes in (typically built over the past 10-15 years), organizations expect as much as 5-10 years between significant investments in modernization. And some of the oldest core systems have now reached 40+ years — far longer than the average life-span of most companies today!

IT assets are relatively long-lived largely because modernization often represents a significant investment that doesn’t deliver new business value in a form that is very visible to managers or customers. Therefore organizations put off that investment until the case for change becomes overwhelming. Instead, they extend and modify their increasingly creaky platforms by adding features and making updates to (more or less) meet business needs.

For decades, this tension between investing in modernization versus making incremental enhancements has played out across technology-enabled businesses. Every year some companies take the plunge and modernize a core system or two, while others opt to put yet another layer of lipstick on the pig.

We see this pattern being disrupted as the demands being placed on legacy systems undergo a fundamental shift. Previously, system changes were driven by business requests for incremental features, and IT had to deal with a major new technology platform or architecture only every five years or so. Today, the viable lifespan of any business model is shrinking, driving demand for wholesale feature changes on a nearly continuous basis. For many of these new features, the benefits of leveraging one of the ever-expanding varieties of new architectures and platforms are significant. For example, your current infrastructure was probably not designed to enable you to connect your supply chain directly to an e-commerce channel or provide customer self-service via mobile devices. Sure, you may be able to force that square peg into your round hole, but the chances of an elegant, extensible solution are slim.

The “lifecycle model” of software systems is becoming irrelevant. The companies that will excel in the future will be the ones that learn to incrementally modernize and then continuously evolve their core technology assets to thrive in an ever-more volatile business and technology environment.

The first step is admitting you have a problem

Organizations that are constrained by creaky platforms are often slow to identify this as the root cause of their trouble. Instead, as release cycles grow longer and delivery quality declines, fingers get pointed at IT or
at product/service vendors who are getting bogged down trying to work around the underlying problems caused by mounting technical debt in a toxic systems environment.

Figure 1 illustrates some key indicators of an underlying creaky platform, grouped according to whether they are felt more acutely by business or IT stakeholders and by whether they are intuitive or measurable. As you look through the factors, keep in mind that none of these should be considered “normal” or something that “comes with the territory” in IT. In fact, there are organizations out there operating in complex and volatile enterprise environments that do not experience any of these problems.

Figure 1 — Signs of a creaky platform.

We have often found that the intuitive factors manifest themselves in advance of the measurable factors. These can therefore be considered leading indicators that, should they appear, signal you to dig a bit deeper into your underlying technology infrastructure.

When evaluating a complex situation, organizations typically respond well to logical, fact-based arguments supported by quantitative data. So, as you consider whether you need to make a case to modernize, you should apply some measurements to areas of intuitive pain. There are different techniques for quantifying business and IT pain.

Quantifying business pain

Many of the business pain items in Figure 1 are straightforward to measure and don’t require special techniques. Simple counts of bugs, feature backlogs, and release frequencies help quantify the business impact
of an underlying creaky platform. These items are especially effective when presented as trends over time versus point-in-time readouts.

However, one common type of business pain — cumbersome business processes — benefits from a more structured quantitative approach. There are many ways to do this, but one of the best is a technique from lean thinking called value stream analysis.1 This approach analyzes customer value–producing processes and identifies areas of waste. Measuring waste provides a powerful quantification of the business impact of cumbersome processes.

Quantifying IT pain

Technical debt is the phrase used to summarize the pain caused by the cumulative effect of all the tactical, short-term, “band-aid” solutions we use on IT projects. Just as in life, where it often makes sense to take on debt, at times it makes sense to go into technical debt in order to reach certain delivery milestones. Organizations that take a conscious decision to increase technical debt are usually sophisticated enough to plan to pay it down over time once the short-term objective has been reached. The problem is that technical debt is often built up unconsciously and, like a runaway credit card account, reaches a point where it becomes very difficult to bring the balance down.

Organizations often unwittingly amass technical debt because the major metrics managers have access to and are measured against are scope and budget-based and rarely include intrinsic quality metrics. The challenge is finding ways to balance these velocity-based metrics with quality metrics that can highlight the hidden cost of the continual tradeoffs that are made.

More objective measures of software quality do exist and can be used to track and control technical debt. Individually, none gives the whole picture, but together they begin to tell a cogent story. These metrics fall into three broad categories:

• Static code analysis (complexity, duplication, cohesion, tangling, etc.),
• Rule violations/programming errors (which can be identified using Checkstyle, FindBugs, PMD, etc
• Test quality (coverage)

Later in this article, we highlight some techniques for using metrics to drive remediation efforts, but when attempting to quantify technical debt, we have found it helpful to compare metrics against a set of well-known applications. Benchmarking a bundle of metrics against a few open source and internal projects of varying, but known, quality provides a clear comparative view of an application’s quality.

Sounding the call to action

Having decided that a system needs to be modernized, you need to get money to do it. With many priorities competing for funding, spending on legacy systems can be a tough sell. In our experience, timing is everything, and specific situations or events can open the door to modernization:

• New leader. Often new executives look to make their mark with a major initiative, and businesses give them leeway to do so. New people are also not vested in relationships and decisions made before their time, giving them more freedom to consider alternatives.
• New rules. New regulations or standards create non-negotiable drivers to update legacy systems. Depending on the scale of change, this may present an opportunity to make a case to deliver the change on a modernized platform as opposed to modifying the existing one.
• Business crisis. Most businesses respond assertively to threats and crises. One of our customers enjoyed years of a near monopoly, despite a core creaky platform that caused them to consistently underdeliver against their product roadmap. The impetus for modernizing that platform came when a major customer
  left for a competitor because it had lost faith in the roadmap.
• Opportunities lost. Losing prospective new customers because of the application’s defects and apparent age is a powerful motivator for modernizing a creaky platform.
• New strategy. Aligning modernization with a key business strategy is wise. For example, we worked with an organization that ran an ad-driven community Web site. When business leaders decided to sell that Web site as a reskinnable platform to multiple customers, the development team made a successful argument to invest in modernizing the site.
• Technology breakthrough. Sometimes a new development in technology changes the economics of remediation or creates a new source of return on that investment. Thinking through the applications of new technology to your remediation efforts is worth the time.

IT-business collaboration is critical

Too many organizations make a fundamental error by approaching modernization as an IT-only problem. In so doing, they miss an opportunity to create new business value. They also tend to prioritize the work from a technical perspective, which often results in a quite different approach and solution than one created collaboratively with business stakeholders.

The perils of the IT-only approach were brought home to us during a consulting engagement in which the IT department of an investment bank asked us to validate their modernization roadmap. The roadmap was a plan to replace 29 systems over almost five years, resulting in a cutting-edge IT infrastructure. One of the first things we did was to ask key business stakeholders if the roadmap was aligned with their priorities. We were shocked to discover they weren’t even aware of the initiative. They were very concerned that by proceeding without business input, IT was likely to just rebuild all the redundant and inappropriate systems the business was struggling with, warts and all.

This is an extreme case, but it happens more frequently than you might expect. Fortunately, this story has a happy ending. We were able to broker a conversation between IT and the business that resulted in a major rationalization of the application portfolio and delivered a leaner, better-performing system much more quickly than the initial roadmap. The most successful modernization efforts are jointly planned and executed to deliver against IT and business priorities, incrementally evolving toward a better state for all stakeholders.

Deciding how to proceed

Once you’ve got your funding, you are faced with a decision about how to proceed. The two primary dimensions to consider are refactoring versus rewriting and “big bang” versus incremental. A rewrite can recreate exact feature parity with the existing application, just implemented in a new technology, or it can include redesigning the functionality. Despite the added complexity in testing, we strongly recommend taking the opportunity to identify what functionality is still really needed by the business. The keys to success in this scenario are:

• Working in tight coordination with the business and end users to gain constant verification of fitness for purpose
• Working in very small increments that can be fully validated and vetted
• Keeping the existing application in place to retain all existing functionality in other areas

Refactoring vs. Rewriting

Our general advice is, where possible, to look first at incremental refactoring. Good development practices should always include a refactoring phase when each new feature is added to maintain a simple, elegant, and well-factored design.
We find refactoring is best performed incrementally. Executing a large-scale refactoring exercise in isolation from the main code line (i.e., on a separate branch) should be considered dangerous. The key to a successful refactoring effort is doing it hand in hand with your normal project or production support team, integrating as you go.

If an application has deteriorated to such an extent that refactoring efforts are too big or painful to countenance, then you are faced with a total rewrite. Again there is a choice between big bang and incremental approaches.

Big Bang vs. Incremental

Replacing an application in a big bang is rarely our recommended strategy. Attempting to create feature parity with the legacy application extends timelines to the point that requirements are likely to have changed significantly between design and final delivery. Without feedback from live usage, it is likely the new version won’t meet all the business needs. The risk of the final cutover is also large, since the new application has yet to be battle-tested in production and a full data migration will be required.

Our preferred approach is a phased, incremental strategy. Though this may seem counterintuitive given the extra effort required to work around the existing application, our experience has shown that this minor cost is heavily outweighed by the reduced risk of the migration, the fitness for purpose of the resulting application, and the decreased disruption posed by the overall process.

Many business justifications for replacing an application include claims that the new application will be more “extensible.” There is a false premise that extensibility comes from up-front design activities that define modules, extension points, XML configuration, and the like. Our firm belief is that the best way to end up with an extensible platform is to extend it as you go. If you build your application incrementally, there is a good chance that you will make it extensible, particularly if you put in place the practices and patterns required
to extend an application continuously, such as automated testing and simple modular design. Incremental approaches tend to increase the likelihood that your application will support ongoing extension.
The real challenge, then, is effectively performing an incremental rewrite of an application. We have some recommended methods and advice on approach and coordination.

Using metrics and visualization to drive remediation

Technical debt, just like its financial cousin, has the nasty habit of compounding. If you don’t pay it down regularly, then the ultimate recourse is declaring bankruptcy and reaching for the rewrite. The problem with the code metrics tools we mentioned earlier is that they tend to provide too much information to drive actionable remediation decisions. I remember attempting to run Checkstyle across a Fortune 500 client’s code base, and the program core-dumped before completing! In contrast, correlation and visualization are two particularly useful techniques for obtaining a holistic overview of the health of a system and also directing remediation activities.

Correlation

When one client was struggling to make an impact on their technical debt, we helped them by correlating multiple metrics to direct their remediation activities on the highest-priority problem areas. Our premise was that if an area was complex but rarely touched, then it was less dangerous than one that was under heavy development. Likewise, a complex area covered with good automated testing is less critical than a similar one with no test coverage. Following this thinking, we created an aggregated risk metric that correlates complexity, test coverage, and volatility. Volatility was defined as a function of source control commit activity on the area of code — frequent activity indicated high volatility. This definition allowed us to pinpoint a small set of high-risk areas to address first; in a haystack of millions of lines of code, it called out a few very specific places to begin refactoring and rationalizing.

Visualization

Toxicity, another aggregated metric, has been playing a prominent role in our “system health checks” at ThoughtWorks. Toxicity charts stack multiple static analysis metrics for classes, methods, or components within an application, providing a combined “toxicity” score for each area of the code base (see Figure 2). This gives our clients guidance on where to start looking to fix problems.

Figure 2 – Toxicity chart

Using visualization in this way allows us to avoid drowning in a sea of data. The human visual cortex is much more efficient at complex pattern recognition than most programs we could write, so it makes sense to leverage that capability.

The basic stacked bar chart used for toxicity is a good start, but if you want to correlate multiple variables, then tree maps are a powerful tool in that their combination of size, location, and color allows you to overlay more complex information onto a single image (see Figure 3). The nested nature of the visualization maps well onto the hierarchical nature of most code bases; color and size are then used to aggregate other metrics such as lines of code, complexity, or coverage. Again, this provides a single-shot overview of health as well as forensic information on where to look for the smoking gun.

Figure 3 – Treemap of code complexity

Tree maps show how the code is organized into packages and classes and visualizes their relative sizes. The size of the various rectangles represents the size of the class files and the encompassing packages. Coloring is then used to layer on an extra metric of interest — in this case, complexity.

Taking this technique one step further, a three-dimensional visualization called a “code city” gives a real feel for the personality of the different neighborhoods in a code base (see Figure 4). This view of an application as a city supports the analogy that you need to pair program when the code base is so dangerous that you’re afraid to go in alone. A code city visualization is similar to a tree map (see above) but uses the third dimension to overlay an extra metric to correlate. An ideal combination is correlating complexity to test coverage. So, if the visualization maps lines of code to area, complexity to height, and test coverage to color, then a neighborhood containing large, tall, red buildings would represent an area of the code base that contains large, highly complex, and untested classes. Clearly this would be an area in which you would want to proceed with extreme caution.

Figure 4 – CodeCity visualization

As helpful as they are, automated metrics are only part of the story in avoiding technical debt. Code has to communicate effectively with both computer and human audiences. Automated functional and unit tests can tell you how well the code communicates with computers by verifying the expected behavior. However, automated metrics can only hint at how well the code communicates with humans. Human involvement is ultimately needed in the evaluation of any code base. We prefer doing this in real time through pair programming, though code reviews and other techniques can provide similar benefits. Remember: metrics should be the beginning not the end of the conversation.

How to replace your legacy application

Team Considerations

Before embarking on replacing an application, it is worth taking stock of your existing organization. Conway’s Law states that the architecture of an application will come to mirror the communication patterns of the organization that created it. This can be summarized as “Dysfunctional organizations tend to create dysfunctional applications.” To paraphrase Einstein, you can’t fix a problem from within the same mindset that created it, so it is often worth investigating whether restructuring your organization or team would prevent the new application from displaying all the same structural dysfunctions as the original. In what could be termed an “inverse Conway maneuver,” you may want to begin by breaking down silos that constrain the team’s ability to collaborate effectively. Of course there are many situations where this may not be realistic, but remember that Conway’s Law talks of the “communication structures” of an organization rather than reporting structures. There are often opportunities to improve communication pathways in lightweight ways without having to grapple with thornier organizational issues.

A set of recurring themes emerges from teams that have successfully executed incremental application rewrites:

• Working on the main code line (or trunk) is vital to avoiding painful merges or missing important improvements occurring in the underlying application.
• Having the team (at least partially) populated with people who have lived with the pain of the existing application and have a deep understanding of the subtleties of the business and technology domain
  is key in shaping the new product to meet the business’s needs.
• Creating a small, colocated team is also recommended, as is having a clear and focused charter of the business need that each phase of the project is delivering.
• Practicing and executing data migrations is best done from the outset of the project while it is still a tractable problem.

Technical Approach

A favored approach for incrementally replacing a live application is the so-called “strangler application” named after the family of tropical strangler figs. These plants grow quickly around an existing tree, using its existing structure for support and shape. Over time they thicken and fuse, completely surrounding and replacing the original tree, leaving a new version standing in its place as the old one withers and dies. The strangler application uses the same approach of creating a thin wrapper around an existing application, then gradually peeling or slicing off and replacing functionality. Features are gradually migrated from the legacy to the new application until nothing of value remains in the old, enabling a graceful retirement.

Common patterns include:

• Intercepting requests at the front of an application, then redirecting certain ones to the legacy application and others to the new application
• Sharing a single integration database or regularly trawling the old database to populate the new one
• Peeling back the application vertically tier by tier (maybe replacing the database, presentation layer, or business logic first)
• Intercepting events
• Some combination of the above

Figure 5 – A basic “strangler application” sequence

Most of these patterns involve creating an extra piece of infrastructure, such as an indirection layer or a polling system, which may appear to be wasted effort. Interestingly, we regularly find that the strategies required to enable a gradual migration often prove to be valuable long-term architectural buffering devices that both provide resilience during system upgrades or outages and offer the seams needed for future enhancements. An example of this was when we provided a major ISP a mechanism to replace their application stack incrementally by separating the system that captures new client orders from the system that processes them. Later on this separation proved invaluable in enabling them to continue taking orders even while their main order processing system was down for maintenance.

Breaking the cycle of pain

The recommendations here provide suggestions for how to modernize, but that is only half the battle. To really elevate your enterprise to the next level, you need to break the modernization-degradation cycle permanently. Fortunately, the tools and approaches that help you incrementally modernize are exactly the same ones that can eliminate the need to ever do it again:

• By using sophisticated automated metrics to continuously identify degrading areas of your system, you can strike a better balance between incremental remediation and adding new features going forward.
• By practicing an incremental, evolutionary approach to architecture (called a “strangler” in the context of remediation efforts, but just “evolutionary architecture” outside that context), you can avoid the need to embark on big-bang replacements.
• By continuing to add tests at the same rate as you add new functionality, you can create a hygienic technical environment that gives you the confidence to make significant architecture or technology changes when a new business requirement necessitates them.

Organizations that continue to think “system lifecycle” will, in the long run, lose ground to those that think “system evolution.” If you’re about to invest in a modernization effort, why not do so in a way that positions you for a fundamentally different future?

Much of my work over the last few years has involved helping organizations “adopt” Agile. It is, after all, a poor, unloved orphan and needs to find a good home. The key to whether the new approach sticks doesn’t seem to be affected by how many checklists, process maps or charts of roles and responsibilities we provide; what matters is whether an organization can adjust their collective and individual attitudes.

There’s a great quote from the 14th Dalai Lama that says:

“If you don’t like what’s happening in your life, change your mind.”

Beyond the double meaning of “if you don’t like it, decide to like it” is the more important idea that to change your behaviors you need to change your thoughts.

So that may just seem like good, Gandhi-esque, “be the change you want to see in the world” platitudes. The real question is how can you help an organization shift its attitude from a Theory X to a Theory Y approach to management? Two things that I’ve found to help are words and actions …

Words

The language we use is a direct pathway into our minds – shift your language and your mind will follow. A change in words can facilitate a change in attitude. One clear example of this I noticed was around the use of the word “requirements”. I agree with Steve Yegge’s insight that gathering requirements is wrong-headed to start with (and I love his take on the elevator pitch), but what I’m interested with here is how the word itself is used and the connotations attached to it.

“Requirements” is a weapon word if ever I’ve heard one, particularly when used with the phrase “sign off”. Requirements are all about ordering people to do things: “I demand that you do this. Exactly this. Nothing more and nothing less. I require it!” Requirements brook no discussion and allow no room for invention. More importantly requirements stem from a combative view of relations between development teams and their customers. Signing off on requirements is all about taking a legalistic, defensive approach to covering your behind should things go awry further down the line, which they almost certainly will if you take a combative and legalistic approach to interacting with a development team.

If, however, you shift an organization’s language to talk, not about signing off on requirements, but rather about agreeing on goals, then you begin to shift from a combative to a collaborative mode of engaging. The change in mindset is two-fold: there’s a flip from saying “I” require this of “you” to saying “we” share these goals; and there’s also an expansion from the narrowing, closed, nature of requirements to the broadening and open nature of goals. A simple shift in words helps drive the evolution from defensive and hostile relationships, premised on a lack of trust, to open and collaborative interactions rooted in a sense of common purpose.

Just look at the language in the phrase “requirements capture“. What does that tell you about how requirements are handled? Once captured are they covered by the Geneva Convention or are they still considered enemy combatants?  So when it comes to planning a project, decide whether you prefer white-boarding or water-boarding sessions.

You can witness similar shifts by focusing less on “sign-off” and more on “feedback“. If you get the right feedback in place early on, then final sign-off becomes a mere formality, but if you miss out on the feedback then sign-off becomes a painful and costly hurdle.

So that’s an example of how words can help shift attitudes, what about actions?

Actions

I know of nothing more soul-destroying than sitting in a darkened meeting room while a group of people watch someone update a spreadsheet or a webapp on an overhead projector. You can feel the life and energy vacate the room as people’s attention shifts from the task at hand to suggesting keyboard-shortcuts to make the process easier or just focusing on ways to keep their eyelids open.

I’ve found that a different approach can really help foster a sense of team as you’re making a plan. It’s all about physically engaging the whole team. Here’s how it goes:

• Print or write out your stories / tasks / goals onto index cards, post-its, whatever you have at hand,
• Lay them out on a large table,
• Get the whole team around the table,
• Ask them to move the cards into the order in which they want to tackle them,
• Stand back.

It takes some teams a little while to get going – some people might be waiting for the boss to tell them the order, other people might be reticent or sceptical and someone else will be muttering about Gantt charts and resource leveling – but once someone takes the lead and starts moving cards around, a debate immediately ensues. Hands start leaping in, cards get grouped and moved and the volume rises. Then the table gets crowded, new cards get created, existing cards get rewritten, fierce debates erupt over moving a card a few millimeters to the left or right and a plan starts to take shape.

And it’s not just any plan, it’s a plan that the whole team has collaborated over, buys into and understands. It’s also, crucially, a living plan. Not an untouchable plan that is set in stone and can only be updated by the project manager with multiple sign-off from all stakeholder committees. This is important because we all know that no plan survives first contact with reality.

There are various subtleties to this style of group interaction for a planning session. The subtle movement and grouping of tasks that you can achieve is much more nuanced and multi-layered than in a gantt chart or spreadsheet. Also since no one is “driving” and controlling the mouse the plan that emerges is truly collaborative; the whole team has had an unmediated impact on the outcome. To me, though, the real key is what we’ve learned from the Wii and the group-hug: we are physical, social, creatures and we are most engaged and effective when we embrace that physicality and stand up, roll up our sleeves and get stuck in. If we are just using our eyes and our mouse finger we’re literally not firing on all cylinders.

This is the very reason that approaches like Agile and XP favor stand-ups and huddles, white-board sessions and team walks – start acting like a team and you might actually become a team.

“But what about distributed teams?” I hear you cry. Good question, we’re all out-sourced and off-shored now, so how can we leverage this full-contact planning style?

Have you ever noticed how much teamwork sucks across distributed teams? I think that’s largely because it’s nearly impossible to engender the same sense of team as you can when people sit next to each other, eat together and go out together after work. This will need to be the subject of another post, but the bottom line is that, even though telepresence technologies and collaboration tools can help, your teams will be much more effective if they take into account geographic realities. Enable each group in each location to own their own plan, where possible allow them to form their own teams.

In fact the key is to set goals rather than requirements, focus on early feedback over final sign-off and above all persuade the team to STAND UP!

The big fat lie

I understand how queuing theory can help identify and reduce bottlenecks in processes and have used finger-charts and kanban-boards to do this for a while, but I still find calling this a “pull system” to be slightly disingenuous. All that’s happening is that more “stuff” is being pushed based on a trigger when certain buckets get too low. This reminds me of my annoyance with early technologies on the web that were touted as being “push” but were really just “repetitive-pull” (but not in a good way). I’ve never seen a software organization where the developers have said to the business or product people “we’ve got nothing to do, can you think up some new projects or features for us please?”.

My objection was always that even though looking at a software delivery process as a pull system was vaguely useful for debugging some process bottlenecks, it was a fundamentally incorrect way of characterizing what was really going on. It’s just not true that features or stories are pulled down through the analyze-develop-test-build-deploy-not-necessarily-in-that-order pipeline by the customer or even by a release manager; they are pushed down that pipe by a product or project manager and the pushing often requires a healthy dose of lubricant.

Pushing string

My moment of realization came in a discussion about process over-runs in the definition or analysis phase of a project. The team recognized that requirements weren’t being communicated clearly to the developers and were planning to spend longer on the analysis and definition phase to make the requirement documents longer and clearer. Their diagnosis was correct but their prescription for how to fix the problem would just exacerbate the pain. If people aren’t finding a document useful the answer isn’t to make the document longer!

This situation reminded me of the idea of pushing string. Attempting to push a single-page document is hard (it bends and wobbles), so adding more pages should increase the weight, or “thud-factor“, of the document making it easier to push … surely there’s nothing wrong with this logic!

Empowered to pull

The solution I suggested is to invert the process, empower the developers, and get them to pull the information they need. This fits with the general concept of “people-centered processes” that attracts me to lean and agile approaches (remember a bad process will beat a good person every time). If you allow your development teams to take responsibility for the product and features they’re building they will then pull the information they need at the times they need.

So it may still be a push system for features, but the flow of information is now driven by a pull system. This flip in direction allows all the benefits of just-in-time production and saves the problems associated with maintaining a large inventory. If you write all the details of a feature too far in advance it is certain highly likely that much of that work will be wasted since the details will change or the context will shift. Documents have a shelf-life just like any other organic product and that shelf-life is much shorter than you’d like to think.

There are many measures of code that are objective and metrics driven, but there are others that are more subjective and taste based. I can’t tell you from a quick glance how many afferent couplings there are in a given piece of code, but I do have an almost immediate sense of how elegant the code is. In my mind elegance is all about solving complex problems with simple solutions. That’s the art rather than the science of computer programming.

When I read some code and get that gut feeling that the solution isn’t elegant, one of the first pointers is often how long each method or function is and how deeply indented they get. If you open a class and it just contains a handful of methods that are short (no more than 2 to 4 lines in each) and reasonably named, I breath a natural sigh of relief and know that the class is going to be pretty easy to understand. If on the other hand, there’s some crazy-long Dostoevskian beast of a method that I can’t fully see without scrolling up and down, then I start quaking like an intellectual weakling knowing that I’m going to endure much pain trying to grok what was in that coding genius’ mind when they wrote it. Then once I’ve worked out what they were trying to achieve I’ll have to go and figure out what they actually achieved … in long complicated methods the implementation often diverges from the intent!

Much of this complexity is immediately evident in how the code is nested and indented. Every indentation is the site of some kind of logical fork in the possible pathways through that method. Put in other terms there is a close correlation between the cyclomatic complexity (CCN) of a method and the amount of nesting and indentation. Or explaining this classic metric for complexity another way: each indentation is an invitation to extract a method into a beautiful, small, testable, readable and well-encapsulated unit.

When you’ve spent time living in a codebase where people vigorously accept these invitations to refactor that are offered up by our friend white-space then you tend to have a gut response of pain, horror and loathing when you see a heavily indented chunk of code.

The differences between these two styles of coding is clearly visible in the profile of the first non-whitespace characters on each line. If you look at the code and blur your eyes a bit this profile can look rather like the little LEDs on a graphic equalizer. The profile of indentation within a given source file composes a revealing signature that can be interpreted as a sound wave. Imagine hooking up a music player that could read these patterns. The heavily refactored codebase would purr gently at you with soothing sussurations, while those deeply nested and psychotomatically complex methods would scream at you like a shrill and brain-scrambling siren on a police car.

So step up please you gurus of mashing up Eclipse plugins with Garageband and create me a little plugin that will sing a song of source code whenever I open a file. But before you do that, maybe I should reach for the earmuffs.

Their current flagship project, the hippo roller, though not much more than a tough plastic barrel and pulling handle, is beginning to have an immense impact on the role of women in developing societies.

In many small villages a large proportion of a woman’s day is spent fetching water. This design allows a single person to transport a much larger amount of water much more quickly with much less effort (I might be able to push a car, but I could never carry one on my head). The beauty of this invention isn’t just in the immediate benefit of more water, but in the benefits that flow downstream (all puns intended) of freeing women and children up from this back-breaking work and giving them time to focus on education and self-improvement. This then has a massive cumulative beneficial impact on the rest of the village. All this just from applying a very old solution (the wheel) to a very old problem. Social change through re-inventing the wheel.

In a similar vein Engineers Without Borders have been applying advanced design to alleviate a growing problem in Dafur refugee camps. The problem is that collecting firewood is becoming an increasingly dangerous activity. The area immediately surrounding the camps has been denuded of tinder so the women have to go further and further to gather the wood required to feed their families. This has the double-whammy impact of both increasing malnutrition as less meals are cooked and increasing the chances of being mugged or raped by the marauding gangs in the outlying areas.

The solution the clever kids from Berkeley designed to meet this problem is beautifully simple – make a more efficient stove. The more efficient the stove, the less fuel you need to cook the same amount of food. The less fuel you need, the less time needs to be spent scavenging. The less wood that is removed from around the camps, the less the resources get depleted. The less the resources get depleted, the more people the area can sustain. That’s what I call a virtuous circle.

Another not-for-profit I’ve been lucky enough to be involved with, Inveneo, have a more high-tech solution, but a similar understanding of how to trigger these positive feedback loops by introducing efficiency gains early on in the process. Even though you may pay more for the hardware, if you draw down less power the downstream benefits are magnified to dwarf the initial outlay. By improving the power consumption of their hardware and software Inveneo manage to make computing infrastructure affordable enough for poor societies to sustain the ongoing costs themselves.

What ties these all together is maximizing the efficiency of resource utilization – getting more results from less input. But what makes it even better is that the beneficial results seem to be exponential. The savings fan out as the compounding effects of the initial efficiencies multiply.

The eureka moment for me was realizing that changing the world needn’t involve a lot of hard work – it’s all about leveraging small efficiencies. Archimedes knew this when he said “give me a fulcrum on which to rest and I will move the world.” By increasing efficiency at an early stage in a process the benefits compound along the way, giving us the lever required to shift the direction of our resource depleting way of life.

A user’s interaction with a piece of software or web site is as much emotional as it is functional. Compare the soft, warm, fuzzy feeling you get when first interacting with a product from 37 Signals, say, to the stomach churning reaction you get when booting up Lotus Notes, for example.

This immediate emotional response will pervade the whole of a user’s long-term impression of a product, imbuing their relationship with whatever feeling was conjured up in those preliminary interactions. They say that in most job interviews the interviewer makes up their mind within the first 5 minutes. The same is equally true for software.

That’s the reason that I’m often accused of prematurely optimizing the UI. I’ve been round the block enough times to have been bitten by the following type of interaction:

Me: “This is just a wire-frame, we’re only trying to prove out the functional flows of the application. Don’t worry about the layout or the look and feel – that will be dealt with later.”

Client: “Ooh, I don’t like that font … and I think that’s our old logo. Maybe we should give it a drop-shadow or something. I’m disappointed, I expected more from you guys …”

After a host of similar experiences I now make sure that I give whatever product I’m working on an appropriate level of design and polish for the current stage, while still trying to evoke positive emotional responses. In my mind it’s always worth spending an hour or two tweaking a CSS file to tighten the layout, soften the lines and add a little sparkle – a little :hover goes a long way. Then even if the application isn’t functionally complete at least it looks professional and imbues the client with a sense of confidence and pleasure. This will put them into a much more positive state of mind, which can only be a good thing for the next stages of the project.

Call me shallow if you want, but when I arrive at a (studiously) un-styled site, like our guru Martin Fowler’s, I have a negative knee-jerk reaction making me resist reading the wise words he’s actually written. When, however, I land on a site where they care about the first bite I feel compelled to carry on reading all day.

If someone in a meeting has a bit of chopped herb stuck in their teeth it’s really hard to focus on what they’re saying – you get distracted and have a negative emotional response. In the same way drizzling some coulis and sprinkling a bit of chopped herb onto a plate before serving can provoke a positive emotional response which fools the diner into enjoying their food more.

When we came onboard manual intervention was needed for any of their builds or deployments to work and so it was rare for more than a couple of builds or deployments to be completed successfully in a day. Now we often have up to 1,000 builds running every day – what’s more the majority of them now pass!

This article looks at a few of the techniques we’ve had to put in place to enable this transformation and what we’ve learnt along the way.

Dividing up builds – separation of responsibilities

Initially we had two versions of the build – one that would run on commit and a second that would only run at night. The two versions did pretty much the same things, except the nightly builds would deploy to a shared testing environment. Deployments only happened at night to avoid disruption to the multiple teams working in this shared environment (another smell to be dealt with later). The on-commit build did compilation, unit testing and packaging – the nightly build did the same and then handled environment preparation and deployment.
There were a few problems with this:

• The on-commit builds ran slowly because they were doing a lot of work
• The nightly builds were very brittle since responsibility for different parts of the build (compilation, packaging, deployment etc) was split among multiple teams – a failure at any step would mean QA didn’t have a new build to test that day
• It wasn’t DRY

Our solution was to divide the monolithic build up into several smaller steps and tie the steps together into a “build pipeline”, with each step triggered by a successfully completed “upstream” build. Our pipeline was divided into quick, assemble, package, deployment and regression builds. “Downstream” builds would pick up the artifacts prepared at the previous step in the build.

This lead to various benefits:

• Each build ran more quickly
• We didn’t repeat steps
• It was easier to divide up responsibilities for keeping the different builds green:
  • Developers :: quick build (compilation + unit tests)
  • Build team :: assemble build (jar, war, ear creation etc)
  • Deployment team :: package + deployment (preparing RPMs / configuration / deploying)
  • QA team :: regression suites
• Because we used the “last known good” upstream artifact, deployments weren’t blocked by a last minute problem upstream in the pipeline.
• We were now in a position to split the builds up over more boxes to achieve greater throughput through parallelism

This concept of dividing the responsibilities (at least for first and second line build support) had a major impact on improving success rates and response times to failures through an increased sense of ownership of the process.

Tying the builds together – HTTP publishing

In an ideal world the input for a downstream build should be the output from an upstream build – the appearance of a new artifact from one build would trigger the next build down the chain. In our situation that wasn’t quite possible (the server topology and optimal dividing lines between builds, from a speed point of view, didn’t favor communicating via artifacts) so we took a different approach.

Our solution was for builds to communicate with each other via small properties files, published locally and accessed over HTTP. On successful completion each build publishes a properties file (to the local Cruise web-server) containing the build number and SVN revision of the build. We built a custom Cruise publisher to handle this. We also built a custom Cruise HTTP modification-set to watch for upstream builds and an HTTP label-incrementer to keep all the build numbers in the pipeline in sync. The same build number is used through each of the steps of the build, making it easier to check whether changes have flowed through to the QA environment – more on this later.

Watching the river flow – visualizing the build pipeline

Once we had divided up the builds and added further applications, we ended up with close to 100 builds that needed to be managed and monitored. Since they were spread across multiple boxes to increase throughput this wasn’t as easy as just watching one large cruise page. So we built a centralized page for aggregating together the build results.

The dashboard page was comprised of a few separate components:

• Current build status grid
• Graph of passing v failing builds over time
• Graph of build times per type / step in the pipeline
• Quick list of which builds are currently broken
• Some basic code metrics

We also provided similar drill-down pages, filtered just to show data on a single application or step in the pipeline. These then link through to the actual Cruise pages for log files, failure reasons and test results.

Current build status grid

The current build status grid was probably the greatest breakthrough. The basic idea was to show the current status of all our builds in a simply structured manner. Over a few iterations we evolved a grid structure (a bit like battleships!). The columns in the grid represented steps in the build pipeline and the rows the different applications we were building. The columns were repeated for each active branch. Cells were then green if the build was good, red if it was bad and yellow if the last build was good, but was more than 24 hours old (stale). Mouseovers give more details about the selected build, such as when it ran and what its build number was.

From this view informative patterns were clearly distinguishable with a cursory glance. It became obvious if problems were localized or spread across a specific application or step in the pipeline.

Initially we emailed a snapshot of the grid first thing in the morning and at the end of the day to management and the teams. This had the wonderful effect of halting all the annoying questions and sparking the interesting ones. Instead of people asking whether a build was successful or what build number was last deployed to QA we got questions like:

• Why are all the assembly builds broken?
• Why is all of application X broken?
• Why hasn’t application Y built for 2 days?
• Why is there a build on the prod support branch?
• Wow – everything’s green – shall we go to the pub?

This definitely shone the spotlight in a few uncomfortable places, but the visible improvements this lead to made it worthwhile. It became an invaluable tool for educating people about how the pipeline works. Once people understood and were reliant on the data we replaced the emails with a dynamic, self-service, application hosting the data.

Graphing build metrics over time

We also had great success with some of our time-based visualizations. Simply graphing the times of each step in the build pipeline over the last month quickly highlighted problems as they developed. In one case we realized that all the packaging builds were getting geometrically slower and slower – a quick bit of research showed that each build was adding new artifacts to SVN and the ensuing checkouts were suffering. We fixed this problematic practice and had a massive positive impact on the total throughput of the pipeline.

The historic view of successful vs. failing builds over time was also a particularly useful tool while we were focusing on improving the stability of our deployments to QA / UAT. This gave us a simple report on how we were doing – 50% success last week, 75% this week – woo hoo!

Sparklines

We found a simple yet effective way of communicating a large amount of information about how a build has been performing by using “sparklines”.

For a given build these graphs communicate the number, duration and success of a large quantity of builds. You can quickly get an idea of how things are trending. Again mouseovers provide more details, context and drill down.

Under the hood

Initially the grid was generated and emailed by a scheduled round-robin script hitting all the build servers. But this was slow, required us to configure all the build server addresses in one place, wasn’t fault tolerant and stored no build history. So we moved the build telemetry dashboard to a simple Rails webapp employing CSS wizardry for the visualizations. The data was populated via a basic RESTful API. Each build has a custom HTTP publisher (again we built this) that fired off a simple set of information about the build in a POST request:

• Project name
• Build label
• Date / time
• Build duration
• Build successful
• Hostname of the Cruise server

Build naming conventions

The one extra key to making this all work was standardizing the format of the Cruise project name so we could parse out the application, branch and pipeline step without adding any extra complicated meta-data. We chose “<branch>-<application>-<step>” as our format, e.g. “trunk-bigapp-quick” or “3.20-littleapp-assemble”.

Where’s my build?

Using our custom label incrementer meant that each build in the pipeline for a given SVN revision would have the same build number (e.g. “trunk-bigapp-quick.123”, “trunk-bigapp-assemble.123”, “trunk-bigapp-deploy.123”). This made it much easier for developers and QA to work out what builds and features had made it through to the QA environments. We built on this by providing a customized version of the currents status grid to answer exactly that question – where’s my build?

The grid displays the latest build number at each step – clearly showing how far changes have progressed and also highlighting how a broken step breaks the pipe. This has become a vital self-service tool for the dev, QA and deployment teams, and has saved the build team from a lot of very dull questions.

A growing awareness of the importance of the pipeline has allowed us to devote efforts towards optimizing the total throughput. All the good lean principles definitely apply at this point – looking at optimizing the whole system and working out where we’re queuing or waiting too much nearly always deliver the best results.

And the rest …

These tools freed us up to use the build team as a beach-head for spreading general coding, development and automation best practices throughout the organization. Some of these included:

• Untangling spaghetti dependencies with Ivy
• Spawning multiple virtual machines for regression testing
• Automating Cruise configuration and version updates
• A whole host of black-belt Ant Fu
• Correlating multiple code metrics to drive code quality improvements

But those are stories for another occasion …

[This article was originally written for an internal ThoughtWorks innovation newsletter - I've made some minor edits for this blog]

The mailing lists at ThoughtWorks have been running hot since Jay Fields stoked the flames of the dynamic versus static language schism. His basic point is summarized in the opening line:

“Given a good test suite the return on investment simply does not justify the use of static typing”

Basically he’s saying that statically typed languages have benefits, but there is an associated cost – anyone who’s seen object definitions disappear off the screen with Java 5 generics will understand this. The benefits definitely exist – modern IDEs (auto-complete, refactoring, etc), compiler optimizations, basic verification and clarity in communicating integration requirements – but there is also a cost incurred in terms of complexity and inflexibility. Neal Ford has started calling this the “static type tax”. Jay’s solution is, rather than relying on static typing for partial protection, rely on tests for full protection.

Work to rule

This pain or tax associated with static types reminds me of the parody of union rules gone bad – “I can’t plug that in, I’m a carpenter, you need an electrician”. These rules start out having important uses (protecting workers against abuse and ensuring quality results), but can end up just causing pain and making the jobs they’re trying to protect less secure by making the industry less responsive. There is an obvious correlation with the trade-offs between dynamic and statically typed languages: it’s nice to know from an integration and tooling perspective exactly what type of object a method requires; it’s also nice just to be able to give that same method any object that can respond to the messages it will be sent.

In strong and statically typed languages, like Java, the method signature is a shorthand explicitly stating what type of object is required for the method to work. In a dynamic or weakly typed language, like Ruby or JavaScript, you have more flexibility about what you can pass into the method, but less visibility into whether it will work or not – you have to check (visually or through tests) that the object you pass in can respond sensibly to the instructions it’s given.

One of the pains of the statically typed method signature is that various different requirements tend to get bundled up together. A method may well require a parameter to be of type (implement interface) X, where interface X defines 10 methods, even though the method will only call one of those declared methods. Ideally you would define a new interface for each combination of methods you want to call so you can keep the contract as simple to fulfill as possible, but this in itself would be a burden as the amount of interfaces would explode.

Risk assessments

The arrival of complexity and restriction in the name of preventing accidents (passing an object that can’t react to the messages sent to it) reminds me of how school trips have become harder and harder to organize because of the ever more stringent “risk assessments” required:

“Endless lengthy risk assessments and fears of being blamed if anything goes wrong have put teachers off taking children away on trips and led to youngsters being cooped in classrooms instead.”

I remember loving some fairly wild school or youth club trips as a kid. Rock scrambling was one of my favorites, which involved climbing, jumping, swimming, hopping and whooping round sections of coastline like the beautiful Lulworth Cove. Sure there were risks involved, but learning to handle yourself in those risky situations lead to really important lessons and personal growth. Risk breeds responsibility.

I have similar feelings about using dynamic languages – with great power comes great responsibility. Programming itself is inherently risky (when was your last zero-defect release?) so programmers have to make decisions about how to act responsibly. One way is to impose risk assessment forms on every method call through static typing; another option is to force objects to act responsibly and be good citizens. This is the route that interests me more. Testing definitely plays an important role in acting responsibly, but it’s not the end of the story.

Team collaboration : object collaboration

Coming back to the analogy of objects as workers on a job, rather than using strict role definitions in an attempt to control quality, I would much prefer to use the example of self-organizing Agile teams. On an Agile development team there is an expectation that people can work in a cross-functional manner with common ownership of the code-base. This means that team members should be interchangeable and comfortable in a variety of areas – just like in dynamic languages there is no type checking before picking up tasks (“sorry I only do JavaScript / Java / SQL”). This leads to highly flexible and productive teams, but there are a few techniques needed to make it work. Some of these techniques are conceptually transferable to working responsibly with dynamic languages:

• communication
• simplicity
• responsibility

Explicit and clear communication is the biggest of these – the others can be derived from this principle. The importance of communication in teams is fairly obvious, but it can also be important at the code level in areas like explicit tests, clearly named methods and transparent intentionality.

Simplicity aids communication and also lowers risk. The simpler the routine is, the easier it is to understand, the easier it is to see what is required of collaborating objects and the less likely it is that bugs will creep in.

Responsibility underpins this all. In teams it means talk to people if you’re not sure, or tell people if you’re going to do something strange; in objects it means testing your interactions, handling errors appropriately and checking compatibility where necessary.

Logicalis have put together some advice on how to reduce the impact of IT on the environment. The suggestions range from reducing hardware requirements through virtualization and other consolidation techniques to old favorites like double-sided printing, video-conferencing, electronic forms and turning off your desktop at night.

One idea I hadn’t really thought of was the impact of data archiving strategies (or Information Life-Cycle Management as they grandly title it) on energy usage. Data needs hard drives need power. Choosing what to compress, archive and delete can have a major impact.

If you don’t think that powering down your PC at night can make a difference Global Action Plan’s recent report might open your eyes as it shows that the environmental impact of IT is almost identical to that of the much-maligned aviation industry.

Ultimately, as with so many environmental issues, the best solution is to make sure that the people who make the decisions are the same as (or at least aligned with) the people who are impacted. I’m not saying sys admins should go and live in the semi-submerged Bangladeshi delta. My advice is much simpler – make sure IT departments are responsible for their electricity bills. The most striking statistic in the “An Inefficient Truth” report (PDF 1.3MB) was that not only do 68% of IT departments not pay for the energy their systems consume, 56% of them don’t even see the bills.

Making environmentally sound decisions isn’t about warm fuzzy feelings – it’s about hard-nosed, joined-up, economic thinking.