The thinkfoo test for Continuous Delivery – What does good look like?

Dave Farley makes an excellent point in his recent post that many people in our industry have never worked on an efficient project team and wouldn’t know good if they saw it. Indeed many are probably blissfully unaware of just how inefficient they are comforted in the knowledge that they seem to be no worse than anyone else. So how do you know if you are good?

Some fourteen years ago Joel Spolsky wrote an often quoted piece he called the Joel Test.

This was great stuff back in 2000 but we have moved on a long way since then and I propose an updated test that reflects modern development practices founded in the principles of Continuous Delivery (CD).

I’ve worked with some incredibly productive teams (and some not so productive ones) I prefer to work with productive ones and I have over the years formed some strong opinions of exactly what good looks like. I realised I have been informally using a set of criteria to evaluate teams for some time and thought I’d write it down and share it.

So here is my “thinkfoo test”; efficient development teams will be able to answer yes to most if not all of these questions, if your team can’t and you think you have an efficient team with a good process maybe its time to take the red pill and think again?

1. Do your developers push changes to a single trunk branch at least twice a day?

2. Do you practice ATDD and write automated acceptance tests for every feature?

3. Do you practice TDD?

4. Do you run your full set of acceptance tests, unit tests and performance tests on every commit to trunk?

5. Do you have automated single step repeatable builds and deployments?

6. Does your team release to production at least every two weeks?

7. Do you practice pairing as the normal way of working?

8. Do your teams self organise?

9. Is your team in a noisy collaborative environment?

10. Do you have cross functional teams of BA’s, developers, QA and operations responsible for end to end delivery of working software?

11. Do you have a single product backlog (with bug’s and features together)?

12. Do you use minimal stories as specifications rather than detailed up front designs?

13. Do you have regular retrospectives to continually improve your process?

Lets look at each of these in a little more detail

1. Do your developers push changes to a single trunk branch at least twice a day?

  • feature branching is evil, long lived feature branches doubly so. Trunk development is, for good reason, a key requirement for CD.
  • trunk should be in an always releasable state, every commit should be a release candidate.
  • To be continuous your integration must be frequent; committing to mainline twice a day should be considered a minimum.

2. Do you practice ATDD and write automated acceptance tests for every feature?

  • these should be built in advance and alongside the feature and treated as the source of truth or living specification. The feature isn’t done until all the acceptance tests pass. By doing this and keeping work in progress to a minimum the need for bug tracking is diminished.

3. Do you practice TDD?

  • many people are confused about what level to set their tests at that’s why I specifically want to know if you have both ATDD (are you building the right thing) and TDD (are you building it right).
  • The “red, green, refactor” cycle is critical to producing good clean code
  • Your whole team needs to practice TDD, I wince every time a development lead says “We don’t enforce it but sure you can work that way if you want”.

4. Do you run your full set of acceptance tests, unit tests and performance tests on every commit to trunk?

  • #1 is a prerequisite of this if you are not using a single trunk your continuous integration isn’t, end of.
  • performance tests are a critical part of your build pipeline many products fall under the pressure of their own success.

5. Do you have automated single step repeatable builds and deployments?

  • if you can’t do repeatable automated deployments there’s not much point in doing #1 through #4 as you won’t know that what you deployed is what you tested.

6. Does your team release to production at least every two weeks?

  • if you do iteration after iteration without release I don’t really buy that you are agile, this is usually a waterfall project dressed in agile clothing.

7. Do you practice pairing as the normal way of working?

  • see my post on pairing for my views on why this is essential to building quality software assets.

8. Do your teams self organise?

  • teams that do top down allocation of tasks in my experience always end up optimising resource usage over feature delivery, it discourages collaboration, common code ownership and pair rotation don’t do it!

9. Is your team in a noisy collaborative environment?

  • the opposite of Joel here I agree with uncle bobs views of headphone clad developers with their backs to each other, no single developer can be productive enough to jeopardise whole team productivity don’t allow it and don’t hire people who insist on working this way. Great software is built by teams not individuals beware the hero developer!

10. Do you have cross functional teams of BA’s, developers, QA and operations responsible for end to end delivery of working software?

  • and the roles should be blurred rather than designated, BA’s should write some tests, QA’s should write some code, developers should write deployment scripts and talk to customers, ops guys should be coding etc etc.
  • Your developers should be on first line support for your production systems.

11. Do you have a single product backlog (with bug’s and features together)?

  • do the most important thing first and optimise for flow of delivery over resource utilisation (see managing your backlog for more on this)

12. Do you use minimal stories as specifications rather than detailed up front designs?

  • stories are about enabling conversation not detailed specification see for some great thoughts on user stories.
  • Joel believes in detailed up front specifications sorry but he is just wrong on this, in any project lasting more than half an hour the specification will change, living specifications in the form of automated acceptance tests are the only way to keep specs current.

13. Do you have regular retrospectives to continually improve your process?

  • nothing is perfect; core to the principles of Continuous Delivery is continuous improvement retrospectives are an essential part of that learning and improving.
Tagged ,

TDD teaches you how to code well

You may have seen the recent   Kent Beck, David Heinemeier Hansson, Martin Fowler discussion on Is TDD Dead? where David Heinemeier Hansson contests that TDD is “poisoning” good designs. My take on this can be summarised with the below diagram;


Not all Test Driven Code is good and not all Good Code is test driven. But a lot of code (probably most) is neither good or test driven. I contest that to be considered good, code must be both well designed and well tested. Now I could have labelled the above “tested” or even “testable” but I have seen very little code that is written without TDD which gets anywhere near approaching “well tested” or even “testable” no matter how well intentioned the developers. Moreover I believe that by practising TDD you have much better chance of getting to good than by not doing so.

I say this because I have found that people who consistently write good clean code have almost invariably been people who have mastered TDD. Why could this be?

I think the reasons for this is are that in applying TDD you learn that keeping your tests manageable is at least as hard as writing the underlying code. To avoid brittle difficult to understand tests you want tests that test only one thing and you have to balance this with code that is itself manageable and does one thing well. Consequently;

TDD leads you to code that has small interfaces and low cyclomatic complexity.

TDD tells you things about your code; Tests that require a lot of dependant objects to be constructed or mocked are telling you your code has high coupling. Tests that test more than one thing can hint at poor cohesion – classes under test are doing too many things.

Learning about techniques like mocking and stubbing teach you about coding to interfaces rather than concretions.

TDD teaches you about best ways to handle error conditions and exceptions

TDD teaches you about separation of concerns, why for example its beneficial to separate threading from business logic.

As you get practice writing tests you learn that testing multiple layers of abstraction at the same time is hard and intuitively start to understand what layers of abstraction really mean.

TDD also teaches you to understand what it is to be a user of your code and code for the users rather than the implementers benefit.

In becoming a master practitioner of TDD you provide yourself with all these learning opportunities. Some people having done this feel that TDD can occasionally be overkill (Dan North talks about Spike and Stabilise and opportunity cost of TDD here) but it is the very expertise they have gained from TDD that gives them the ability to make this judgement and to proceed in a way that is safe to retrofit tests to. It’s worth pointing out though that most people I know that have have become competent at TDD never really want to code without it (myself included).

So finally if you are wondering if your code is good or bad and haven’t yet mastered TDD I would suggest it probably has considerable room for improvement.  Check out Steve Freeman and Nat Pryce’s Growing Object-Oriented Software Guided by Tests and Kent Beck’s Test Driven Development.  But also check out what good code looks like, understand and apply SOLID read Uncle Bob Martins Clean Code

Tagged ,

Why pair programing is as much about business continuity as it is about code quality.

I heard a statement recently, it’s a common problem and is probably familiar to many of you.

It goes something like this ;

“We will need to wait until next week when Max is back from holiday to do that work, that module is his baby ”

When I heard this my initial reaction was confusion, why was no one else able to work on this code? It was in the same language, the same repository, was built in CI, had tests it made no sense. Then it dawned on me no one else in the team had EVER worked on this module, they had no clue what it did it how it did it, confusion turned to horror, did anyone know what this guy’s code was like? What if he left for good, got hit by a bus? Was this module going to rot forever or need rewriting?

I realised I’d heard this in a different form some time previously in another organisation from a new CTO who had not had extensive experience in a pairing environment;

“So Paul is leaving, what do we need to do for handover? ”

It left us old hands a bit stumped sure we were losing a very talented developer but there was nothing unique about his knowledge of our code that needed “handing over”.

The common negative response to pair programming from management is “I can’t afford to use two people to do one persons job” . As developers our common reaction to this is to explain how pairing can be faster, result in less bugs etc. This is understandable as these are the benefits most obvious to developer. But I would argue that it is business continuity that could be the killer argument for pairing.

Done correctly pairing addresses business continuity in a number of ways;

Knowledge sharing – many people have actually worked on the code this is far superior to a code review; think of pairing as having co-written the book and code review as being told the story by someone who has seen the film.

Code becomes easier to understand on each pair rotation (pairs should rotate or swap partners at least once a day to reap the benefits of pairing) as this rotation occurs small improvements in code clarity are made as the new person gets their head around what’s been done so far. This makes it a much more valuable and flexible piece of software six months down the line when some new requirement is needed of it.

No ego’s: the code is collectively written and ergo is collectively owned – no need to wait for Max to get back from holidays if it needs changing today it can be worked on today without upsetting anyone.

It’s a truism to say that code takes on the form of the team or teams that write it. If only one developer is working on a system the benefits of breaking down the code into modules are diminished a single person can only work on a single piece of code at a time. This is one reason why it’s common to see software written by a team of single developers be one big blob of software. I have seen pairing address this in this way: as pair’s rotate, realising that a piece of work can be done in parallel if the code is structured in a particular way they do so and your most important feature gets into your customers hands faster.

The software a company builds should be considered a company asset, not some individuals baby. I have seen pair programming to be an extremely powerful way to ensure this is the case. If your organisation is developing software the question you should be asking is not can you afford to have developers pairing but can you afford not to?

dave speaking


Watch my talk at the pipeline conference where I discuss introducing pairing and other XP practices to a team.




Tagged , ,

Using story swarming to reduce WIP

Some years ago I was on a training course on lean thinking and one particular exercise has stuck in my mind ever since; Teams of six or so were given a piece of A4 paper and were told to arrange themselves so that none of them were in contact with the floor. The solution most people took was simply to get all of the team to stand on the paper at the same time. The tutor then folded the paper in half and told us to repeat the exercise. It was still possible on one foot for the team to stand on the paper. But the paper was folded again and again until people were resorting to climbing on one another or declaring the task impossible. Finally the tutor took the piece of paper away entirely. Suddenly one team realised that the paper was completely surplus to requirements and the goal could be achieved even with no paper at all by having the whole team jump on the count of three. The point of this exercise was to demonstrate “Muda” or waste and how the presence of a tool or resource tends to make people use it even if in practice it is not needed at all.

One of the tenets of lean production, which as you will know is one of the foundations of the approaches to software development collectively know as agile, is the reduction of waste in all its forms. Inventory that is not being used and work in progress (which is product that is not yet ready) is an example of one of seven “wastes”. In software development code is inventory, if this code is not usefully in production it is my conjecture that this is waste and we need to find ways to minimise it.

One way of addressing this waste will be familiar to any practitioner of agile development; namely the use of an iterative approach, one of the benefits this gives is that we deliver useful features into the production code every iteration hence reducing the wasteful inventory of code that is work in progress. We also strive to prioritise our work so that the most important feature is delivered first and we do not put wasteful effort into less important features and to ensure that we do not “over deliver” excess inventory that is not yet needed..

You’ll hear it stated frequently that having a single product backlog that is ordered as a single queue is essential to a successful agile project. The idea being that work is done in strict priority order. In practice what often then happens is that each pair will grab the top available card in the backlog and begin work on it, the next pair will grab the second and so on. What this means is that in a team with six developers you suddenly have three cards running in parallel typically all placing conflicting demands on your QA’s, BA’s and the customer at the same points in an iteration which then leads to features being held up or bugs being found late in the day. Worse still it could be your top priority card gets held up because of some problem with a lower priority card that conflicts with (either through necessity or some error in the solution).

A consequence of which is that some or all (and not necessarily the lower priority ones) end up being incomplete at the time of cutting a release. Subsequently juggling of features which are ready or perhaps not wanted yet has to occur. Methods to tackle this include “feature branching”, “branch-by-feature” and “feature switches”. These allow us to switch on or off particular features either by clever use of version control systems or with a code switch to “turn off” code that is not ready. You will hear proponents of both approaches. But it seems to me that whichever approach is used the wrong problem is being addressed in that they are all ways of dealing with excess code inventory be it code that is not ready to put into production at the end of an iteration or code that delivers a feature that is not wanted yet.– there are great tools to help with this to be sure but my point is that if this is needed at all;

(a) There is excess inventory either in the form of unfinished code or in the form of features delivered ahead of when they are wanted or needed

(b) A process has been introduced purely to deal with this excess inventory, which in itself is a form of waste.


So what else can we do to deal with this excess inventory? Well the thing here is to eliminate the excess in the first place.

An alternative approach to the pair by pair grabbing of stories, and one which is getting some attention, is to have your entire team “swarm” on a story until its done. This basically means that the whole cross functional team (Devs/QA/BA/Customer) works on the top priority item together breaking down the whole feature into small manageable tasks (often each of these takes less than a couple of hours for a pair to complete)

We have been swarming on stories for quite a while now and I would like to share my observations on some of the benefits we have seen;

  1. Our top priority story gets all the attention this means it gets delivered first and sooner than it would if the team was spread out on other stories we have found that even unusually large stories that we would have previously expected to take most of a two week iteration often get completed in a few days.

  2. The team delivers testable functionality early on in an iteration keeping QA’s busy and revealing bugs quickly

  3. The customer is focused on the item most important to them and has a clear view of the progress being made and what the team is working on

  4. More robust to absence whether planned or unplanned as all of your team knows about the feature being developed we have found for example that there is better pair rotation and BA’s and QA’s get included more frequently in those pairs.

  5. The code delivered ends up coherent as the context switching that can occur with pair rotation and multiple stories in play does not occur.

  6. And of course it encourages small frequent commits and CI

  7. We have found that we rarely need to “turn off” features with this approach at the worst case scenario you have a single story in play at the point you want to release and in practice this has not often occurred but where it has a relatively simple approach with feature toggling has sufficed – its only a single feature switched off and it won’t be in production switched off for long after all as its our top priority story for the next release.

In summary I think if you are finding the need to use advanced version control tools to manage feature branching, or regularly using coded feature toggles to manage multiple “in progress” features you are probably suffering from Muda in your process and I would highly recommend that you give swarming a try after all do you really need that piece of paper or can you just jump?

Tagged , , ,
%d bloggers like this: