Can Continuous Delivery succeed where Agile has failed before?

Follow @thinkfoo

It’s my hypothesis that Continuous Delivery through its focus on the actual mechanics of the production of software provides us a step change in how we deliver value through software to the business. I also argue that the failure of what is perhaps the most popular Agile approach in use today, Scrum, to put this at the heart of the development effort has been a key contributor to the disappointing results of many agile transformation programs.

The leader on scrum.org on “What is Scrum?” says;

“Scrum is a management and control process that cuts through complexity to focus on building software that meets business needs. Management and teams are able to get their hands around the requirements and technologies, never let go, and deliver working software, incrementally and empirically.”

From https://www.scrum.org/Resources/What-is-Scrum

So guess how many times the Scrum Guide, also available from that site, mentions the words  “Software” or “code”? Quite a lot you’d think yes? But no, the actual number of times is zero, not once, zip, zilch that’s right not one single mention of the very thing its supposedly focussed on building. In fact I’d argue that if you read this guide and did not already know that Scrum was about building software, there is no way you could ascertain that from the text.

Don’t get me wrong, there are many good things in Scrum, but it’s focus exclusively on the organisational patterns and processes is simply not enough, on its own, to ensure the delivery of working software that meets business needs.  This much was recognised by Jeff Sutherland when he wrote about the very first Scrum project;

“…a key to entering a hyperproductive state was not just the Scrum
organizational pattern. We did constant component testing of topic areas, integration of packages, refactoring of selected parts of the system, and multiple builds per day.”

From AGILE DEVELOPMENT: LESSONS LEARNED FROM THE FIRST SCRUM By Dr. Jeff Sutherland October 2004
https://www.scrumalliance.org/resources/35

Actually talk of code is very much at the fore in this early six page document on Scrum; it mentions code 12 times and software 36 times. Somewhere along the line however it seems that Scrum has forgotten to emphasise this central element of a successful product, one that to my mind is far more fundamental than organisational patterns. Unfortunately I’d argue that it is these elements of eXtreme Programming that were much more crucial to the success of a software project than the organisational concerns that Scrum majors on.

Ironically this lack of talk about code, I think, is one of the reasons using Scrum as the route into agile has been a popular choice as the route in for non-technical management. It talks about things they understand and inputs they can influence without worrying them about the tricky technical stuff.

Because we fail to ensure the basic building blocks of continuously delivering working software are in place we have repeatedly undervalued the importance of key technical skills. Its my experience that introducing agile organisational practices without the fundamental technical practices in place is not only unlikely to improve matters but can be downright dangerous – trying to deliver working software every couple of weeks without test automation, or clean code inevitably slows to a point of failure. Often this will result in cargo cult ceremonies instead of actual improvements. You may have seen the BBC Red Dwarf episode where Rimmer spends most of his six week period of revision fine tuning his revision timetable, leaving him with no time to actually do any revision. It seems to me that the focus on organisation patterns and process we see so often in agile leads to us failing to spend any time on actually writing better software.

Enter continuous delivery;  in contrast to the  Scrum Guide Continuous Delivery mentions software 588 times and code 405 times, now I accept that this is a bigger document but at 463 pages that’s still over twice per page on average for the two words combined. There is no doubt what this book is about – its about automating the software delivery process. It focuses on the desired outcome – how to get working software reliably and predictably into the hands of users.

Continuous Delivery, by building on XP  and Lean, provides us,  I think, with a step change in software delivery. By bringing the basic engineering practices of software, automation, building in quality, TDD, ATDD, clean code and code craftsmanship to the fore we put the code where it should be, right at the heart of our development efforts. All of the really effective development teams I have seen have these engineering practices at the core of how they deliver software. In fact by focussing on automating the mechanics of the software process they have, perhaps paradoxically, been able to consider the softer organisational issues that may be even more challenging.

Start with Continuous Delivery other good stuff can follow.

The fundamental building block’s of automation and good engineering practice must be seen as at least as important as the introduction of organisational patterns like Scrum. Scrum and CD practices can be used together successfully, but whilst great benefit will almost certainly be gained from CD alone, using Scrum in isolation is, I believe, doomed from the start.

As a final thought, if you are a non-technical manager reading this article you may be assuming your technical team are learned in these core technical practices, sadly the state of our industry is not so great. Anecdotal evidence suggests only around 1/3 of developers have or apply these core engineering skills on a regular basis, a quick search on a popular job search site today threw up the following numbers (within 100 miles of London);

2,218 jobs for agile
779 jobs for (BDD or ATDD or TDD or “clean code” or craftsmanship)

Not very scientific, I admit, but its consistent with other anecdotal evidence from show of hands at conferences etc. and it gives an indication of the perceived relative importance of these skills. I say again I have never seen Agile Software development be a success without these things at the core – Like Rimmer we need to stop worrying about our revision time table and focus on the actual revision.

See my talk on how implementing continuous delivery turned around our development efforts here.

Follow @thinkfoo

Roulette Kata – exploring the affect of Time and Randomness in TDD

Follow @thinkfoo

I did this Kata in response to a challenge set by @jjefries1 here
attempted by @RonJeffries here

For ease of reference the summary of the kata is repeated here;

A roulette wheel has numbers from 0 to 36. A spin of the wheel takes 20 seconds and leaves the ball on  a random number.

Design and implement a roulette wheel using TDD. Think about how using time and random numbers affects your design. Consider how much functionality each test covers and what responsibilities you are testing.

I wanted to drive this in a way where I did not need to mock out clocks or random number generators. To do this I thought that a pattern I have seen used extensively before would be suitable; where the generation of tick events was the responsibility of one class, whilst another had the job of responding to those events. Secondly I planned to  going with the model of “Object’s send messages to share data” so I knew straight away that I would be looking at a tell don’t ask approach.

The first notes I made on the kata were:

This is event based so avoid “World” stuff in test/codelets and to try for public methods are have void return types.

That about constituted the amount of “up front design I did before hitting the keyboard.

My very first failing, as opposed to not compiling,  test looked like this;

	@Test
	public void shouldNotifyStoppedAfterSpin() {
	WheelObserver wheelObserver = mock(WheelObserver.class);
	RouletteWheel wheel = new RouletteWheel(wheelObserver);
	wheel.spin();
	verify(wheelObserver, times(1)).stopped();
	}

view raw RouletteWheelTest.java hosted with ❤ by GitHub

Note that at this point I am not taking any interest in time at all. Only that the observer is notified that the wheel has stopped after the spin.

I questioned where to go next and in writing my summary from the kata on cyber-dojo I think I went the harder route to start with. I took on verifying random number outputs, this took several goes round the red/green/refactor cycle and the final test I ended up with was a bit of a behemoth;

	@Test
	public void shouldSpecifyRandomBallLocationWhenStopped() {
	final boolean seenAll[] = new boolean[1];
	seenAll[0] = false;
	WheelObserver wheelObserver = new WheelObserver() {
	Set<Integer> seen = new HashSet<Integer>();
	public void stopped(final int location) {
	if (location < 0 || location > 36)
	throw new IllegalArgumentException();
	seen.add(location);
	if (seen.size() == 37) seenAll[0] = true;
	}
	};
	RouletteWheel wheel = new RouletteWheel(wheelObserver);
	for (int x = 0; x < 1000; x++)
	{
	wheel.spin(0);
	wheel.tick(20000);
	}
	assertTrue(seenAll[0]);
	}

view raw test.java hosted with ❤ by GitHub

To push the design forward my next test looked like this;

	@Test
	public void shouldSpecifyBallLocationOnceWhenStopped() {
	WheelObserver wheelObserver = mock(WheelObserver.class);
	RouletteWheel wheel = new RouletteWheel(wheelObserver);
	wheel.spin(0);
	wheel.tick(20000);
	wheel.tick(20001);
	verify(wheelObserver, times(1)).stopped(anyInt());
	}

view raw gistfile1.txt hosted with ❤ by GitHub

This as you can see led to some changes in the interface. Which I didn’t jump straight to but my RouletteWheel class was starting to take shape;

	import java.util.Random;
	public class RouletteWheel {
	private final WheelObserver wheelObserver;
	private long timeStartMs;
	private final Random random = new Random();
	public RouletteWheel(final WheelObserver wheelObserver) {
	this.wheelObserver = wheelObserver;
	}
	public void spin(final long timeStartMs) {
	this.timeStartMs = timeStartMs;
	}
	public void tick(final long timeMs) {
	if ((timeMs - timeStartMs) >= 20000)
	{
	final int location = random.nextInt(37);
	this.wheelObserver.stopped(location);
	}
	}
	}

view raw RouletteWheel.java hosted with ❤ by GitHub

There were quite a few more evolutions of this but the next interesting bit of work to talk about on a TickProvider class, something to generate those tick events

	@Test
	public void shouldNotifyTickWhenStarted() {
	RouletteWheel rouletteWheel = mock(RouletteWheel.class);
	TickProvider tickProvider = new TickProvider();
	tickProvider.start();
	verify(rouletteWheel, atLeastOnce()).tick(anyInt());
	}

view raw TickProviderTest.java hosted with ❤ by GitHub

Again we are getting the basic communication right without worrying about timing.

to get to that we eventually ended up with;

	import org.junit.*;
	import static org.junit.Assert.*;
	import static org.mockito.Mockito.*;
	import static org.mockito.AdditionalMatchers.*;
	import org.mockito.InOrder;
	public class TickProviderTest {
	@Test
	public void shouldNotifyTickWhenStarted() {
	RouletteWheel rouletteWheel = mock(RouletteWheel.class);
	TickProvider tickProvider = new TickProvider(rouletteWheel);
	tickProvider.start();
	verify(rouletteWheel, timeout(150).atLeastOnce()).tick(anyInt());
	}
	@Test
	public void shouldNotifyTickEvery100ms() throws Exception {
	RouletteWheel rouletteWheel = mock(RouletteWheel.class);
	TickProvider tickProvider = new TickProvider(rouletteWheel);
	tickProvider.start();
	verify(rouletteWheel, timeout(500).atLeast(5)).tick(anyInt());
	}
	@Test
	public void shouldIncrementEachTickByApprox100() throws Exception {
	RouletteWheel rouletteWheel = mock(RouletteWheel.class);
	TickProvider tickProvider = new TickProvider(rouletteWheel);
	InOrder inOrder = inOrder(rouletteWheel,
	rouletteWheel,
	rouletteWheel);
	tickProvider.start();
	Thread.sleep(200);
	inOrder.verify(rouletteWheel)
	.tick(and(gt((long)0), lt((long)20)));
	inOrder.verify(rouletteWheel)
	.tick(and(gt((long)90), lt((long)120)));
	inOrder.verify(rouletteWheel)
	.tick(and(gt((long)190), lt((long)220)));
	}
	}

view raw TickProviderTest.java hosted with ❤ by GitHub

Finally we got to the point where we needed to wire up an assembly of the RouletteWheel and the TickProvider. I did this by testing in TTDAIYMI (TDD as if you meant it) style writing the code directly in the test before extracting to a static method on RouletteWheel

	@Test
	public void shouldBuildGame() {
	WheelObserver wheelObserver = mock(WheelObserver.class);
	RouletteWheel rouletteWheel = new RouletteWheel(wheelObserver);
	TickProvider tickProvider = new TickProvider(rouletteWheel);
	rouletteWheel.spin(1000);
	verify(wheelObserver, timeout(3000).times(1)).stopped(anyInt());
	}

view raw shouldBuildGame.java hosted with ❤ by GitHub

The TDD approach has led my final code I think to show good separation of concerns between the generator of tick events (TickProvider), the consumer of those events (RouletteWheel) and the consumer of the stopped event that gives the final location of the ball (WheelObserver). This last one I particularly like as it totally separates presentation from the model, its also easy to envisage how the code could be adapted to publish intermediate events whilst the wheel was “spinning”.

	import java.util.Random;
	public class RouletteWheel {
	private boolean spinning = false;
	private final WheelObserver wheelObserver;
	private long spinForMs;
	private long currentMs = 0;
	private final Random random = new Random();
	public static void spin(final WheelObserver wheelObserver, final int spinDuration) {
	RouletteWheel rouletteWheel = new RouletteWheel(wheelObserver);
	TickProvider tickProvider = new TickProvider(rouletteWheel);
	tickProvider.start();
	rouletteWheel.spin(spinDuration);
	}
	RouletteWheel(final WheelObserver wheelObserver) {
	this.wheelObserver = wheelObserver;
	}
	void spin(final long spinForMs) {
	this.spinning = true;
	this.spinForMs = spinForMs;
	}
	void tick(final long timeMs) {
	currentMs = timeMs;
	if (spinning && currentMs >= spinForMs)
	{
	spinning = false;
	final int location = random.nextInt(37);
	this.wheelObserver.stopped(location);
	}
	}
	}

view raw RouletteWheel.java hosted with ❤ by GitHub

	import java.util.Timer;
	import java.util.TimerTask;
	class TickProvider {
	private final TimerTask tickTask;
	TickProvider(final RouletteWheel rouletteWheel)
	{
	this.tickTask = new TimerTask() {
	private long startMs = System.currentTimeMillis();
	public void run() {
	rouletteWheel.tick(System.currentTimeMillis() - startMs);
	}
	};
	}
	void start() {
	Timer timer = new Timer();
	timer.schedule(tickTask,0, 100);
	}
	}

view raw TickProvider.java hosted with ❤ by GitHub

	public interface WheelObserver {
	public void stopped(final int ballLocation);
	}

view raw WheelObserver.java hosted with ❤ by GitHub

No doubt someone will point out some horrendous threading (or other) bug, please note this is a Kata or practice and I didn’t pair on it. But if you do spot one can I suggest you respond by proposing a test that would expose it? Remember TDD is not about passing when software works, its about improving your design and as a bonus you get some tests that should fail when it doesn’t behave as you intended. I hope I showed that here.

For the full gist including test classes see below;

gist https://gist.github.com/daveneedstoknow/4cbb6a7d707d8efe4cb5fff1ee748923

You can also see my working on the excellent cyber-dojo here I hope no-one will make further changes at the time of writing my work goes up to revison 106. Note that revisions after that may not be mine.

You’ll see (as is my typical experience of TDD and coding in general) that progress is not quite as linear as some explanations will have led you to believe, including here 😉

The total time I spent on this was around the 3 hour mark so higher end of allotted time (I took a break in the middle to teach my nephew some TDD 🙂

I enjoyed the Kata, thanks to @jjefries1 for suggesting it. I tackles some of the more challenging things in TDD but I think it also shows that if TDD is followed it does give you that nudge to a better final design.

Follow @thinkfoo

If you want the benefits of pair programming – Start by sorting the furniture out.

Follow @thinkfoo

Lets say you’ve read the various justifications for pair programming and you have decided to give it a trial or perhaps you’ve already tried it and not been successful. What things can you do to give pairing the best chance of success and make it a pleasant experience for all?

Lots has been said about the day to day practice of pairing but here I want to look at some of the more practical physical office considerations that can, in my view make or break a team.

Sort the furniture out

A poorly laid out office can destroy your chances of a successful pairing experience before you have even begun. If your desks are arranged like those on the left above you are not going to have a fun time. Not only this you could even be breaking regulations on providing a safe working environment for office workers. The pairing station on the right shows the set up at transFICC.com

Get rid of those L shaped desks and dividers if you have them Straight desks with no obstructions underneath provide a much more comfortable place to sit. Aligning PC ‘s lengthways at the back of the desk can free up leg room and don’t have under-desk pedestals – put them at the end of desk rows if you must have them.

This is the single most important piece of advice in this article if you do nothing else – do this.

Provide breakout areas

“lots of drawing on walls” Gavin Tapp CC BY 2.0  via http://www.flickr.com

Its not all typing. Its really important not to spend all the time coding at the workstation, we often need to go and sketch out ideas and discuss them away from the screens. Pairs should be encouraged to and given somewhere to breakout and have design discussions. Seating areas with tables and whiteboards everywhere should be the norm for any agile team. Painting the walls with whiteboard paint is really effective.

Provide standardised machine configuration

“My laptop stickers came” Doug Belshaw CC BY 2.0 via fickr.com

How someone has their machine set up can be a very personal thing, keyboard short-cuts, themes etc. This can make it very difficult to enable a sense of shared ownership and collaboration. So don’t just pair by bringing along one of the developers laptop’s. Get  pairing stations set up with two keyboards, two mice and two screens. Configure these in a standard way ideally with automation such as puppet or chef.  This enables you to clean install the station every night and will ensure that any developer can work on any workstation and will immediately be in a familiar environment. It also encourages the frequent small check-ins approach used in XP & Continuous Delivery.  You should be checking in your IDE configuration with your source code to ensure “Everything is in Source Control”.

I recommend the team agrees on a single IDE (or text editor if you must, if you are still having vim/emacs wars maybe this is an opportunity to move on a decade) for pairing.

Tip have a USB hub on the desk with spare slots – this allows people to plug in personal mice/keyboards if they have particular styles they like or want to for hygiene reasons, it also provides somewhere to charge your phone 🙂

This is an opportunity by the way to get a much higher specification machine for development – you need only enough machines for one between two.

Hot desking

“Pairing Station” by Andrew CC BY SA 2.0 via http://www.flickr.com

Pairing with the same person for day after day not only gets monotonous it also misses out on the knowledge sharing and team building that pairing brings. You need to be rotating pairs at least daily, for this reason I don’t recommend trial pairing with just a single pair you are unlikely to see any real benefits. If you are doing this, hot desking is really a pre-requisite as people are, by definition, moving around every day. But you can think about a couple of things to make this run better.

Make sure every desk is equipped with a fully set up pairing workstation as mentioned above with dual keyboards and screens.

Have a bank of desks dedicated to each team but encourage full hot desking within that area.

Many people want to keep using the same chair so just get some labels and stick them on its a small price to pay for comfort.

Don’t forget to have white boards and the team board near by.

Clean desk policy

 CC 0 Public Domain via Pixabay.com

Clean desk policy can be controversial but it ensures that desks are truly hot.

Have someone responsible every evening (Cleaner or office admin perhaps) for clearing all the papers and books into a document tray

Also have them replenish every desk with a supply of post-it notes, sharpies, pens, note paper and whiteboard markers. When we started doing this almost no-one complained any more about the clean desk policy, it seems people value having tools to hand over an untidy desk.

Final thoughts

Although this article is written with pairing in mind, even if you are not pairing regularly many of these tips still apply and will in themselves encourage greater team collaboration. In particular sorting out the furniture, providing breakout areas standard machine setups with dual keyboards and screens will allow ad hoc pairing to occur on an informal basis. The “Could you just help me with this a minute” situation.

So before you embark on your pairing trial I would strongly recommend these measures, I consider the furniture and standardised workstations pretty much essential.

• Sort the furniture out
• Provide Breakout areas
• Provide standardised machine configuration
• Hot desk
• Clean desk

 

Further reading

Pair Programming – The most Extreme XP Practice? – Dave Farley

Why pair programming is as much about business continuity as it is about code quality – Dave Hounslow

Follow @thinkfoo

 

 

 

 

How Branching Affects Team Culture – Talk at Pipeline Conf 2016

Follow @thinkfoo

At the Pipeline Continuous Delivery Conference 2016 I was given the opportunity to talk about our experiences introducing continuous delivery at http://www.your-loop.com

“Great story of agile & XP”

“Great arguments, nicely supported with actual examples from practice”

“Great examples and very engaging”

Covering themes I explore elsewhere on this blog including Pairing and Automated Testing, This was my first conference talk and I am delighted to have received overwhelmingly positive (91%) feed back, thanks everyone.

And thanks to all at the pipeline team for organising a great event!

Slides from the talk are available here.

Dave Hounslow – You Are What You Eat – How Branching Affects Team Culture – PIPELINE Conference 2016

People said some nice things about my talk. Image -wordclouds.com

Follow @thinkfoo

5 ways to avoid the Technical Debt Trap

Follow @thinkfoo

Technical debt is toxic debt, much like a payday loan you take on technical debt often in a state of panic when that feature just has to be out there right now. Further just like a payday loan if not paid off quickly it can rapidly become a millstone around your neck where you find yourself forever in a paralysing technical debt trap.

So what can be done to avoid the pain of Technical Debt in the first place?

#1  Acceptance Test Driven Development

Code that has no tests cannot be refactored safely, code that cannot be refactored is code that will rot, the second you write code without a regression test you have taken on some debt.

For me ATDD is even more important than TDD in this respect, once you have worked on a system with close to 100% acceptance test coverage you will wonder how you ever made changes safely without it. I have had to go back to working on systems that lack good acceptance test coverage having worked on some with close to 100% coverage and it is paralysing. More than this it is shocking to see the different attitude of developers to refactoring in such environments, without full regression suites (and if you don’t use ATDD you probably haven’t got a full regression suite) the attitude becomes “If it ain’t broke don’t fix it” rather than constant improvement. By using ATDD and treating the test as the source of truth you enable the freedom to change how code does stuff without changing what it does.

 

#2 Don’t feature branch – work on trunk

Feature branches discourage refactoring. I hadn’t really appreciated this benefit of trunk based development until recently when a colleague of mine rasied it. He had previously always worked with teams who used feature branching, when he moved to our team which was using trunk based development he was blown away by the freedom it gave him.  He enthusiastically declared

“It’s fantastic I can do fundamental refactoring’s knowing I won’t have people breathing down my neck at some point in the future because they have merge hell”.

Sound familiar? With trunk based development the worse case scenario is maybe ½ a days worth of code to merge in.

I find it ironic that feature branching is often championed as allowing developers to work independently on features when the truth is the opposite.
 

#3 Test Driven Development

TDD encourages good design by nudging us in the direction of loose coupling, strong cohesion and clean encapsulation. It doesn’t of course guarantee any of these things by my experience is that coders who have and do practice TDD tend to write cleaner more maintainable code. Code that is well written is easier change our efforts upfront reduce the debt to the future.

 

#4 Refactor all the time

Good citizen’s pick litter up and don’t add to it carelessly so always leave code a little better than you found it. Personally I find a little bit of refactoring is a great way to understand how the code does what it does. When faced with a method or a class that does more than one thing if we start breaking it up into smaller methods teasing apart layers and introducing abstractions its purpose becomes clearer. This has benefits both for us and for future pairs who work on the code.

 

#5 Limit your work in progress

Control your work in progress, working at a sustainable pace is key to maintaining velocity. If you are constantly firefighting technical debt will keep piling up. Techniques such as swarming can help you here.

Finally

Remember that ultimately it usually costs a lot more to pay back than what was saved by doing it well in the first place. Left to fester you end up taking on another piece of debt to cover the debt you already took out, then another and another until the debt becomes crippling. It’s much better to save up and invest in quality upfront than borrow from your future productivity.

Follow @thinkfoo

Is this a feature request or a bug?

Follow @thinkfoo

How does a bug differ from a feature request?

I’m sure like me you have been in many a heated discussion on whether something is a bug or a feature. In my view a great deal of time is wasted in these arguments that could be spent more productively.  A long time ago I concluded that, at best, it wasn’t a valuable distinction, at worst that to distinguish them is in fact damaging.

Consider the following possible definitions of bugs and stories

---

Feature Request

A proposed change to how your software currently behaves in the hands of users to deliver greater business value.

---

Bug

A proposed change to how your software currently behaves in the hands of users to deliver greater business value, which you want to blame the development team for doing wrong in the first place.

---

Lets think about two important points in these statements

• “Currently behaves In the hands of users”
• “which you want to blame the development team for”
  

“Currently behaves In the hands of users”

If no one can see this behaviour its not done yet. So during story development we should simply adjust and fix things as we find them – we want to get to done not half done, if you need to raise and track bugs at this stage you probably have one or more of the following problems or something else that needs to be addressed as the root cause

• Stories too big – taking too long – think about how you can break down the delivery of value
• Working in silos, with lack of integration of QA/BA/Developers/Product Owner’s
• No definition of done – “In the hands of users” is a pretty good place to start in my book.

“Which you want to blame the development team for”

This really seems to me to be the nub of the issue. When we call something a bug there is an implicit message that the development team did a bad job. My experience is that development teams generally set themselves very high standards on quality that this attitude can damage morale and ultimately lead to lower quality and rate of delivery. Calling things bugs can lead to wrong expectations such as;

• the cost of fixing it will not impact other work – devs did bad they can make up the time
• its automatically the most important thing to do which it may or may not be

The “No Bugs” messaging that has been around can also lead to prioritisation of bugs over stories when the bug fix may have minor business value compared to other changes in play.

Given these two definitions is it a useful distinction?

It is my contention that distinguishing bugs from stories adds no value. I have had good success with treating bugs and stories the same once the software is in the hands of users. They both have some business value associated with them and they both need to be prioritised based on that value. So having a single backlog makes much more sense to me. Moreover separating these out into separate tracking systems or even labelling them bugs or stories in the same system is a bad idea stop doing it now!

But people like raising bugs I’ll never be able to get rid of <insert favourite bug tracking software here>?

Depending on your company culture you will find it easier either to call everything a feature or everything a bug. I don’t think it actually matters much, although it can be fun to start a new project with a bug along the lines of “Unable to see login page” before a single line of code has been written. Whichever way you go my recommendation is to pick one and treat it as a single backlog.

A word of caution, if you call everything a bug remember ONLY the product owner should control the priority of the backlog that means sysadmins, customer service etc don’t get to set the priority – sure they get a say but the product owner has to balance all the business priorities and make the call.

Follow @thinkfoo

How to succeed at agile – remember the safety net.

Follow @thinkfoo

I have been involved in very many successful agile projects and have got involved at different stages in their evolution. In some of my earlier projects at companies such as Parallax I was pioneering agile techniques alongside people such as Dave Farley, who went on to co-author the seminal book Continuous Delivery. Later with organisations such as Commerce Decisions Ltd  I continued to work on successful projects with people who were developing the ideas around automated deployment and test driven development.

What characterised these early forays was the feeling that we were doing something new and bold we had all worked in more traditional waterfall processes before and had found them lacking, believing there must be a better way, with the enthusiasm and courage of pioneers we drove forward into the brave new world which others made were also looking at and gave us the Agile manifesto.

More recently I worked with Dave again on his flagship project at LMAX this was a project that he, Martin Thompson and others where they were able to pose the question “What if we did all this agile stuff right from the start, where would we end up?” I’m still in touch with many of the excellent people there and I believe the evidence speaks for itself, LMAX were #1 in the Sunday Times Tech Track 100 in 2014 and I am sure that the excellent tech such as the disruptor that they developed and expertise using continuous delivery techniques have been a huge contributor to their success. Beyond this unlike many companies, they continue to get faster and better at delivering software all the time as demonstrated by being named best overall testing project in testa 2013.

So this agile stuff works and we know it works, to me it seems that software process is now largely a “solved problem” so why is there still resistance to uptake, clinging to upfront design, planning and other ideas that have been shown to fail again and again?

Walking the tightrope

I think that one of the key barriers to adopting agile is fear, different individuals carry different understandable levels of fear with them. I find that the young and bold often lack the knowledge and experience of the pain when software projects go bad. These people quick to embrace the challenge and excitement of getting to production code quickly, whilst older more conservative types show more caution, fearful of letting go of the supports of careful upfront design and planning. We should not be surprised at this; for them letting go of these practices is like suggesting they cross a raging river by slinging a line across and walking the tight rope, with no safety net and no practice.

(By George Barker – from stereographic image, Public Domain, https://commons.wikimedia.org/w/index.php?curid=9032429)

This is understandably a terrifying idea to people who have been more accustomed to carefully designing a bridge whilst doing their best to predict every future risk and possibility with great attention to detail.

So should they just toughen up?

Of course not, the truth is they are absolutely right to be cautious! Quickly slinging a line across a ravine and leaping onto it with no kind of safety net or training is not bold, it is reckless. I have seen many people being burnt by doing exactly this, among my peers we have often referred to this as having been “Scrummed”. This is because of how scrum, and by association agile, is often implemented with an over emphasis placed on process and lack of emphasis on the development techniques that eXtreme Programming and Continuous Delivery demand. Moreover there is a perception that agile is all about process when the reality is that agile  is a collection of techniques that enable a different approach to software design and development. It is precisely these techniques that provide the essential safety net that makes the walk on the agile tight-rope both exciting and rewarding, but ultimately safe. Practices that include;

– The build pipeline – “Continuous integration on steroids”

– Acceptance Test Driven Development(ATDD) – to ensure a full regression suite

– Automated Deployment – to ensure we can actually deploy and crucially upgrade the thing,

– Performance testing – so we know it meets memory and processing requirements

– Merciless re-factoring – to ensure we change the software to reflect our evolving understanding of the world rather than ram our requirements into a model that no longer works.

– Test Driven Development (TDD) – to help us build software that has high cohesion, low coupling, small methods and classes all those good design principles that enable easy re-factoring (Yes that’s right TDD is not really about testing at al you heard here first)

– You aren’t going to need it (YAGNI) – so we don’t loose time maintaining the inventory of code that nobody is using – Keeping the build always releasable – the discipline to ensure we are always integrated and always green forces us to consider quality on every commit. To name but a few.

Be careful out there kids!

Simply dropping upfront planning and design without first gaining and applying these essential skills is, I am convinced, the reason many peoples first experience with “agile” ends with them crashing onto the rocks below. I think we need to be careful how we introduce people to agile, we need to emphasise the safety aspects, spend time building in testing (especially ATDD) and learning how to do good design with TDD.

Beyond this we should also remember that we are now trying to bring agile to the next wave of settlers, not the pioneers and that they, perhaps, are not quite so bold as some early adopters. Moreover simply putting the safety net in place does not in itself enable everyone to show courage, as Martin Fowler discussed, its a balance we need to let people gain height gradually and learn to trust the new practices that protect them and enable them to build better software better and faster.

After all if you have never walked on a tightrope, no matter how good the safety net looked, I suspect most of us would be apprehensive (to say the least) about striking out across Niagra Falls on a slack line.

Simple alternatives to feature branching – #1 Build alongside

Follow @thinkfoo

With Continuous Delivery (CD) every commit is a potential release candidate and all pushes are to a single central trunk. For people new to CD it an be very hard to reconcile frequent commits with this concept; How can I commit twice a day when this feature is going to take several days to be in a state that I want it to be public?

A good example we had of this recently was a full reworking the registration pages for the https://www.your-loop.com energy monitoring website.The registration process requires the installation of hardware to do meter readings, ensuring that radio communications are working and serial numbers etc. have to be entered and the flow can vary depending on the type of meter a user has. The rework involved changes of page flow and content. The nature of the changes meant we didn’t see a clear way to break it down into smaller sizes that made sense to the user. It was also clear this is not something that could be completed in a single commit.

Why commit to trunk?

The key things we are trying to ensure by committing frequently to trunk are;

– New automated acceptance tests are developed as we go and run on the new feature in every CI build .

– Existing acceptance tests continue to run on the existing code in every CI build.

– Ease of manual testing so the changes can be viewed (without redeployment to switch for example) .

– Continuous integration and merging into all developers work .

Feature branching – just say no.

Its common for people to revert to what they know when presented with a challenge and some of the team were immediately reverting to suggesting a feature branch. The problem with this of course is that feature branches make all of the above desired objectives difficult, expensive or simply impossible. Every time you feature branch, a release fairy dies,  just say no!

Feature Toggles were not ideal.

One option discussed was to use a feature toggle. The problem with toggles is how to toggle them on and off. Feature toggles often take the form of a singleton property, perhaps in a configuration file or dependency injection that switches between different implementations at build time. The problem with this approach is test isolation, I want all my regression tests to run whether the feature is currently on or off, and this is an awkward thing to do in your CI environment unless you run tests sequentially, even then it requires duplicated toggles in test code that you must keep in synch.

There are other approaches to feature toggling that can happen at runtime which I will talk about in another post but this wouldn’t have been much help here as we needed a global new/old switch.

Build alongside

The approach we used is to “build alongside” by building the new registration pages with a different URL which is not linked to from the site;

So if the existing registration page entry point is;

http://www.example.com/registraion

We build

http://www.example.com/newregistration

Whilst the new feature is being developed all buttons/links etc. point to the existing registration pages entry point. We can write selenium tests that drive the new registration pages and leave all of the existing tests in place. QA and the product owner can access the new pages without any configuration (by simply going direct to the URL), but they are sufficiently hidden from the public to not get seen. In practice we called it a slightly more obscure name and there was really no risk beyond seeing a “work in progress” should a customer have guessed the URL which was unlikely.

Once all are happy with the new pages the new URL is renamed to the existing one and the old code is deleted.

It’s a simple approach that everyone understood and met all of the desired objectives regarding frequent commits, CI, manual testing and frequent merging with everyone’s code.

The thinkfoo test for Continuous Delivery – What does good look like?

Follow @thinkfoo

Dave Farley makes an excellent point in his recent post that many people in our industry have never worked on an efficient project team and wouldn’t know good if they saw it. Indeed many are probably blissfully unaware of just how inefficient they are comforted in the knowledge that they seem to be no worse than anyone else. So how do you know if you are good?

Some fourteen years ago Joel Spolsky wrote an often quoted piece he called the Joel Test.

This was great stuff back in 2000 but we have moved on a long way since then and I propose an updated test that reflects modern development practices founded in the principles of Continuous Delivery (CD).

I’ve worked with some incredibly productive teams (and some not so productive ones) I prefer to work with productive ones and I have over the years formed some strong opinions of exactly what good looks like. I realised I have been informally using a set of criteria to evaluate teams for some time and thought I’d write it down and share it.

So here is my “thinkfoo test”; efficient development teams will be able to answer yes to most if not all of these questions, if your team can’t and you think you have an efficient team with a good process maybe its time to take the red pill and think again?

1. Do your developers push changes to a single trunk branch at least twice a day?

2. Do you practice ATDD and write automated acceptance tests for every feature?

3. Do you practice TDD?

4. Do you run your full set of acceptance tests, unit tests and performance tests on every commit to trunk?

5. Do you have automated single step repeatable builds and deployments?

6. Does your team release to production at least every two weeks?

7. Do you practice pairing as the normal way of working?

8. Do your teams self organise?

9. Is your team in a noisy collaborative environment?

10. Do you have cross functional teams of BA’s, developers, QA and operations responsible for end to end delivery of working software?

11. Do you have a single product backlog (with bug’s and features together)?

12. Do you use minimal stories as specifications rather than detailed up front designs?

13. Do you have regular retrospectives to continually improve your process?

Lets look at each of these in a little more detail

1. Do your developers push changes to a single trunk branch at least twice a day?

• feature branching is evil, long lived feature branches doubly so. Trunk development is, for good reason, a key requirement for CD.

• trunk should be in an always releasable state, every commit should be a release candidate.

• To be continuous your integration must be frequent; committing to mainline twice a day should be considered a minimum.

2. Do you practice ATDD and write automated acceptance tests for every feature?

• these should be built in advance and alongside the feature and treated as the source of truth or living specification. The feature isn’t done until all the acceptance tests pass. By doing this and keeping work in progress to a minimum the need for bug tracking is diminished.

3. Do you practice TDD?

• many people are confused about what level to set their tests at that’s why I specifically want to know if you have both ATDD (are you building the right thing) and TDD (are you building it right).
• The “red, green, refactor” cycle is critical to producing good clean code

• Your whole team needs to practice TDD, I wince every time a development lead says “We don’t enforce it but sure you can work that way if you want”.

4. Do you run your full set of acceptance tests, unit tests and performance tests on every commit to trunk?

• #1 is a prerequisite of this if you are not using a single trunk your continuous integration isn’t, end of.

• performance tests are a critical part of your build pipeline many products fall under the pressure of their own success.

5. Do you have automated single step repeatable builds and deployments?

• if you can’t do repeatable automated deployments there’s not much point in doing #1 through #4 as you won’t know that what you deployed is what you tested.

6. Does your team release to production at least every two weeks?

• if you do iteration after iteration without release I don’t really buy that you are agile, this is usually a waterfall project dressed in agile clothing.

7. Do you practice pairing as the normal way of working?

• see my post on pairing for my views on why this is essential to building quality software assets.

8. Do your teams self organise?

• teams that do top down allocation of tasks in my experience always end up optimising resource usage over feature delivery, it discourages collaboration, common code ownership and pair rotation don’t do it!

9. Is your team in a noisy collaborative environment?

• the opposite of Joel here, no single developer can be productive enough to jeopardise whole team productivity don’t allow it and don’t hire people who insist on working this way. Great software is built by teams not individuals beware the hero developer!

10. Do you have cross functional teams of BA’s, developers, QA and operations responsible for end to end delivery of working software?

• and the roles should be blurred rather than designated, BA’s should write some tests, QA’s should write some code, developers should write deployment scripts and talk to customers, ops guys should be coding etc etc.

• Your developers should be on first line support for your production systems.

11. Do you have a single product backlog (with bug’s and features together)?

• do the most important thing first and optimise for flow of delivery over resource utilisation (see managing your backlog for more on this)

12. Do you use minimal stories as specifications rather than detailed up front designs?

• stories are about enabling conversation not detailed specification see https://leanpub.com/50quickideas for some great thoughts on user stories.
• Joel believes in detailed up front specifications sorry but he is just wrong on this, in any project lasting more than half an hour the specification will change, living specifications in the form of automated acceptance tests are the only way to keep specs current.

13. Do you have regular retrospectives to continually improve your process?

• nothing is perfect; core to the principles of Continuous Delivery is continuous improvement retrospectives are an essential part of that learning and improving.

TDD teaches you how to code well

Follow @thinkfoo

You may have seen the recent   Kent Beck, David Heinemeier Hansson, Martin Fowler discussion on Is TDD Dead? where David Heinemeier Hansson contests that TDD is “poisoning” good designs. My take on this can be summarised with the below diagram;

Not all Test Driven Code is good and not all Good Code is test driven. But a lot of code (probably most) is neither good or test driven. I contest that to be considered good, code must be both well designed and well tested. Now I could have labelled the above “tested” or even “testable” but I have seen very little code that is written without TDD which gets anywhere near approaching “well tested” or even “testable” no matter how well intentioned the developers. Moreover I believe that by practising TDD you have much better chance of getting to good than by not doing so.

I say this because I have found that people who consistently write good clean code have almost invariably been people who have mastered TDD. Why could this be?

I think the reasons for this is are that in applying TDD you learn that keeping your tests manageable is at least as hard as writing the underlying code. To avoid brittle difficult to understand tests you want tests that test only one thing and you have to balance this with code that is itself manageable and does one thing well. Consequently;

TDD leads you to code that has small interfaces and low cyclomatic complexity.

TDD tells you things about your code; Tests that require a lot of dependant objects to be constructed or mocked are telling you your code has high coupling. Tests that test more than one thing can hint at poor cohesion – classes under test are doing too many things.

Learning about techniques like mocking and stubbing teach you about coding to interfaces rather than concretions.

TDD teaches you about best ways to handle error conditions and exceptions

TDD teaches you about separation of concerns, why for example its beneficial to separate threading from business logic.

As you get practice writing tests you learn that testing multiple layers of abstraction at the same time is hard and intuitively start to understand what layers of abstraction really mean.

TDD also teaches you to understand what it is to be a user of your code and code for the users rather than the implementers benefit.

In becoming a master practitioner of TDD you provide yourself with all these learning opportunities. Some people having done this feel that TDD can occasionally be overkill (Dan North talks about Spike and Stabilise and opportunity cost of TDD here) but it is the very expertise they have gained from TDD that gives them the ability to make this judgement and to proceed in a way that is safe to retrofit tests to. It’s worth pointing out though that most people I know that have have become competent at TDD never really want to code without it (myself included).

So finally if you are wondering if your code is good or bad and haven’t yet mastered TDD I would suggest it probably has considerable room for improvement.  Check out Steve Freeman and Nat Pryce’s Growing Object-Oriented Software Guided by Tests and Kent Beck’s Test Driven Development.