A big, big blog

Almost every project I have worked on over the last 6 years has incorporated a code coverage analysis tool into the build. Most of them have failed the build if the level of coverage has not reached a pre-defined standard and the others have at least reported on the coverage attained by the test suite. It’s nice to show a colourful chart showing a well-tested code base. Stake holders love that.

I recently joined a new project where the build was wired up with Cobertura which was happily reporting that the coverage levels were exceeding 90%. Everyone seemed happy. Nice charts, lots of green bars, warm fuzzy feeling.
In an idle moment, and out of the blue (I’d just built and checked in and was about to start something new), a few questions popped into my head:

Why did we have code coverage checks?
Why would these checks fail the build?
Why would we set the target to 90%?

The easy answers were:

To ensure that our code is tested.
To stop us from ignoring untested code.
It seems like a large enough number.

Hmm, those answers weren’t cutting it for me. I wasn’t feeling warm and fuzzy any more. Slightly upset, I asked myself:

How can I feel warm and fuzzy again?
What piece of testing information would make me happy?

Thankfully, the answer to those questions were also easy to come by:

I would be happy if the desired functions of the system were behaving as they should.
If the functional test suite was complete, correct, and showing 100% passes then we were delivering the right solution. Warm and fuzzy again. Phew.

But what about code coverage? Wasn’t that important? Did the code coverage report perform a useful task for us? Apparently, with regard to the functional tests, the answer was no. Functional tests drive the application; if the application behaves correctly code coverage is irrelevant.
Yes, that’s right, I was saying that I didn’t care if the functional tests exercised all of the code base as long as the system behaved as I expected in the situations I placed it.
So I removed the functional tests from consideration in the code coverage reports. That felt better. What about the integration tests? Same story. If the subsystem behaved the way I expected then the code coverage was irrelevant. I cared about behaviour not how much the code was exercised. The integration tests were also removed from consideration in the code coverage reports.

What about unit tests? Aah, that was it; the penny dropped. Unit tests drive the design (well they do if you test-drive design). One of the project development goals was to test-drive the system. Write a test. Watch it fail. Write some code. Watch the test pass. Refactor, rinse, repeat. We wanted to do this to help ensure that the system was well-constructed and easy to maintain. We care about unit tests exercising particular parts of the codebase. We would only write code to either satisfy a failing test or to refactor to improve design and eliminate duplication. 

The build was now changed to only instrument the codebase for the unit test run. No other testing would be part of the code coverage analysis. The build failed. We no longer had 90% coverage. Code had been exercised during integration or functional testing that had not been exercised during unit tests. That code was not test-driven. It had been exercised coincidentally.

Our build was now doing its job of reporting on a failure to adhere to one of the development goals. The penny had dropped. That’s why we had code coverage reports incorporated into the build. Code coverage is important for unit tests. It shows that unit tests are driving the development.

My answers to the original questions were now:

Why did we have code coverage checks?
To ensure we are only writing the code we need.
Why would these checks fail the build?
If we forget to test-drive.
Why would we set the target to 90%?
We won’t, we’ll set the target to 100% and discuss any exceptions to this rule if and when we encounter them.

Every now and again I get to use an application that just works. Sometimes I feel compelled to tell others about it.

We had a product development workshop this weekend and one of our aims was to ensure that our target customers would be happy with the experience when they eventually use our application. To reach that goal, we’d invited a selection of our target customers to help shape the direction of the product.

These customers are users of a system that isn’t quite cutting it on a number of levels; one of which is the capture and re-presentation of data in an appropriate context. It’s important for us to get it right so we got them to walk us through screen flows, describing the data to be captured and also how it should be represented within different contexts.

In preparation, Andy had stumped up 79USD for a copy of Balsamiq Mockups and within a few minutes had our customers happily designing screens and flows. The LoFi sketch look and feel helped to keep a sense of things being changeable and our customers were quite happy to move controls and data fields around until they felt comfortable.

They went away feeling that they had contributed to designing a usable product and we went away knowing we had enough information to make a start on development. Hats off to the people at Balsamiq for producing such a cool application.

Out of a spike, nothing but the newly acquired knowledge should be part of the final solution.

Both technology and technical creativity move along at a quick pace and people are always looking for better ways to solve their particular problems. The quest for improvement is a natural one. It is part of human nature. Innovation abounds and the abundance of new open source projects tackling similar problem spaces provide ample opportunity for the curious software developer to scratch that particular itch, be it as a contributor or a user.

Most software projects I’ve been involved with have included some new element of technology, or some new feature of a trusted technology. Whether this is a feeling of responsibility to always provide the optimal solution, or whether it is an eagerness to play with a new toy is debatable and there may be an element of both underpinning the quest for improvement. Fortunately, there has usually been the awareness of a responsibility associated with innovation. The introduction of the unknown should always be seen as a risk and should not be embarked upon without some means of mitigation.

If you try something new then you really need to have a safety net to take away the risk.

Typically this risk mitigation takes the form of a spike or technical task to prove the appropriateness of the technology to the solution. In the agile world a spike story is usually scheduled early on and is targeted to prove that the technology feature satisfies the project’s needs (as opposed to a time-boxed technology investigation where you schedule a fixed amount of time to explore a technology to see what features it offers).
In some cases, new technology adoption will involve the scheduling of a time-boxed technology investigation followed by a targeted spike. Firstly, find out what the technology can do then, secondly, prove that one of the identified features is suitable for the solution.

The use of spikes can be seen as reasonable and responsible risk-mitigation. And what’s more, it’s win-win. On one hand, proving the technology is a good thing - we made the right choice. On the other hand, disproving the technology is also a good thing - the risk mitigation was effective. What can go wrong?

Sadly, people occasionally forget that spikes are meant to be throw-away implementations. The only important artifact from a spike should be the knowledge acquired during the spike. Everything else should be thrown away. Ok, before anyone jumps on my case about that last sentence, it should read

“Out of a spike, nothing but the newly acquired knowledge should be part of the final solution“.

Feel free to keep the spike code in subversion (or whatever SCM you use), just don’t use that code for anything else. A spike is a spike is a spike.

I have seen, too many times, programmers become attached to the spike, not wanting to let go.

“If I just add a little more then it will solve our problem. Almost there…look it works!“

Sure, it may be functionally correct, but how much attention to maintenance/readability/abstraction was paid during this spike? When I first thought to write this entry I was tempted to suggest that you code your spike in as procedural code as you can; violate your checkstyle checks; remove all temptation to include it in your main codebase. I almost suggested that any extra effort to provide the normal things we aim to produce in our coding efforts (clean, readable code with appropriately chosen abstractions) are fluff in this context.
A little over the top? Probably.

The focus of the spike should have been to prove or disprove the technology, nothing more, nothing less.

However, I’m trying not to send the wrong message here. I’m not advocating that programmers should throw away good habits; in fact, I’d hope that not to be the case. I’m advocating programmers acknowledge that spikes are disposable and should be treated as such. The nature of a spike precludes a programmer’s finest coding efforts and, consequently, spiked code should never be considered to be part of the final solution.

Look for shortcuts to get you to the aim of the spike. If this forces a temporary relaxation of rigour then acknowledge the reasons for the relaxation and complete the spike. This approach cannot be taken, however, if the goal of the spike is uncertain. The intent of the spike has to be clear. You need to know when to stop. There needs to be constraints applied to the story, similar to acceptance criteria, that make it obvious that the story is complete.
e.g. Stop if the library is not thread-safe.

Another manifestation of the programmer becoming attached to a spike is solution-affinity. This occurs when a programmer tackles the spike using the same metaphor as the project.

e.g. the project is concerned with car finance and the spike aims to prove that a particular workflow engine is suitable. If the spike uses concepts from car finance to prove the workflow engine (i.e. exactly the same metaphor as the project) then it becomes difficult to relinquish the approach taken in the spike. Any efforts to implement the technology will mirror that of the discarded spike. Again, this is a problem because the intent of the spike is different to the intent of the project. Different rules apply.

To avoid solution-affinity, attack the spike with a different metaphor. Try to divorce the proving of the technology from the overarching domain. Spike the workflow engine using a different domain. Change the context to free the programmer from bias built up during the spike.

Spikes can be effective risk mitigation tools but try to remember:

• Never forget the purpose of the spike.
  
• Always know when to stop.
• Spike using a different metaphor.
• Keep your rigour but optimise your approach to keep to the intention of the spike.
  
• Throw away all but the acquired knowledge.