Contents:
Tempted by a Demon
At XP Agile Universe this week, one of the keynote speakers was Watts Humphrey. In his talk he spoke of all the data that can be made available during software development, how valuable it is, and how easy it is to get much of it. Even for a subjectivist like me, it’s hard to argue with data if it’s easy to collect it.
Watts made clear when he gave his talk that he didn’t know much about the agile methods, but by the end of the conference he did, because he did us all the courtesy of attending sessions and tutorials to learn what we’re all about. In keynoters we’ve seen at the XP conferences, this is rare, and rather impressive.
Ann, Chet and I had the privilege of sitting with Watts at the conference dinner. Watts has a very discernible data fetish, and he was asking us what we do and how we know that it works. After some discussion we agreed to do a little experiment in our work over the next few weeks. I’ll describe here what we learned and what we plan to do. I’ll ask Watts, if he has time, to comment on our plans and help us refine them. Any flaws in understanding are ours, not his.
Basic Premise
Although it’s only an approximation, it’s pretty safe to say that the number of errors remaining in a system, after all the stages you go through, are roughly proportional to the number of errors you put in at the beginning. Pairing removes x percent, the compiler y percent, unit tests z percent, and so on. For an accurate figure, you’d have to deal with the errors that the compiler finds that the tests would also find, and so on, but basically, if your program starts with 100 errors and leaves only two, you’d expect that in a program where you started with 200 errors, there would be about four errors remaining.
XPers, especially those of us who come from a dynamic language background, typically don’t inspect their code much, other than whatever is noticed during pairing. We compile, and if it compiles, we run the tests, and when the tests all run, we clean up the code and move on. In our conversation with Watts, we got a different idea. We’re presuming that most of the errors we type in are syntax errors, and that they’ll be caught by the compiler, or logic errors that will be caught by the tests. But of course, there are some errors that won’t be caught by those nets, and that will slip through.
Watts challenged us to scan the code before compiling, aiming for a clean compile every time. The idea isn’t to try to beat the compiler. Instead, the idea is that in scanning carefully enough to eliminate compiler-detected errors, we’ll also detect and correct other errors, and that some of those errors will be ones that the compiler and tests wouldn’t have found. The question, of course, is whether the errors found are worth the effort of the scan. We’re thinking that since we write code in such tiny chunks, the scan won’t be difficult and so the payoff might be worth the short delay.
So we decided to try it. The trick is to find an approach that lets us figure out whether it’s worth it, and we think that’ll be tricky. We’re sure to find some errors, but it’s not so clear whether they will be errors that wouldn’t have been caught any other way, or that would have slowed us down by being caught later. Plus, whatever we do has to be easy. We normally go very fast and [we think] very reliably, so that slowing down isn’t going to feel good.
Initial Approach
We’ll pause before each compile and scan our changes for errors, and deal with the ones we detect. (We’ll talk more about just how in a moment.) Only after the scan will we compile and test. If the compiler finds errors, we’ll make a note of what they were, to improve our ability to scan. We expect that over time, doing the scan will actually reduce the errors we make, so that errors found during scan may decrease. We’ll track that.
After tracking and fixing the compile errors, we’ll run the tests. We’ll make notes of what unexpected errors are found by the tests. (Our normal practice is to run the tests once, expecting our new test to fail, before writing the code to make it work.)
During inspection, we’ll track the number and type of the errors found. At first, we have no idea what we’ll find, so we’ll just take clear notes. Later on, we may have classes or something to check off. Syntax errors, we’ll just fix. Logic errors that we see, we’ll comment in the code or our notes, then compile and run the tests. The idea is to see whether the tests will find the problem. We’ll record whether they do or don’t. The interesting case, of course, is when they don’t.
If the tests find the error, we’ll fix it. If they do not, we’ll leave it in, with the comment on it. The idea is to find out whether a later phase of testing finds the problem, or whether a future unit test will find it. Now, having seen the defect, our testing is surely contaminated by knowing what the bug is, but we’ll do our best to be clear on whether the future test that finds the bug is a likely one, or one that possibly comes from our having seen the code defect. It’s a bit subjective, but we’ll just do our best. The readers can judge whether we’ve been fair.
(An alternate approach would be to fix the problem right away and make a note of it. That might be a better way to improve on inspection. Yet another way would be to write a test right away. That might be a good way to improve on testing. For now, we’re interested in how well our current process works, so we’ll try to contaminate it as little as possible.)
After each session, we’ll record our notes as part of the article that describes that session. Over time, I expect that we’ll come up with some cute tabular way to record the info, and maybe even some Visual Studio tools to help us along. At the beginning, we’re going to try hard not to go all techie, and instead just use pen and paper to record what happens. That seems like the simplest thing that could possibly work.
Conclusions
Well, of course there aren’t any conclusions yet! What’s wrong with you, imagining that there would be? But we’ll pass this on to Watts for his comments, and we welcome questions and comments from any reader: our thoughts are certainly not clear on this as yet. I’m pretty sure it’s not all due to the excellent Amber Waves beer we had during lunch today at Grizzly Peak, where my son is brewmaster.
Watch these spaces …