Quality Assurance

A very experienced QA manager recently emailed me some tough questions. Here they are, with are my answers. As always, comments and further questions are welcome!

How is Software Quality Assurance and Software Configuration Management integrated into Extreme Programming?

XP defines two levels of testing. The first is unit testing, which must be performed by the programmers as they work. Each class implemented must have programmer-developed unit tests, for everything that “could possibly break”. These tests are to be written during coding of the class, preferably right before implementing a given feature. Tests are run as frequently as possible during development, and all unit tests in the entire system must be running at 100% before any developer releases his code. (By release, we mean transferring from his own code space to the code integration area. This is handled differently, of course, depending on the code management tools in place.)

The second level of testing is called functional testing. Each feature of the system (which is defined by something we call a User Story, rather like a Use Case) must have one or more functional tests that test it. The functional tests are the responsibility of what we call the “customer”, the body responsible for defining the requirements.

The implementation and running of functional tests can be done by the Software QA group, and in fact this is an ideal way to do it.

Within XP, are there any specification baselines, test baselines, QA Acceptance testing, and CM Release Management/Change Control?

I’m not sure I know what you mean by these terms. Let me take a shot at how XP works in these areas: if I’m way off base, please ask again.

XP is an inherently incremental process, with software being released to “production” as frequently as possible. This generally means that programmers release their work to the common development pool approximately daily, and that means that if a full system were built on that day, their code would be included in that build. The time period between full system builds varies depending on the environment: since you have chosen a particularly difficult integration language (C++), I could imagine that you would build less frequently. We would recommend, however, that the full system be integrated as often as possible, at least daily. (This may seem aggressive to you. We’d have to talk about what is possible in your environment.)

Since XP is incremental, developers are working in short time increments we call iterations: we recommend about three weeks. Features (user stories) are broken down to the point of detail that allows a developer and his partner to implement the stories they’re working on in that time period. We like the functional tests for that iteration to be complete and available no more than half-way through the iteration. (This usually means that QA is writing tests for the next iteration while this one is going on.)

All through the iteration, programmers can use QA’s functional tests to determine whether they have met the requirements. (They are also using their own unit tests to determine whether their individual classes are doing what they should. This is usually at a much finer level of detail.)

Baselines work this way: when the code for a story is released, all the functional tests for it should be in place, and will ideally be working. Inevitably some will not, especially with teams just beginning with XP. One of the quality measures in the process is the daily graph of performance on functional tests. The general shape of this graph, over the course of the full system release period, is that of two s-curves: the upper curve is the total number of tests written, the lower curve is the number running at 100%. A healthy project of course shows these curves coming together at 100% by the end of the schedule.

The code management software needs of course to reflect the requirements scheduled for release. This is determined by the “customers”, as part of the planning components we call the commitment schedule (overall plan for a major release) and the iteration plan (plan for a (three week) iteration). The baseline of what is in the system tracks what is actually requested by the customers. Development doesn’t care whether this is new functionality or a change to old. They don’t care whether a given user story addresses something that was planned for or not. XP is completely flexible with regard to change management: development merely estimates how long any desired feature will take, and works on it when “customer” schedules it into an iteration. (Dependencies of course exist, but we find that far fewer exist than most developers believe. Drilling into that subject is beyond the scope of this email.)

When do the all the customer sign-offs occur?

Customer sign-off is continuous. Each iteration has its functional tests. Everyone is fully up to date on which tests are working and which are not. If tests scores are trailing implementation by too much, the customer will inevitably schedule more work against older features that are incorrect (or whose requirements have changed). When test scores are tracking implementation, the customer knows it and is comfortable requesting new functionality.

Because the test scores are public and visible, everyone has the same level of understanding of where quality is. Generally scores are showing a good curve toward release, and everyone gets increasing comfort as the release date shows up. And, of course, if tests are not tracking, everyone knows that and the priority of getting things right naturally increases.

The overall idea of this part of the process is to provide the most rapid feedback possible to everyone, customers and developers alike. That’s why we like all the functional test run every night. Next morning, if anything has been broken the day before, everyone knows it and can deal with it effectively (since it was only yesterday’s work that could be the problem). The faster the feedback, the faster development of quality software can proceed.

What are the Quality Assurance and Software Configuration Management roles and responsibilities with Extreme Programming?

We prefer for there to be a separate organization for functional testing (probably exactly like your QA function, with testing results made public very quickly). XP, however, only says that there must be functional tests: it does not specify organizationally how they must be done. Experience is that testing is best done by a separate function – but one that is very tightly integrated with development rather than at the end of a long pipeline.

Configuration management is also up to the team. It is usually necessary to have one or more individuals responsible for CM. We have no special rules or practices addressing how a group would manage the requirement to build multiple systems from one code base. Our main approach would be: for each release configuration, there must be corresponding functional tests, and these must be run before that configuration is released to the (real) customer. We would think that development would proceed by running kind of a “union” of all the functional tests of all the configurations.

We’d probably have to talk more specifically about how your specific organization needs to build configurations to say much more about that.

Do you use IEEE, SEI, ISO9000 standards as references to acquire the fundamentals of defining accurate requirements for customers and software engineering users? How can a person write storyboards without having the basics of pinpointing and developing sound requirements?

We would agree that those who play the customer role have to know what they want. We do not, however, recommend any particularly formal requirements writing or recording mechanism. Instead, what we are working toward (XP is for small teams, after all) is to have a clear understanding in the heads of customers, developers, and testers as to what is wanted.

Rather than have, say, an “analyst” sit down with the customer and laboriously translate his mumblings into something representing what is wanted, and then having a “designer” take the analysis and build a design, and so on, small teams function best if the customers and designer/developers talk to one another until they develop a common vocabulary of what is needed and how it will be done. In XP, we would like to have a common level of understanding in all heads, each focused on its own particular interests:

Customers: what’s needed, what’s the business value, when do we need it?
Developers: what’s needed, how can I build this, how can I test my code, how long will it take?
Testers: what’s needed by the customers, how can I test whether developers have done it?

As you can see, the testers’ functional tests are what close the loop, assuring everyone that what was asked for was what we got. The best way to do XP is with a separate functional testing organization that is closely integrated into the process. It would be delightful to have that organization run by an experienced QA manager trained in XP.

Is Extreme Programming not for Software Quality Engineering and Software Configuration Management practitioners.

XP is a development discipline that is for customers (in their role as specifiers and their role as investors and their role as testers and acceptors) and for developers. As such, the Quality Engineering and Configuration Management roles are critical to the effort. They have to be assigned and played in a way that is consistent with the mission of the group, the level of criticality of quality, and so on. We’d need to talk in detail about your situation to see just where the XP terminology connects with yours, but your QA functions need to be done in any effective software effort, whether in a separate organization or not. So XP certainly is for software quality engineering and software configuration management, as part of a healthy overall process.

That said, XP is aimed at smaller projects (like yours) and it sounds like yours has a much higher level of QE and CM than is often seen in companies of your size. That should give you a strong leg up in building quality software, and we should strengthen your contribution to quality as we bring XP into the team.

I’ve enjoyed trying to answer your questions and hope that we’ll get together soon to talk in more detail. As you have seen in your transition from mil-spec to industry, there is that need for tailoring and streamlining. What is quite possible is to bring the benefits of testing orientation into the realm of rapid development. XP believes, and shows, that more testing, done with rapid feedback, actually speeds development.

Thanks for asking!

Posted on:

Written by: Ron Jeffries

Categorization: Q and A

Recent Articles

Codea Calculator II

Ignatz and Jmv38 on the Codea forums commented on the previous article. I had hoped to do more anyway so here's the next one.

Codea Calculator

Based on a simple example on the codea.io forums, I decided to write an article showing all the stuff I might do on a production calculator project. Wow.