Test-first development with FitNesse

During the last few years, I have worked in all roles of the testing process, using server-side JavaScript, Perl, PHP, Struts, Swing, and model-driven architectures. All projects differed, but they all had some commonalities: the deadlines ran late, and the projects had difficulties accomplishing what the customer really needed.

Every project had some kind of requirement, some were very detailed, some, only a few pages in length. Those requirements usually underwent three phases:

• They were written (either by the customer or by the contractor) and received some kind of official acceptance
• The testers tried to work with the requirements and found them more or less inadequate
• The project entered a phase of acceptance testing, and the customer suddenly remembered all kinds of things the software needed to do additionally/differently

The last phase led to changes, which led to missed deadlines, which put stress on the developers, which in turn led to more mistakes. The bug count started to rise fast, and the overall quality of the system declined. Sound familiar?

Let’s consider what went wrong in the projects described above: The customer, developer, and tester did not work together; they passed the requirements on, but each role had different needs. In addition, the developers usually developed some kind of automated tests, and the testers tried to automate some tests as well. Usually, they couldn’t coordinate the testing sufficiently, and many items were tested twice, while others (usually the hard parts), not at all. And the customer saw no tests. This article describes a way to solve these problems by combining requirements with automated tests.

Enter FitNesse

FitNesse is a wiki with some additional functions for triggering JUnit tests. If these tests are combined with requirements, they serve as concrete examples, thereby making the requirements even more clear. Furthermore, the test data is logically organized. The most important thing about using FitNesse, however, is the idea behind it, meaning that the requirements turn out to be written (partly) as tests, making them testable and, therefore, their fulfillment verifiable.

Using FitNesse, the development process could look like this: The requirements engineer writes the requirements in FitNesse (instead of Word). He tries to get the customer involved as much as possible, but that usually cannot be achievable on a daily basis. The tester peeks at the document repeatedly and asks difficult questions from day one. Because the tester thinks differently, he does not think, “What will the software do?” but “What might go wrong? How can I break it?” The developer thinks more like the requirements engineer; he wants to know, “What does the software have to do?”

The tester starts writing his tests early, when the requirements are not even done yet. And he writes them into the requirements. The tests become part of not only the requirements, but also the review/acceptance process for the requirements, which has some important advantages:

• The customer gets to think about the tests too. Usually, she even gets involved in creating them (you might be surprised how much fun she can have with this).
• The specification becomes much more detailed and precise, as the tests are usually more precise than mere text.
• Thinking early about real scenarios, providing test data and calculating examples results in a much clearer view of the software—like a prototype, only with more functions.

Finally, the requirements are handed to the developer. He has an easier job now, as the specification is less likely to change and because of all the included examples. Let’s look at how this process makes a developer’s job easier.

Implementing test-first

Usually, the hardest part of starting test-first development is that nobody wants to spend so much time writing tests, only then to find a way to make them work. Using the process described above, the developer receives the functional tests as part of his contract. His tasks change from “Build the thing I want and you are done, until I examine your work and make changes” to “Make these tests work and you are done.” Now everybody has a better idea of what to do, when the work is to be completed, and where the project stands.

Not all those tests will be automated and not all will be unit tests. We usually divide tests into the following categories (details will follow):

• Data-driven tests that need to be implemented as unit tests. Calculations are the typical example.
• Keyword-driven tests that automate application usage. These are system tests and require the application to be running. Buttons are clicked, data is entered, and resulting pages/screens are checked to contain certain values. The test team usually implements these tests, but some developers benefit from them as well.
• Manual tests. These tests are either too expensive to automate and the possible errors not severe enough, or they are so fundamental (start page not displayed) that their breakage would be discovered at once.

When I first read about FitNesse in 2004, I laughed and said that it would never work. The idea of writing my tests into a wiki that turned them automatically into tests seemed too absurd. Turned out, I was wrong. FitNesse really is as simple as it appears to be.

This simplicity starts with the installation. Just download the full distribution of FitNesse and unzip it. In the following discussion, I assume you have unzipped the distribution to C:fitnesse.

Start FitNesse by running the run.bat (run.sh on Linux) script in C:fitnesse. By default, FitNesse runs a Web server on port 80, but you can specify a different port, say 81, by adding -p 81 to the first line in the batch file. That’s all there is to it; you can now access FitNesse at http://localhost:81.

In this article, I use the Java version of FitNesse on Windows. However, the examples can be used for other versions (Python, .Net) and platforms as well.

Some tests

FitNesse’s online documentation provides some simple examples (comparable to the infamous money example from JUnit) to get you started. They are fine for learning how to use FitNesse, but they are not sufficiently complicated to persuade some skeptics. Therefore, I will use a real-world example from one of my recent projects. I’ve significantly simplified the problem, and the code, not taken directly from the project, was written for illustrative purposes. Still, this example should be sufficiently complicated to demonstrate the power of FitNesse’s simplicity.

Let’s assume we’re working on a project that implements a complex enterprise Java-based software for a large insurance company. The product will handle the company’s whole business, including customer and contract management and payments. For our example, we will look at a tiny part of this application.

In Switzerland, parents are entitled to one child allowance per child. They only receive this allowance if certain circumstances are met, and the amount varies. The following is a simplified version of this requirement. We will start with “traditional” requirements and move them to FitNesse afterwards.

Several phases of child allowance exist. The claim starts on the first day of the month the child is born and ends on the last day of the month the child reaches the age border, finishes his or her apprenticeship, or dies.

On reaching the age of 12, the claim is raised to 190 CHF (Switzerland’s official currency symbol) starting on the first day of the month of birth.

Full-time and part-time employment of parents lead to different claims, as detailed in Figure 1.

The current rate of employment is calculated on the active work contracts. The contract needs to be valid, and if an end date is set, it needs to be situated in the “activate period.” Figure 2 shows how much money a parent is entitled to, depending on the age of the child.

The regulations governing these payments are adapted every two years.

On first reading, the specification might sound exact, and a developer should be able to easily implement it. But are we really sure about the boundary conditions? How would we test these requirements?

Scenarios can be a great help in finding exceptions and boundary conditions, as they provide a good way to get domain experts to talk about business.

Scenarios

For most projects, the requirements engineer hands the specification to the developer, who studies the requirements, asks some questions, and starts to design/code/test. Afterwards, the developer hands the software to the test team and, after some rework and fixes, passes it on to the customer (who will likely think of some exceptions requiring changes). Moving the text to FitNesse won’t change this process; however, adding examples, scenarios, and tests will.

Scenarios are especially helpful for getting the ball rolling during testing. Some examples follow. Answering the question of how much child allowance is to be paid to each will clarify a lot:

• Maria is a single parent. She has two sons (Bob, 2, and Peter, 15) and works part-time (20 hours per week) as a secretary.
• Maria loses her job. Later, she works 10 hours per week as a shop assistant and another 5 hours as a babysitter.
• Paul and Lara have a daughter (Lisa, 17) who is physically challenged and a son (Frank, 18) who is still in university.

Just talking through these scenarios should help the testing process. Executing them manually on the software will almost certainly find some loose ends. Think we can’t do that, since we don’t have a prototype yet? Why not?

Keyword-driven testing

Keyword-driven testing can be used to simulate a prototype. FitNesse allows us to define keyword-driven tests (see “Totally Data-Driven Automated Testing” for details). Even with no software to (automatically) execute the tests against, keyword-driven tests will help a lot.

Figure 3 shows what a keyword-driven test might look like. The first column represents keywords from FitNesse. The second column represents methods in a Java class (we write those, and they need to follow the restrictions put on method names in Java). The third column represents data entered into the method from the second column. The last row demonstrates what a failed test might look like (passed tests are green). As you can see, it is quite easy to find out what went wrong.

Such tests are easy and even fun to create. Testers without programming skills can create them, and the customer can read them (after a short introduction).

Defining tests this way, right next to the requirement, has some important advantages over the traditional definition of test cases, even without automation:

• The context is at hand. The test case itself can be written with the least possible amount of work and is still precise.
• If the requirement changes, there is a strong chance that the test will change as well (not very likely when several tools are used).
• The test can be executed at once to show what needs to be fixed to make this new/changed requirement work.

To automate the test, a thin layer of software is created, which is delegated to the real test code. These tests are especially useful for automating manual GUI tests. I developed a test framework based on HTTPUnit for automating the testing of Webpages.

Here is the code automatically executed by FitNesse:

 
package stephanwiesner.javaworld;
import fit.ColumnFixture;
public class ChildAllowanceFixture extends ColumnFixture
{
   public void personButton() {
      System.out.println("pressing person button");
   }
   public void securityNumber(int number) {
      System.out.println("entering securityNumber " + number);
   }
   public int childAllowance() {
      System.out.println("calculating child allowance");
      return 190;  
   }
  
   [...]
  
}

The output of the tests can be examined in FitNesse as well (see Figure 4), which greatly helps with debugging. In contrast to JUnit, where one is discouraged from writing debug messages, I find them absolutely necessary when working with automated Web tests.

When testing a Web-based application, error pages are included in the FitNesse page and displayed, making debugging much easier than working with log files.

Data-driven testing

While keyword-driven testing is fine for GUI automation, data-driven testing is the first choice for testing code that does any kind of calculation. If you have written unit tests before, what is the single most annoying thing about those tests? Chances are, you think of data. Your tests will be full of data, which often changes, making maintenance a nightmare. Testing different combinations requires different data, probably making your tests complicated, ugly beasts.

With data-driven testing, the data is separated from the code. It is usually created in some kind of table, stored as a CSV file, and read in by the test code. With FitNesse, we can conveniently store, change, and access this test data.

Let’s change the table illustrated in Figure 2 into a test. To do that, copy the table to Excel, change it, and use FitNesse’s Excel import function. This is a useful function, as large tables tend to be hard to maintain in FitNesse. The result is demonstrated by Figure 5. Each row represents a test. The columns with the question mark represent Java methods that we need to write, while those without a question mark contain the input data for the test.

Again, the test is quite easy to understand when read in the context of the requirements. If you have ever wondered when to use equivalence classes for your test cases, here they are. The last rows are probably the most important, as the requirements don’t say anything about how to handle invalid data or what to do with children older than 20 and still in school. The keyword “error” states that an exception is expected.

FAQ

Demonstrating the possible ways to combine requirements, testing, and development with FitNesse leaves many open questions. Some answers to those questions follow.

What becomes of the manual testers?

The work of a manual tester often consists of the repetitive task of executing manual regression tests. This is not only expensive, but also boring and error-prone. Test automation can reduce the amount of manual work needed for regression testing; it does not, however, make the manual tester obsolete. He will have more time for the interesting tests, such as complex scenarios spanning several areas of the application. Though these tests take longer to find errors, the costs of not finding them are higher.

Is acceptance testing not necessary anymore?

One might think so. If the customer does define/review all test cases within the requirements, the software should be “ready” when all tests are green. My practical experience, however, is that the customer will come up with additional requirements. Still, there should be fewer changes, and those can be far more easily classified as real changes (resulting in additional payments).

Does writing requirements in FitNesse offer advantages to Word documents?

Even without any tests, using a wiki for requirements might give you some advantages. The documents are automatically available on a Web server and can be accessed by many people simultaneously. Full text search is available and documents can be linked. I find two things especially noteworthy:

• Creating a data dictionary and linking terms to this is easy. Even more important is the “where-used” function. If we have to change a term, we can see at once where the term is used and find potential areas of conflict from the change.
• As Bugzilla (or Jira) is Web-based, we can easily link artifacts (bugs, tasks, discussions, etc.) between these two systems, making our work much more efficient.

FitNesse allows us to execute tests in a certain order, JUnit forbids this. What is your experience with this?

With JUnit, you tend to have lots of setUp() and tearDown() methods that create and clean up, respectively, the same objects. In certain cases, it can be nice to have all related tests on a single FitNesse page and be able to rely on their order. Thus, the first test can create a person, the next three can do some modifications, and the last will delete it. What if the first test fails, and the person is not created? No problem. Since the first test failed, I won’t look at the others until the first is fixed. I don’t use these sequences often, but sometimes it can make my work much easier.

Is FitNesse ready to be used in large projects?

Most developers will understand the advantages of FitNesse quite easily and can live with its simplicity. Nontechnical persons might find it difficult to work without features like an automatic spellchecker or a WYSIWYG text editor. A simple example is that FitNesse only allows three categories of headlines, while our requirements contained five. We extended FitNesse to provide these additional levels.

FitNesse does not include any PDF export or the like. I am used to working with Word documents up to 60 pages in length. Though they might be hard to handle online, they are easy to print. After I moved them to FitNesse, I couldn’t print them anymore, as they each turned into several pages. I had to program a workaround for this.

FitNesse does have a simple mechanism to show what documents changed and even by whom, but not what changed. The authorization functions are limited. Starting a new project with FitNesse is, of course, much easier than reworking a running one. For small- to medium-sized projects, I consider FitNesse highly recommendable. Large projects can be completed, but they do come with a price and should be evaluated carefully.

Conclusion

I found FitNesse to be a good tool for requirements engineering. It did help implement a test-first software development. It does come with a price, however. I had to invest several weeks to adapt both FitNesse and the processes. My project is not finished yet, but I see a strong rise in the quality of both the requirements and software. And the job as a tester is much more fun than it was before.