Test-driven GUI development with FEST

Alex Ruiz follows up his popular JavaOne 2007 presentation with this introduction to test-driven GUI development with FEST. Learn what differentiates FEST (formerly TestNG-Abbot) from other GUI testing frameworks and get a hands-on introduction to Swing GUI testing with this fast-growing developer testing library. Video demonstrations and working code examples are included.

Test-driven development (TDD) is an application-development technique wherein automated tests drive the software design and development process. Although TDD involves writing many tests, practitioners emphasize it is a design and development methodology. In TDD, tests are used to specify what the application code should do. When a test fails, developers implement the code necessary for the test to pass. The objective of TDD is clean production code that works as expected.

Developers using the iterative practice of TDD have reported the following benefits:

• Shorter code implementation time
• Reduction in code defects
• Fewer instances of overcomplicated and unnecessary code
• Reduced likelihood of introducing bugs when fixing bugs, refactoring or introducing new features

In spite of these benefits, developer adoption of TDD for GUI development has been slow, mainly because of the challenge of writing tests for GUIs. In addition, conventional unit testing, such as testing a class in isolation, often is not appropriate for GUI testing: A GUI “unit” can be made up of more than one component, each of them enclosing more than one class. In many cases, functional testing is a more effective way to test GUIs.

In this article I use code examples and video demonstrations to introduce test-driven GUI development with FEST, an open source library I created with Yvonne Wang Price to facilitate functional GUI testing in TDD environments. I start by using the java.awt.Robot class to create functional tests for a sample GUI. I walk through the steps of setting up a test and GUI, then explain the shortcomings of using Robot for functional GUI testing. I then introduce FEST and run the same tests using the FEST library. I conclude by discussing the differences between GUI testing with FEST vs. the Robot class.

Note that this article assumes you are familiar with GUI design and development and functional testing methodology. You need not be familiar with test-driven development to follow the examples.

Functional GUI testing with java.awt.Robot

The following factors are essential to creating a robust functional GUI test:

1. Being able to simulate user events
2. Having a reliable mechanism for finding GUI components
3. Being able to tolerate changes in a component’s position and/or layout.

java.awt.Robot is a class introduced in JDK 1.3 that can be used to test GUIs built using Swing and AWT. Robot meets the first of the three above requirements because it can be used to simulate user interaction by controlling the application environment’s mouse and keyboard.

To demonstrate functional GUI testing with the Robot class, I first will create a simple Swing GUI that contains a JLabel and a JButton. The GUI’s expected behavior is that when the user presses the JButton, the text of the JLabel will change from “Hi” to “Bye!” As good TDD practitioners do, I will start my development effort with a failing test.

Listing 1. A functional GUI test using java.awt.Robot

// Omitted imports and package declaration.

/**
 * Tests for {@link Naive}.
 *
 * @author Alex Ruiz
 */
public class NaiveTest {
  private Naive naive;

  @BeforeMethod public void setUp() {
    naive = new Naive();
    naive.setVisible(true);
  }

  @Test public void shouldChangeTextInTextFieldWhenClickingButton() throws AWTException {
    JButton button = naive.getButton();
    click(button);
    JTextField textField = naive.getTextField();
    assertEquals(textField.getText(), "Bye!");
  }

  private void click(JButton button) throws AWTException {
    Point point = button.getLocationOnScreen();
    Robot robot = new Robot();
    robot.setAutoWaitForIdle(true);
    robot.mouseMove(point.x + 10, point.y + 10);
    robot.mousePress(MouseEvent.BUTTON1_MASK);
    robot.mouseRelease(MouseEvent.BUTTON1_MASK);
  }
}

My next step is to create the GUI and eliminate the compilation errors in Listing 1. Figure 1 is a screenshot of my development setup for testing and building the GUI. Click the screenshot for a video demonstration of the steps taken to eliminate compilation errors from my code in preparing for running the test.

Listing 2 shows the source code for my simple Swing GUI.

Listing 2. A simple Swing GUI

// Omitted imports and package declaration.

/**
 * Understands a simple GUI.
 *
 * @author Alex Ruiz
 */
public class Naive extends javax.swing.JFrame {

  // Omitted code generated by GUI builder.

  private javax.swing.JButton jButton1;
  private javax.swing.JLabel jLabel1;

  public JButton getButton() {
    return jButton1;
  }

  public JLabel getLabel() {
    return jLabel1;
  }

}

When I run the following test, I expect it to fail.

As you can see from the video demonstration, the test launches the GUI and uses the Robot class to simulate a user moving a mouse pointer over the JButton and pressing the left mouse button. The JLabel does not change to the expected text, “Bye!”, so the test fails. The test has given me a good idea of what’s wrong with my code, so I just return to my IDE and write the necessary code to make the test pass, as shown in Figure 3.

Listing 3 shows the revised code. Note that all I had to do was change the text of the JLabel to show the expected text, “Bye!”, when the user clicks the JButton.

Listing 3. Revised code to make the test pass.

jButton1.addMouseListener(new java.awt.event.MouseAdapter() {
    public void mouseClicked(java.awt.event.MouseEvent evt) {
      jButton1MouseClicked(evt);
    }
  });

  private void jButton1MouseClicked(java.awt.event.MouseEvent evt) {
    jLabel1.setText("Bye!");
  }

Shortcomings of functional GUI testing with java.awt.Robot

Although I was able to simulate user events with the Robot class, I could not meet the other two requirements of functional GUI testing. Looking back to Listing 1, you may notice the following shortcomings of the test:

1. I had to look up GUI components myself because Robot does not provide a way to find them. To get a reference to a GUI component using Robot,I would have to use accessor methods (“getters”) or loop through the component hierarchy. Using getters for testing adds clutter to my GUI code. Likewise, looping through the component hierarchy can be tedious and may introduce code duplication.
2. Tests written using the Robot class are fragile: Any change in layout will break them.

The second issue poses a serious problem. When testing my example Swing GUI, I assumed the button was bigger than 10 pixels. If for some reason the button had been any smaller than the expected size, the test would have failed, as shown in Figure 4.

Tests designed to test the behavior of a GUI should not be affected by changes in the GUI layout. java.awt.robot also is a fairly low-level class and requires too much code to simulate user events. In the next section I’ll introduce an alternative to functional GUI testing with the Robot class.

Introducing FEST

FEST (Fixtures for Easy Software Testing) is an open source library licensed under Apache 2.0 that makes it easy to create and maintain robust functional GUI tests. Although several open source projects have been devised for testing GUIs, FEST is distinguished by the following features:

1. An easy-to-use Java API that exploits the concept of fluent interfaces to simplify coding. For example, FEST lets me rewrite the code from Listing 1 as follows:

   private FrameFixture naive;
   
     @BeforeMethod public void setUp() {
       naive = new FrameFixture(new Naive());
       naive.show();
     }
   
     @Test public void shouldChangeTextInTextFieldWhenClickingButton() {
       naive.button("byeButton").click();
       naive.label("messageLabel").requireText("Bye!");
     }
   

2. Assertion methods that detail the state of GUI components (for example, requireText("Bye!")). Such assertion methods can be useful when practicing TDD because they read like a specification.
3. Support for both JUnit 4 and TestNG.
4. Screenshots of failing tests, which can be embedded in an HTML test report when using JUnit or TestNG. This configurable feature is useful when verifying that a test or group of tests failed because of an environment condition and not a programming error.
5. A Groovy-based domain-specific language that simplifies GUI testing even further. (This feature is still under development and is considered experimental.)

Although FEST does provide some unique features, it does not reinvent the wheel. Instead of creating yet another mechanism for component lookup and user-event simulation, FEST builds on top of Abbot, a mature, open source project for GUI testing. In fact, FEST was born from a previous project I created with Yvonne Wang Price, TestNG-Abbot. The initial mission of TestNG-Abbot was to provide TestNG support to Abbot, but TestNG-Abbot quickly evolved from being a “glue” project into a nice API for GUI testing. Yvonne and I then decided to make the library independent of any test framework (JUnit or TestNG)] and changed its name to FEST to reflect its new nature.

Tests created with FEST are robust because they are not affected by changes in layout or component size. In addition, FEST provides features not available in other GUI testing libraries — its simple but powerful API being the most important one.

Test-driven GUI development with FEST

In the following sections you’ll get to know FEST by walking through a testing example. You should download FEST now if you want to follow along with the examples. Figure 6 is a sketch of the example GUI to be tested. It represents a login dialog where the user enters her user name, password and domain name to log in to the system.

The expected behavior of the dialog is as follows:

1. The user enters her user name and password, both required.
2. The user selects from the drop-down list the domain to which she wishes to connect.
3. If any of the fields is left blank, a pop-up dialog box notifies the user that the missing information is required.

As I did in the first example, you will start with a failing test.

Listing 4. A FEST test that verifies an error message

// Omitted imports and package declaration

 1  public class LoginWindowTest {
 2
 3    private FrameFixture login;
 4
 5    @BeforeMethod public void setUp() {
 6      login = new FrameFixture(new LoginWindow());
 7      login.show();
 8    }
 9
10    @Test public void shouldShowErrorIfUsernameIsMissing() {
11      login.textBox("username").deleteText();
12      login.textBox("password").enterText("secret");
13      login.comboBox("domain").selectItem("USERS");
14      login.button("ok").click();
15      login.optionPane().requireErrorMessage().requireMessage("Please enter your username");
16    }
17
18    @AfterMethod public void tearDown() {
19      login.cleanUp();
20    }

Notes about the test

The test in Listing 4 uses FEST to invoke the GUI being tested, simulate user events and verify that the GUI works as expected. More specifically, the test does the following:

• Uses a org.fest.swing.fixture.FrameFixture to manage and launch the window to test (lines 6 and 7).
• Ensures that the text field where the user enters her user name is empty (Line 11).
• Simulates a user entering the password “secret” in the appropriate text field (Line 12).
• Simulates a user selecting a domain from the drop-down list (Line 13).
• Simulates a user clicking the “OK” button.
• Verifies that a pop-up window (a JOptionPane) is displayed showing an error message with the text “Please enter your username” (Line 15).

Two more important observations about the test shown in Listing 4:

• FEST performs component lookup using the component’s unique name, which guarantees that a GUI component always can be found, as long as it hasn’t been removed from the GUI. It is possible to find components using custom criteria specified in a org.fest.swing.ComponentMatcher, but it may not be as reliable as using the component’s name. For example, finding a component by its displayed text is problematic because such text could change over time (for instance, as result of changes in the user’s preferences or by adding support for multiple languages).
• It is necessary to release resources used by FEST (such as the keyboard, mouse and opened windows) following the execution of each test (as shown in Line 19). You can release used resources by calling the method cleanUp in org.fest.swing.fixture.FrameFixture, org.fest.swing.fixture.DialogFixture, or org.fest.swing.RobotFixture.

Creating and testing the GUI

Once you have finished writing the test, you create the GUI to test and verify that the test does in fact fail, as demonstrated in Figure 7.

One important detail to notice is that specified component names must match the ones expected in the original test from Listing 4. This way, FEST can find those components easily and simulate user events on them. Listing 5 shows how you would specify the names of components in the login window using a GUI builder.

Listing 5. The GUI’s login window

// Omitted additional code generated by GUI builder.

  jLabel1.setText("Username:");
  jLabel2.setText("Password:");
  jLabel3.setText("Domain:");
  jTextField1.setName("username");
  jPasswordField1.setName("password");
  jComboBox1.setName("domain");
  jButton1.setName("ok");

In Listing 6, you add the user-name validation to make the test pass.

Listing 6. Adding user-name validation

jButton1.addMouseListener(new java.awt.event.MouseAdapter() {
    public void mouseClicked(java.awt.event.MouseEvent evt) {
      jButton1MouseClicked(evt);
    }
  });

  private void jButton1MouseClicked(java.awt.event.MouseEvent evt) {
    String username = jTextField1.getText();
    if (isEmpty(username))
      JOptionPane.showMessageDialog(this, "Please enter your username", "Error", JOptionPane.ERROR_MESSAGE);
  }

Figure 8 further demonstrates the steps to add the user-name validation and see the test pass.

As you see in Figure 8, the test passes and the user-name validation is complete. You’re not quite finished with the login window, however. The requirements specify that you still need to implement the following behavior:

• An error message to be displayed if the user does not enter her password
• An error message to be displayed if the user does not choose the domain to which she wishes to connect
• A successful login

I won’t go over the steps to complete the sample GUI here, but you certainly could build in the additional features yourself. The important point to take away from this simple example is the confidence you will have in a GUI built using FEST. The TDD approach ensures that your GUI will work as expected when you are done with it. You’ll also end up with a comprehensive test suite, which you can use to ensure that any future changes to the GUI (such as changing the component layout or adding some “makeup” to it) will not introduce bugs accidentally.

You’ve seen a fairly easy, successful example of test-driven GUI development using FEST. Next I’ll show you how to handle a less straightforward testing scenario.

Verifying test failures

On some occasions a functional GUI test will run perfectly from within the IDE but will break when executed in a batch with other tests (such as when you are using Ant). This is because functional GUI tests are vulnerable to certain environment-related events, and FEST is no exception. For instance, it occasionally happens that antivirus software runs a scheduled scan while a GUI is under test. If the antivirus software pops up a dialog in front of the GUI, the FEST robot will not be able to access the GUI and will time out eventually, so the test will fail. In this case, the failure is not related to a programming error; it is just a case of bad timing. Fortunately. in such cases you can verify the cause of failure easily by rerunning your test suite.

As I previously mentioned, one of the features of FEST is its ability to embed a screenshot of a failed test in its HTML test report. You then can use this screenshot to verify the cause of a failed test and discover whether it is programmatic or environmental. Configuring FEST to take screenshots of failed tests is pretty simple. The first step is to “mark” a GUI test with the annotation org.fest.swing.GUITest. You can place this annotation at either the class or method level.

Listing 7 is an example of a class “marked” as a GUI test. Every test method in this class will be considered a GUI test, even the ones in subclasses.

Listing 7. A class marked as a GUI test

import org.fest.swing.GUITest;
// rest of imports

@GUITest public class LoginWindowTest {

  @Test public void shouldShowErrorIfUsernameIsMissing() {
    // implementation of the test
  }
}

If you need more control, you can annotate only the methods that should be considered GUI tests. If you override a method marked as GUI test, the overriding method also will be considered a GUI test, even if it does not contain the org.fest.swing.GUITest annotation. This is shown in Listing 8.

Listing 8. Methods marked for testing

import org.fest.swing.GUITest;
// rest of imports

public class LoginWindowTest {

  @GUITest @Test public void shouldShowErrorIfUsernameIsMissing() {
    // implementation of the test
  }

  @Test public void someNonGUITest() {
    // implementation of the test
  }
}

Failure notification

The second and final step is to alert your testing framework (in this case TestNG) to notify FEST when a GUI test has failed. This way, FEST can take a screenshot of the failed test and embed it in the test report. It is quite easy to configure TestNG, thanks to its flexible architecture that supports extensions. The only change necessary is the declaration of the TestNG listener org.fest.swing.testng.ScreenshotOnFailureListener, which is provided by FEST. The following is an example configuration using TestNG and Ant:

Listing 9. Configuring TestNG to notify FEST if an alert fails

<target name="test" depends="compile">

  <testng listeners="org.fest.swing.testng.ScreenshotOnFailureListener" outputDir="${target.test.results.dir}">

    <classfileset dir="${target.test.classes.dir}" includes="**/*Test.class" />

    <classpath location="${target.test.classes.dir}" />

    <classpath location="${target.classes.dir}" />

    <classpath refid="test.classpath" />

  </testng>

</target>

Note that FEST’s support for failure notification currently extends only to TestNG. Similar support for JUnit is under development and should be complete by the time you read this article!

Figure 9 shows an embedded screenshot of a test failure.

Every testing methodology has its weakness and functional testing, with its vulnerability to environmental factors, is no exception. While FEST doesn’t overcome this weakness completely, it does let you account for it. Configuring FEST for failure notification makes it easy to determine whether a test has failed because of an environmental factor or because of a programming error.

In conclusion

Test-driven development is a software-development technique where automated tests drive the design of functional code. In spite of its reported benefits, Java developers have been slow to adopt TDD as a fundamental GUI-development practice because testing GUIs is difficult. In this article I have introduced FEST, an open source library that I developed in collaboration with Yvonne Wang Price, that provides an easy-to-use API for GUI testing. FEST makes it easier to write and maintain robust GUI tests, and that gives you more time to focus on what matters: specifying and verifying the behavior of your Swing GUIs.

FEST is a useful alternative to existing GUI-testing solutions. It’s easy to learn and use, and provides some unique features that can make GUI development more productive and fun. As you’ve learned in this article, FEST facilitates the adoption of TDD when developing Swing-based GUIs on the Java platform.

Concluding thoughts on developer testing

Testing software is imperative, regardless of the software development technique or tools used. Testing software is essential because code that has not been tested is a sure source of bugs. Here are some suggestions for testing software:

• If a test-first approach, like the one that TDD advocates, is not a good fit for you or if it doesn’t work for you all the time, test after you write functional code.
• Pick a testing framework, either JUnit or TestNG, and test your code.
• Write tests that include your GUIs.
• If FEST does not satisfy your testing needs, pick another GUI testing library. (Although we always appreciate feedback that can help us improve FEST!)
• Write tests that give you the confidence that your code works as expected.
• Focus on writing meaningful tests regardless of type (unit, functional or integration).

The bottom line: Choose the tools and techniques that satisfy the needs of your project and start testing your code!