TestNG: The next generation of unit testing

TestNG, written by Cedric Beust and Alexandru Popescu, is a light framework based on Java annotations (for J2SE 5.0) that allows you to design complex unit testing for J2SE 5.0 and J2SE 1.4. Why bother learning another unit-testing framework when you’re already comfortable using JUnit? If you are interested in simplifying your unit-test cases, in leveraging J2SE 5.0 annotations to tag your test classes as well as being backward compatible with J2SE 1.4, in having out-of-the-box support for dependent methods and parallel and asynchronous testing, TestNG is the tool you are looking for.

This article starts with a description of the JUnit annoyances and introduces TestNG through numerous examples. A case study shows how to use TestNG for asynchronous testing.

JUnit annoyances

The JUnit testing framework has been around for a long time and is (hopefully) widely used by Java developers to unit test developed software. However, the framework has some annoying specificities, which the following sections describe.

Multiple instantiations per TestCase

Multiple TestCase instantiations is a well-known issue. The setup() and teardown() methods are called before and after each test method; moreover, the test class that must extend TestCase is also instantiated each time a test method is called. Although multiple TestCase instantiations might prove acceptable for simple test cases, what if you want to set up an object that is to be reused across more than one test method, for example, a Java Database Connectivity connection (or, for that matter, a Java Naming and Directory Interface (JNDI) context)?

PainOneTest.java illustrates the problem:

 
public class PainOneTest extends TestCase {
  /**
   * PainOneTest
   */
  public PainOneTest() {
    System.out.println("PainOneTest");
  }
  /**
   * @see TestCase#setUp()
   */
  protected void setUp() throws Exception {
    System.out.println("setUp()");
  }
  /**
   * @see TestCase#tearDown()
   */
  protected void tearDown() throws Exception {
    System.out.println("tearDown()");
  }
  /**
   * testMe
   */
  public void testMe() {
    System.out.println("testMe");
  }
  /**
   * testYou
   */
  public void testYou() {
    System.out.println("testYou");
  }
}

This code produces the following output:

 PainOneTest
PainOneTest
setUp()
testMe
tearDown()
setUp()
testYou
tearDown()

The TestSetup class circumvents this issue, as illustrated below:

 
public class PainTwoTest extends TestCase {
  /**
   * Test
   * @return
   */
  public static Test suite() {
    return new TestSetup(new TestSuite(PainTwoTest.class)) {
      public void setUp() throws Exception {
        System.out.println("setUp()");
      }
      public void tearDown() throws Exception {
        System.out.println("tearDown()");
      }
    };
  }
  /**
   * PainOneTest
   */
  public PainTwoTest() {
    System.out.println("PainOneTest");
  }
  /**
   * testMe
   */
  public void testMe() {
    System.out.println("testMe");
  }
  /**
   * testYou
   */
  public void testYou() {
    System.out.println("testYou");
  }
  /**
   * main
   * @param _
   */
  public static void main(String []_) {
   Test test = PainTwoTest.suite();
  }
}

This code produces:

 PainOneTest
PainOneTest
setUp()
testMe
testYou
tearDown()

The above solution introduces a static method, and all variables accessed by it must also be declared static. Although I have nothing in particular against static methods, you must take care when implementing test methods for concurrency access (sharing static objects) and static initialization (occurring only once per VM).

Remember, when using Ant, the default behavior is to use the VM that started Ant to execute all the tasks; in the above scenario, you may well have to use the fork attribute of the Java core task.

Furthermore, multiple instantiations of the TestCase class remain. Surely, if we want the test methods to be independent of each other, we would place them into different TestCase classes. So why multiple instantiations? Multiple instantiations allow you to isolate independent methods within the same TestCase class, which is not always the desired behavior. Finally, too much technical code is required for circumventing the multiple-instantiations scenario: I just want a test POJO (plain old Java object) class.

How does TestNG handle multiple instantiations?

TestNG does not require static block initialization and has a flexible configuration scheme for handling test classes based on regular expressions and XML configuration files. TestNG does not instantiate the test class several times.

Multithreaded support

GroboUtils is an addition to JUnit that supports multithreaded unit tests (since the JUnit framework lacks such support). N. Alex Rupp gives an overview of GroboUtils in “Multithreaded Tests with JUnit” (java.net, August 2003). Although very useful, GroboUtils is not to my taste as it requires too much glue code to handle multithreaded unit tests.

How does TestNG handle multithreaded unit tests?

TestNG has multithreaded and parallel unit tests built in its core. You don’t need to write specific code to handle multithreaded unit tests as they are just a configuration of TestNG.

Why should I extend a specific class and prefix the methods with test?

I would rather tag any method than be obliged to prefix methods with test, a task that JUnit requires. Admittedly, this is a minor point, but all IDEs now give you a view (like the Eclipse outline view) that helps you quickly browse your methods to find the one you want—that is, if they are not all prefixed with the same four letters. From a practical viewpoint, the Eclipse outline view is useless when you have dozens of test methods on a class that start with the same four letters.

How does TestNG handle test method names?

TestNG uses Java annotations as specified by Java Specification Request 175 to tag methods for testing and grouping, and for expected exceptions, and so on.

Introducing TestNG

TestNG is inspired by JUnit and also NUnit a unit-testing framework for .Net.

TestNG introduces new functionalities to unit testing such as:

• Support for Java annotations (if unfamiliar with annotations, see sidebar “What Are Annotations“)
• XML configuration file for test configuration
• No required class extension or interface implementation
• Support for dependent methods and groups
• Support for parallel testing
• Parameters for test methods
• Arbitrary number of invocations plus success rate

Step by step: Your first TestNG program

Let’s get started with TestNG. The remaining sections of this article introduce you to the TestNG features, starting with a basic first unit test: how to configure it and run it.

Here is a simple test class:

 
package org.jyperion.testng;
import com.beust.testng.annotations.*;
public class FirstTest {
  @Configuration(beforeTestClass = true)
  public void configure() {
    System.out.println("configure");
  }
@Test(groups = {"exec-group"})

    public void exec() {
    System.out.println("exec");
    
assert 1 == 2;

  }
}

Note that no interface or class is required for your test class. The @Configuration annotation is the equivalent of the JUnit method setup(); you can annotate any method in your test class. @Configuration(beforeTestClass = true) means that the annotated method will be called just once before testing any method in the class. The @Test annotation is equivalent to the JUnit methods prefixed with the word test.

Next comes the XML configuration test called testng.xml:

 
<!DOCTYPE suite SYSTEM "http://beust.com/testng/testng-1.0.dtd" >
  
<suite name="My First TestNG test">
  
  <test name="Hello Test!">
    <classes>
      <class name="org.jyperion.testng.FirstTest" />
    </classes>
  </test>
</suite>

Finally, TestNG’s invocation:

 java –ea com.beust.testng.TestNG testng.xml

Note: -ea enables assertions.

The following appears on the console:

 
configure
exec
[TestRunner] FAILED: org.jyperion.testng.FirstTest.exec
[TestRunner] REASON: java.lang.AssertionError
===============================================
TestNG Samples
Total tests run: 1, Failures: 1 Skips: 0
===============================================

An HTML report is also created in a folder called test-output:

An interesting feature of TestNG is the test setup mechanism enabled by using the @Configuration annotation in four different scenarios:

• Run configuration method once before all tests
• Run configuration method once before every test
• Run configuration method once after every test
• Run configuration method once after all tests

Moreover, if you use TestNG’s powerful grouping feature (described in subsequent sections), the @Configuration methods can have a determined execution order.

TestNG DTD view

The configuration is defined by a DTD (document type definition). Figure 2 shows the schematic view.

The test annotation

The @Test annotation is defined as:

 @Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({java.lang.annotation.ElementType.METHOD, 
         java.lang.annotation.ElementType.TYPE})
public @interface Test {
  public String[] groups() default {};
  public boolean enabled() default true;
  public String[] parameters() default {};
  public String[] dependsOnGroups() default {};
  public long timeOut() default 0;
}

The groups element allows you to group methods and groups. By configuring testng.xml, you can include or exclude groups using regular expressions:

 
public class GroupTest {
  @Test(groups = {"group-one"})
  public void init() {
    System.out.println("init#group-one");
  }
  @Test(groups = {"group-two"}, dependsOnGroups = {"group-one"})
  public void exec() {
    System.out.println("exec#group-two");
  }
@Test(groups = {"group-three"})

  public void testMe() {
    System.out.println("testMe#group-three");
  }
}

The corresponding testng.xml:

 <!DOCTYPE suite SYSTEM "http://beust.com/testng/testng-1.0.dtd" >
  
<suite name="My First TestNG test">
  <test name="Group Test">
    <groups>
      <run>
        <include name="group-.*" />
      </run>        
    </groups>
  
    <classes>
      <class name="org.jyperion.testng.GroupTest" />
    </classes>
  </test>
</suite>

Parameters

Parameters defined in testng.xml can be passed to the test methods at runtime:

 public class ParameterTest {
  @Test(parameters = { "jndi-name" })
  public void lookup(String jndiName) {
    System.out.println("lookup("+jndiName+")");
  }
}

The corresponding testng.xml:

 <!DOCTYPE suite SYSTEM "http://beust.com/testng/testng-1.0.dtd" >
  
<suite name="My First TestNG test">
  <test name="Parameter Test">
    <classes>
      <class name="org.jyperion.testng.ParameterTest">
        <parameter name="jndi-name" value="java:comp/env/ejb/ECCSB"/>
      </class>
    </classes>
  </test>
</suite>

Note that parameters can be placed within the suite element, within the test element, or within the class element, so that your parameter is available to all the classes in the suite, to all the classes in the test, or to only the class.

The corresponding testng.xml:

 
<suite name="TestNG Samples" 
       verbose="1" 
       parallel="true" 
       thread-count="10">
  <test name="ParallelTest">
    <classes>
      <class name="org.jyperion.testng.ParallelTest" />
    </classes>
  </test>
  
</suite>

The element parallel is set to true to instruct TestNG to run the tests in parallel. thread-count specifies the number of threads in the pool (if omitted, default is 5).

The HTML report in Figure 3 shows that the methods run chronologically and also presents the thread identifier used to execute a test method.

Figure 4 shows the threads associated with the test methods.

TestNG allows two running scenarios:

• Sequential run of tests (default)
• Parallel run of tests

To select which scenario to use, simply edit the testng.xml.

Another nice feature of TestNG is its support of timeouts, which can be used in parallel and nonparallel modes using the timeOut attribute in @Test.

Ant task

TestNG is also shipped with an Ant task:

 
<testng
  fork="yes"
  classpath="${my.classpath}"
  outputDir = "${basedir}/test-output">
    <propertyfile value="${basedir}/testng.xml"/>
    <jvmarg value="-ea" />
<testng>

Using this Ant task (which is part of the TestNG distribution), TestNG could be used for continuous integration, for example.

Support for J2SE 1.4

TestNG supports old-style javadoc-like tags for J2SE 1.4. I do not bother describing them here because they are not of much interest if you are testing J2SE 5.0 software. However, TestNG can be used for testing J2SE 1.4 applications.

Extending TestNG

TestNG is available under the Apache Software License, and, as an open source project, can be extended to enable further development. The TestNG distribution comes with the source, and, in this section, I outline a way to register a listener that listens to events when TestNG executes tests. This approach resembles a SAX (Simple API for XML) implementation in that you are notified while TestNG runs the tests. I illustrate how to extend TestNG by implementing a PDF report generator using the fantastic iText library, released under MPL and LGPL.

Note: Cedric Beust is currently designing a flexible plug-in framework that will allow developers to extend TestNG beyond JUnit’s abilities.

To extend TestNG to generate a PDF report, you (currently) need to:

• Write a class that implements the com.beust.testng.ITestListener interface
• Register that class in the initLoggers() method from the com.beust.testng.TestRunner class
• Rebuild the TestNG JAR

You can find the code for the PDF generator in the JyperionListener.java file; to see the result, open the JypTest.pdf file (both files are downloadable from Resources).

Case study: Price publishing and asynchronous testing

In this simple case study, we simulate sending prices to an electronic communications network (ECN) market (implemented by a Remote Method Invocation server) from a client using Java Message Service (JORAM—Java Open Reliable Asynchronous Messaging—from ObjectWeb is used as message-oriented middleware). This study illustrates how to use TestNG for testing asynchronous code (for more on the topic, read the Testing Asynchronous Code Weblog). Our ECN simulator sends the same price back to the client 90 percent of the time, acknowledging the market’s price. It sends a slightly different bid and ask price 5 percent of the time and an “off” status for that quote another 5 percent of the time. The price publisher is a Java client that publishes messages using Java Message Service (JMS).

The TestPricePublisher is just another JMS client that uses PricePublisher to send prices and then waits for a price to return (using wait(long) so a long delay can cause LongUpdateException). The TestPricePublisher also checks if the price received is identical to the one sent (if not, a WrongPriceException is thrown) and if the price status is on (if not, an OffException is thrown).

The code for asynchronous testing (from TestPricePublisher) follows below:

 
@Test(invocationCount = 20)
public void sendPrice() throws LongUpdateException, WrongPriceException, OffException, InterruptedException {
  this.sendPrice("JYPERION", this.askPrice, this.bidPrice);
  this.askPrice += 0.01;
  this.bidPrice += 0.01;
  synchronized (this) {
    try {
      long TIMEOUT = 100;
      this.wait(TIMEOUT);
      if (this.quote == null)
        throw new LongUpdateException("No message received");
      if (quote.getStatus() == PriceStatus.OFF)
        throw new OffException("Quote " + this.quote.getISIN() + " is OFF");
      if (!quote.isPriceConsistent())
        throw new WrongPriceException("Wrong price for " + this.quote.getISIN());
    } catch (InterruptedException e) {
      throw new InterruptedException();
    } finally {
      this.quote = null;
    }
  }
}
/**
 * @see javax.jms.MessageListener#onMessage(javax.jms.Message)
 */
public void onMessage(Message msg) {
  try {
    
synchronized(this)
 {
      this.quote = (Quote) ((ObjectMessage)msg).getObject();
      LOGGER.info("Notifying waiting threads");
      
this.notifyAll();

    }
  } catch (JMSException e) {
    e.printStackTrace();
  }
}

The corresponding testng.xml:

 <!DOCTYPE suite SYSTEM "http://beust.com/testng/testng-1.0.dtd" >
  
<suite name="Case Study: Price Publishing" 
       parallel="true"
       thread-count="10"
       verbose="1">
  
  <test name="TestPricePublisher" parallel="true">
    <classes>
      <class name="org.jyperion.testng.TestPricePublisher" />
    </classes>
  </test>
  
</suite>

The generated HTML report:

If you run the tests more than once, TestNG keeps them in the generated HTML page.

What is next?

Cedric and Alexandru are currently working on:

• A pluggable module architecture for TestNG (so you can plug in your PDF generation engine in a standard way)
• An Eclipse plug-in for TestNG, see a preview in Figure 6

Conclusion

This article described how to employ some new features for unit testing with TestNG. Among TestNG’s capabilities, I would qualify its support for J2SE 5.0 annotations, flexible test configurations based on XML files and regular expressions, support for dependent test methods, and core multithreaded support as being the most important. TestNG shows that there is still room for improvement in unit testing. I am certain developers would find refreshing ideas in TestNG.

One last thing, if TestNG is lacking a feature, you can argue your case in TestNG users’ mailing list. If Cedric and Alexandru are convinced, most likely, this feature will be added to the next release. Could you be more agile?