(\_ -> Phil Varner)

Since migrating from JUnit 3, TestNG has been wonderful. Groups are the killer feature of TestNG that really make it worth the migration cost. When wanting to test a single method, I no longer need to manually comment or uncomment method names in the suite() method, I can just add a new group and run it from the command-line (well, from Ant. see below). Annotation-based test methods are much nicer, and have a much lower risk of accidentally being left out of the suite.

Just a few caveats before I show the Ant/TestNG setup we're currently using.

• Merely moving to a new framework exposed several unintended test dependencies, so tests then failed because they ran after other tests. With the suite method, they always ran in the same order, so these dependencies were never found. None of ours were important, but there could have been ones that masked bugs.
• Only void methods with names starting with "test" are annotated with @Test. This may seem obvious, but we had a few tests written by a developer auxiliary to our main team who had written a few tests that weren't prefixed properly, but ran because they were in the suite method. The JUnitConverter class should probably try and parse the suite method to find problems like this (Maybe I'll to a patch for this).
• Only void returning methods annotated with @Test are run. Having a test method return a value doesn't generally make sense (it didn't in this case, either), but it may be difficult to understand if you test isn't running even though it's annotated.

Okay, so now onto the good stuff– our Ant/TestNG configuration. In these examples, I've replaced the name of my actual project with "foobar", and the prefix "sdk" indicates that it's the SDK part of the project.

First, in our common.xml file that is imported by all of our individual build.xml files, I added these lines, to define the location of the jar, add it to the common classpath, define the Ant task, and define the location for the reports to go ($twork is set to a temporary directory for the build):

<property name="testng.jar" value="${test.src.dir}/lib/testng-5.7-jdk15.jar" /
>

<path id="foobar.common.class.path">
  ....
  <pathelement location="${test.src.dir}/lib/testng-5.7-jdk15.jar"/>
  ....
</path>

    <taskdef name="testng" classpathref="foobar.common.class.path"
          classname="org.testng.TestNGAntTask" />

    <property name="testng.report.dir" value="${twork}/testng-report" />

Then in the build.xml for the specific tests, I added these targets. To clean, I added a target to delete old results:

    <target name="clean">
        <delete failonerror="false" quiet="false" includeemptydirs="true">
            <fileset dir="${testng.report.dir}" includes="**/*"/>
        </delete>
    </target>

Then I added a couple of targest to either produce a single "suc" (success) or "dif" (failure) file based on the results of the run (these files are used by the continuous build system to report the results of running the tests on a new build).

UPDATE:See this post for an updated version of the following targets.

    <target name="process-results" depends="copy-failure, copy-success" />

    <target name="copy-failure" if="has.failure">
        <copy file="${testng.report.dir}/testng-failed.xml"
                tofile="${T_WORK}/foobar.sdk.${infix}.dif"
                failonerror="false" overwrite="true" />
    </target>

    <target name="copy-success" if="has.success">
        <copy file="${testng.report.dir}/testng-results.xml"
                tofile="${T_WORK}/foobar.sdk.${infix}.suc"
                failonerror="false" overwrite="true" />
    </target>

Then, we have the target that actually calls the testng task. This target is never called directly, only through helper targets. Notice that it adds two listeners: one that will give use intermediate results on the command-line as the tests are running, and one that will give us a summary report at the end. After running, it then calls the previously mentioned targets. One thing I missed at first was that the test class files must be included in both the classpath (so the JVM can find them) and the classfileset element, so that TestNG will know what classes to use for the tests.

    <target name="run-testng" depends="" >
          <property name="excluded-groups" value=""/>
        <testng groups="${groups}" outputDir="${testng.report.dir}"
listeners="foobar.test.sdk.TestListener,foobar.test.sdk.SDKReporter"
excludedgroups="${excluded-groups}" >
            <jvmarg value="-ea"/> <!-- enable assertions -->
            <classpath>
                <pathelement path="${twork.sdk}"/>
                <pathelement path="${foobar.common.class.path}"/>
            </classpath>
            <classfileset dir="${twork.sdk}" includes="foobar/test/**/${t
estcase}.class"/>
        </testng>
        <condition property="has.failure" value="true" >
            <available file="${testng.report.dir}/testng-failed.xml" />
        </condition>

        <condition property="has.success" value="true" >
            <available file="${testng.report.dir}/testng-results.xml" />
        </condition>

        <antcall target="process-results" >
          <param name="infix" value="${groups}"/>
        </antcall>

    </target>

The next thing was to set up a few helper targets to call the run-testng target. The first was "run-testcase", which would allow you to run a group from only a specific TestCase class, for a feel similar to JUnit. This is run with the command line 'ant run-testcase -Dgroups=srg -Dtestcase=BazTestCase'. Note that the group "broken" is excluded by default. If you actually want to run the broken group, you call it with 'ant run-testcase -Dgroups=broken -Dtestcase=BazTestCase -Dexcluded-groups=""' to populate the property exclude-groups so it's redefinition is ignored. Also, we add the most used command-line target, rung. This called with "ant rung -Dgroups=srg", or more commonly when I'm using it, "ant rung -Dgroups=phil". I can just add my name to the groups for a test case, and easily run only that one while debugging code or writing new tests. This alone was worth the migration to TestNG– it's liberating when writing tests.

    <target name="run-testcase" depends="setup">
        <property name="groups" value=""/>
        <property name="exclude" value="broken"/>
        <antcall target="run-testng">
          <param name="testcase" value="${name}"/>
          <param name="groups" value="${groups}"/>
          <param name="excluded-groups" value="${exclude}"/>
        </antcall>
    </target>

    <target name="rung" depends="setup">
        <antcall target="run-testng">
          <param name="testcase" value="*"/>
          <param name="groups" value="${groups}"/>
        </antcall>
    </target>

This is the listener that reports the intermediate results from running each test method. The name of the test class has its front chopped off so most of them will fit in an 80 character column, and it also prints the count of the tests (to gauge how far progressed the tests are) and the run-time for each test (to help gauge if there are any high-runtime/low-value tests out there). The one thing I would like to add but haven't looked at yet is printing out the actual results of the assert failure rather than just the stack trace of where it occured. I currently just look in the HTML report at the end for this.

package foobar.test.sdk;

import org.testng.*;

public class TestListener extends TestListenerAdapter {
    private int m_count = 0;

    private String name(ITestResult tr){
        return tr.getTestClass().getName().replaceAll("foobar\\.test\\
.","") +
            "." + tr.getMethod().getMethodName();
    }

    @Override
    public void onTestFailure(ITestResult tr) {
        log("[FAILED " + (m_count++) + "] => " + name(tr) );
    }

    @Override
    public void onTestSkipped(ITestResult tr) {
        log("[SKIPPED " + (m_count++) + "] => " + name(tr) );
    }

    @Override
    public void onTestSuccess(ITestResult tr) {
        log("[" + (m_count++) + "] => "+ name(tr) + " " + (tr.getEndMillis()-t
r.getStartMillis()) + "ms");
    }

    private void log(String string) {
        System.out.println(string);
    }
}

This is the reporter that I use for the summary report at the end of running all the tests:

package foobar.test.sdk;

import org.testng.*;
import java.util.*;

import static java.util.Arrays.asList;

public class SDKReporter implements IReporter {

    private String name(ITestResult tr){
        return tr.getTestClass().getName() + "." + tr.getMethod().getMethodNam
e();
    }

    public void generateReport(List<org.testng.xml.XmlSuite> xmlsuites ,List<o
rg.testng.ISuite> suites,String c) {

        for (ISuite suite : suites){
            Map<String,ISuiteResult> results  = suite.getResults();
            for(Map.Entry<String,ISuiteResult> entry : results.entrySet()){
                ITestContext itc =   entry.getValue().getTestContext();
                for (ITestResult tr : itc.getFailedConfigurations().getAllResu
lts()){
                    log ("Failed Config: " + name(tr));
                    log (asList(tr.getThrowable().getStackTrace()));
                }

                for (ITestResult tr : itc.getFailedTests().getAllResults()){
                    log ("Failed Test: " + name(tr));
                    log (asList(tr.getThrowable().getStackTrace()));
                }

                for (ITestResult tr : itc.getSkippedConfigurations().getAllRes
ults()){
                    log ("Skipped Config: " + name(tr));
                    log (asList(tr.getThrowable().getStackTrace()));
                }

                for (ITestResult tr : itc.getSkippedTests().getAllResults()){
                    log ("Skipped Test: " + name(tr));
                    log (asList(tr.getThrowable().getStackTrace()));
                }

            }
        }
    }

    public void log(java.util.List<java.lang.StackTraceElement> trace){
        for (StackTraceElement ste : trace){
            String s = ste.toString();
            if (s.startsWith("sun.reflect.NativeMethodAccessorImpl")) {
                log("\n-------------------------------------\n");
                return ;
            }
            log("\t" + s);
        }
    }

    public void log(String s) {
        System.out.println(s);
    }
}