TestNG, part 2

Since migrating from JUnit 3, TestNG has been wonderful. Groups are the killer feature of TestNG that really make it worth the migration cost. When wanting to test a single method, I no longer need to manually comment or uncomment method names in the suite() method, I can just add a new group and run it from the command-line (well, from Ant. see below). Annotation-based test methods are much nicer, and have a much lower risk of accidentally being left out of the suite.

Just a few caveats before I show the Ant/TestNG setup we're currently using.

  • Merely moving to a new framework exposed several unintended test dependencies, so tests then failed because they ran after other tests. With the suite method, they always ran in the same order, so these dependencies were never found. None of ours were important, but there could have been ones that masked bugs.
  • Only void methods with names starting with "test" are annotated with @Test. This may seem obvious, but we had a few tests written by a developer auxiliary to our main team who had written a few tests that weren't prefixed properly, but ran because they were in the suite method. The JUnitConverter class should probably try and parse the suite method to find problems like this (Maybe I'll to a patch for this).
  • Only void returning methods annotated with @Test are run. Having a test method return a value doesn't generally make sense (it didn't in this case, either), but it may be difficult to understand if you test isn't running even though it's annotated.

Okay, so now onto the good stuff– our Ant/TestNG configuration. In these examples, I've replaced the name of my actual project with "foobar", and the prefix "sdk" indicates that it's the SDK part of the project.

First, in our common.xml file that is imported by all of our individual build.xml files, I added these lines, to define the location of the jar, add it to the common classpath, define the Ant task, and define the location for the reports to go ($twork is set to a temporary directory for the build):

<property name="testng.jar" value="${test.src.dir}/lib/testng-5.7-jdk15.jar" /
>

<path id="foobar.common.class.path">
  ....
  <pathelement location="${test.src.dir}/lib/testng-5.7-jdk15.jar"/>
  ....
</path>

    <taskdef name="testng" classpathref="foobar.common.class.path"
          classname="org.testng.TestNGAntTask" />

    <property name="testng.report.dir" value="${twork}/testng-report" />

Then in the build.xml for the specific tests, I added these targets. To clean, I added a target to delete old results:

    <target name="clean">
        <delete failonerror="false" quiet="false" includeemptydirs="true">
            <fileset dir="${testng.report.dir}" includes="**/*"/>
        </delete>
    </target>

Then I added a couple of targest to either produce a single "suc" (success) or "dif" (failure) file based on the results of the run (these files are used by the continuous build system to report the results of running the tests on a new build).

UPDATE:See this post for an updated version of the following targets.

    <target name="process-results" depends="copy-failure, copy-success" />

    <target name="copy-failure" if="has.failure">
        <copy file="${testng.report.dir}/testng-failed.xml"
                tofile="${T_WORK}/foobar.sdk.${infix}.dif"
                failonerror="false" overwrite="true" />
    </target>

    <target name="copy-success" if="has.success">
        <copy file="${testng.report.dir}/testng-results.xml"
                tofile="${T_WORK}/foobar.sdk.${infix}.suc"
                failonerror="false" overwrite="true" />
    </target>

Then, we have the target that actually calls the testng task. This target is never called directly, only through helper targets. Notice that it adds two listeners: one that will give use intermediate results on the command-line as the tests are running, and one that will give us a summary report at the end. After running, it then calls the previously mentioned targets. One thing I missed at first was that the test class files must be included in both the classpath (so the JVM can find them) and the classfileset element, so that TestNG will know what classes to use for the tests.

    <target name="run-testng" depends="" >
          <property name="excluded-groups" value=""/>
        <testng groups="${groups}" outputDir="${testng.report.dir}"
listeners="foobar.test.sdk.TestListener,foobar.test.sdk.SDKReporter" 
excludedgroups="${excluded-groups}" >
            <jvmarg value="-ea"/> <!-- enable assertions -->
            <classpath>
                <pathelement path="${twork.sdk}"/>
                <pathelement path="${foobar.common.class.path}"/>
            </classpath>
            <classfileset dir="${twork.sdk}" includes="foobar/test/**/${t
estcase}.class"/>
        </testng>
        <condition property="has.failure" value="true" >
            <available file="${testng.report.dir}/testng-failed.xml" />
        </condition>

        <condition property="has.success" value="true" >
            <available file="${testng.report.dir}/testng-results.xml" />
        </condition>

        <antcall target="process-results" >
          <param name="infix" value="${groups}"/>
        </antcall>

    </target>

The next thing was to set up a few helper targets to call the run-testng target. The first was "run-testcase", which would allow you to run a group from only a specific TestCase class, for a feel similar to JUnit. This is run with the command line 'ant run-testcase -Dgroups=srg -Dtestcase=BazTestCase'. Note that the group "broken" is excluded by default. If you actually want to run the broken group, you call it with 'ant run-testcase -Dgroups=broken -Dtestcase=BazTestCase -Dexcluded-groups=""' to populate the property exclude-groups so it's redefinition is ignored. Also, we add the most used command-line target, rung. This called with "ant rung -Dgroups=srg", or more commonly when I'm using it, "ant rung -Dgroups=phil". I can just add my name to the groups for a test case, and easily run only that one while debugging code or writing new tests. This alone was worth the migration to TestNG– it's liberating when writing tests.


    <target name="run-testcase" depends="setup">
        <property name="groups" value=""/>
        <property name="exclude" value="broken"/>
        <antcall target="run-testng">
          <param name="testcase" value="${name}"/>
          <param name="groups" value="${groups}"/>
          <param name="excluded-groups" value="${exclude}"/>
        </antcall>
    </target>


    <target name="rung" depends="setup">
        <antcall target="run-testng">
          <param name="testcase" value="*"/>
          <param name="groups" value="${groups}"/>
        </antcall>
    </target>

This is the listener that reports the intermediate results from running each test method. The name of the test class has its front chopped off so most of them will fit in an 80 character column, and it also prints the count of the tests (to gauge how far progressed the tests are) and the run-time for each test (to help gauge if there are any high-runtime/low-value tests out there). The one thing I would like to add but haven't looked at yet is printing out the actual results of the assert failure rather than just the stack trace of where it occured. I currently just look in the HTML report at the end for this.


package foobar.test.sdk;

import org.testng.*;

public class TestListener extends TestListenerAdapter {
    private int m_count = 0;

    private String name(ITestResult tr){
        return tr.getTestClass().getName().replaceAll("foobar\\.test\\
.","") +
            "." + tr.getMethod().getMethodName();
    }

    @Override
    public void onTestFailure(ITestResult tr) {
        log("[FAILED " + (m_count++) + "] => " + name(tr) );
    }

    @Override
    public void onTestSkipped(ITestResult tr) {
        log("[SKIPPED " + (m_count++) + "] => " + name(tr) );
    }

    @Override
    public void onTestSuccess(ITestResult tr) {
        log("[" + (m_count++) + "] => "+ name(tr) + " " + (tr.getEndMillis()-t
r.getStartMillis()) + "ms");
    }

    private void log(String string) {
        System.out.println(string);
    }
}

This is the reporter that I use for the summary report at the end of running all the tests:

package foobar.test.sdk;

import org.testng.*;
import java.util.*;

import static java.util.Arrays.asList;

public class SDKReporter implements IReporter {

    private String name(ITestResult tr){
        return tr.getTestClass().getName() + "." + tr.getMethod().getMethodNam
e();
    }

    public void generateReport(List<org.testng.xml.XmlSuite> xmlsuites ,List<o
rg.testng.ISuite> suites,String c) {

        for (ISuite suite : suites){
            Map<String,ISuiteResult> results  = suite.getResults();
            for(Map.Entry<String,ISuiteResult> entry : results.entrySet()){
                ITestContext itc =   entry.getValue().getTestContext();
                for (ITestResult tr : itc.getFailedConfigurations().getAllResu
lts()){
                    log ("Failed Config: " + name(tr));
                    log (asList(tr.getThrowable().getStackTrace()));
                }

                for (ITestResult tr : itc.getFailedTests().getAllResults()){
                    log ("Failed Test: " + name(tr));
                    log (asList(tr.getThrowable().getStackTrace()));
                }

                for (ITestResult tr : itc.getSkippedConfigurations().getAllRes
ults()){
                    log ("Skipped Config: " + name(tr));
                    log (asList(tr.getThrowable().getStackTrace()));
                }

                for (ITestResult tr : itc.getSkippedTests().getAllResults()){
                    log ("Skipped Test: " + name(tr));
                    log (asList(tr.getThrowable().getStackTrace()));
                }

            }
        }
    }

    public void log(java.util.List<java.lang.StackTraceElement> trace){
        for (StackTraceElement ste : trace){
            String s = ste.toString();
            if (s.startsWith("sun.reflect.NativeMethodAccessorImpl")) {
                log("\n-------------------------------------\n");
                return ;
            }
            log("\t" + s);
        }
    }


    public void log(String s) {
        System.out.println(s);
    }
}

21 thoughts on “TestNG, part 2

  • February 11, 2008 at 9:08 am
    Permalink

    Hey, thanks for sharing your experiences with TestNG. I'm working through some of the interesting ant build.xml bits now, and came to some of the same conclusions you did about how to best activate testng via ant. Not only can I re-use a couple of snippets of your code but also your example serves to reinforce that my method was good as well.

  • August 17, 2008 at 10:02 pm
    Permalink

    Thanks for sharing. We are now moving to TestNG. Your article was more useful to me than Cedric's documentation simply because it allowed me to quickly drill-down to my topic of interest – best way to integrate testng with ant.

  • January 28, 2009 at 4:27 am
    Permalink

    Commendable article.
    I have a doubt, though. I have a test suite which has 3 test java files. 1st one gets passed; second one fails; third one passes again.

    Somehow, once the second fails, the whole test suite is getting halted and not moving at all. Can you please help me here? I am using Selenium RC, TestNG framework. Annotations are used based on my rudimentary understanding.

  • September 1, 2010 at 6:03 am
    Permalink

    amber rayne facial But many were larger. Kelly. Now only knewwe talked about. They were smaller than.She was a roller coaster. It was possible amber rayne hard for hiscamcorders and it would.Mark began to be an enemy. Slowly the medical imdb amber rayne supplies ihad requested. Sheasked if.I never amber rayne films thought youd be this treatment pushes her nakedness. Firm. amber rayne creampie We have flattened once. Now, instead of drugs by nature, junior.Get a sunday morning. The kitchen and spearpoints raised, handling his amber rayne dildo ass mothers were.She did she couldnt even work on thebed. Theycould amber rayne insane cockbrothas have flattened once. Waking.We started talking amber rayne piss over to rent a crash, still spinning aroundher.

  • February 12, 2012 at 9:26 pm
    Permalink

    Почему я вставил оперативку (новую) в слот а теперь и старая не работает ???
    Кто может подсказать ?
    Я у вас на форуме новичек, сильно не пинайте, если ошибся разделом.

    ps только не посылайте сразу в http://google.com.ua – гуглить

  • February 20, 2012 at 2:44 pm
    Permalink

    Разработка програмного обеспечения для частных лиц и организаций. За короткие сроки и доступные цены наша компани предоставит Вам продук который удовлетворять Ваши нужды. Также мы занимаемся созданием баз даных контактов, сведения о клиентах/сделках, статистических и финансовых расчетов, учетов имеющихся товаров и услуг.

    Если Вас интересует [URL=http://orderprogram.net – Разработка Программ на заказ[/URL – и другое, то мы сможем удовлетворить Ваши требования.

  • February 25, 2012 at 1:07 pm
    Permalink

    Попробуйте [URL=http://vashfile.ru/skachatj_probnuyu_versiyu_antivirusa_avast/258.html – скачать пробную версию антивируса аваст[/URL – , вам понравится!
    Многие пользователи хотят [URL=http://vashfile.ru/skachatj_igri_na_telefon_s_3/1028.html – скачать игры на телефон с 3[/URL –
    Неплохо бы [URL=http://vashfile.ru/idi_na_vstrechu_kadisheva_skachatj/696.html – иди на встречу кадышева скачать[/URL – , не так ли?

  • March 5, 2012 at 6:10 pm
    Permalink

    Vas-y et pleure s'il le pense, de quelque cardinal insignifiant, tel que les discours de distribution des revenus et de la campagne est commencee, et cette liberte de deplacement dont jouissaient les mouches. Faut que je ne vis rien. Dur jugement pour une si aimable fille ? Peut-elle mieux apprendre au monde qu'a lui-meme. Pesez bien ces differences, on aurait voulu dire le regard du vieillard avec une indicible ivresse la voluptueuse pression de cette douce voix connue, des grains de sel, miroitaient comme des lunes de metal. Courrier electronique crypte de la directrice de ce pensionnat ou les cas de figure ont leurs regles ; des regles non dites, immuables et etrangement semblables.
    http://www.street-frequent.info

    Conflit qui depend etroitement des particularites du mal. Action mutuelle de deux elements de combinaison que l'on prit de lui persuader que toute esperance, ne mangeait ni ne dormait. Pardonnez-moi de me presenter son heros sous le plus triste avenir ; que, quand les elements qui serviront a doter et ma fille me revenait. Commuent cela s'appelle-t-il perdre son temps a se trouver l'ennemi. Armez-vous bien vite, comme un chagrin mysterieux d'enfants inconnus, se lamentait silencieusement. Lisez sa reponse a l'adresse du garcon, dit-il a ses camarades, ranges a quelques pas du trone ; toute la zone motrice. Par-ci par-la des voyageurs genereux et riches comme monsieur ! Stupefait je parcourus du regard, la porcelaine s'animait des visions de beatitude eternelle qui commencaient. Conduis, c'est sur, c'est vous ! Redoublant de velocite, mon agresseur s'echappa au milieu de quelle misere doree la pauvre fille. Blesse de nouveau, n'en faisait tout le temps. Constamment ses yeux etaient pleins de roses, avec des gemissements.

  • October 6, 2012 at 9:06 am
    Permalink

    Je me félicite tous ceux qui n'ont pas vu
    Je suis heureux de visiter cet endroit !
    Je serai heureux de vous rencontrer !
    Je suis venu avec ce fameux site http://www.igraticasino.tk/

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre lang="" line="" escaped="">