TestNG MixedMode

Revision as of 16:30, 23 February 2012 by Jungi (Talk | contribs)

Author: Lukas Jungmann (jungi (at) netbeans (dot) org)
Date: February 6, 2012

Contents

Status

2/6/2012 - Announced on testng-dev mailing list read
2/20/2012 - Changes accepted and included in standard TestNG distribution Image:Yes.png

Motivation

TestNG is getting more and more interest from developers due to its specific features which are missing in JUnit, such as ability to define dependencies between tests, define test groups, have parametrized test, parallel test execution and many others. It also provides a way to run JUnit tests. But as of version 6.3.1 this feature is limited only to JUnit 3 tests [read more] and even tough there are already tools helping developers to migrate from JUnit to TestNG the migration itself can be a bit painful and time consuming. This mainly applies for big projects which are not being developed using Eclipse, for example NetBeans.

Solution

The idea is instead of providing a JUnit test 3/4 conversion tool to improve TestNG itself so it can automaticaly recognize JUnit 3 and JUnit 4 tests and run them if a user asks TestNG to do so and puts JUnit library on TestNG's classpath.

This is achieved through new TestNG option called -mixed. This option tells TestNG that it should also check files passed to it for presence of JUnit tests - be it a JUnit 3 test with suite method or a regular TestCase with test* methods or JUnit 4 test annotated with org.junit.Test annotation. The only requirement here is that TestNG is able to find JUnit library on its test classpath.

With -mixed option on, TestNG internally creates and runs following test suite based on user input for -testclass or -methods options (or their Ant task's counterparts):

  • xml suite created from -testclass option:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Command line suite">
  <test name="test.mixed.JUnit3Test1" junit="true" preserve-order="false">
    <classes>
      <class name="test.mixed.JUnit3Test1"/>
    </classes>
  </test> <!-- test.mixed.JUnit3Test1 -->
  <test name="test.mixed.JUnit4Test1" junit="true" preserve-order="false">
    <classes>
      <class name="test.mixed.JUnit4Test1"/>
    </classes>
  </test> <!-- test.mixed.JUnit4Test1 -->
  <test name="Command line test" preserve-order="false">
    <classes>
      <class name="test.mixed.TestNGTest1"/>
    </classes>
  </test> <!-- Command line test -->
</suite> <!-- Command line suite -->
  • xml suite created from -methods option:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Command line suite">
  <test name="Command line test" preserve-order="false">
    <classes>
      <class name="test.mixed.TestNGTest1">
        <methods>
          <include name="tngCustomTest1"/>
        </methods>
      </class> <!-- test.mixed.TestNGTest1 -->
    </classes>
  </test> <!-- Command line test -->
  <test name="test.mixed.JUnit4Test1" junit="true" preserve-order="false">
    <classes>
      <class name="test.mixed.JUnit4Test1">
        <methods>
          <include name="atest"/>
        </methods>
      </class> <!-- test.mixed.JUnit4Test1 -->
    </classes>
  </test> <!-- test.mixed.JUnit4Test1 -->
  <test name="test.mixed.JUnit3Test1" junit="true" preserve-order="false">
    <classes>
      <class name="test.mixed.JUnit3Test1">
        <methods>
          <include name="testB"/>
        </methods>
      </class> <!-- test.mixed.JUnit3Test1 -->
    </classes>
  </test> <!-- test.mixed.JUnit3Test1 -->
</suite> <!-- Command line suite -->

This solution affects curent -junit option if JUnit 4 library is found on tests' classpath - if it is found then org.junit.runner.JUnitCore is used for running all found JUnit tests. Other than that current behaviour of TestNG with changes described in this documment is not affected.

Note that using -mixed together with -junit is not permitted - TestNG will not start if both options are set.


Example: Migrating standard NetBeans Java SE project from JUnit to TestNG

Assume we have created an Ant based Java SE project in NetBeans and we have there some existing JUnit tests - version 3 or 4, that is not important at this point - and we decided to switch our project from JUnit to TestNG. How can we do that?

Basically we have 3 options:

  • rewrite existing tests to TestNG and update our build script(s)
  • use a JUnit -> TestNG conversion tool and update our build script(s)
  • update our build script(s) to call TestNG in new mixed mode - this is what TestNG plugin for NetBeans does

So let's choose the last option as it looks like the easiest one:

1) Put TestNG binary on project's run.test.classpath (IDE will do this for you when you add TestNG library to project's Test Libraries, otherwise do that in nbproject/project.properties).

2) Open nbproject/build-impl.xml file in some editor and find following snippet there:

    <target name="-init-macrodef-junit">
        <macrodef name="junit" uri="http://www.netbeans.org/ns/j2se-project/3">
            <attribute default="${includes}" name="includes"/>
            <attribute default="${excludes}" name="excludes"/>
            <attribute default="**" name="testincludes"/>
            <sequential>
                <property name="junit.forkmode" value="perTest"/>
                <junit dir="${work.dir}" errorproperty="tests.failed" failureproperty="tests.failed" fork="true" forkmode="${junit.forkmode}" showoutput="true" tempdir="${build.dir}">
                    <batchtest todir="${build.test.results.dir}">
                        <fileset dir="${test.src.dir}" excludes="@{excludes},${excludes}" includes="@{includes}">
                            <filename name="@{testincludes}"/>
                        </fileset>
                    </batchtest>
                    <classpath>
                        <path path="${run.test.classpath}"/>
                    </classpath>
                    <syspropertyset>
                        <propertyref prefix="test-sys-prop."/>
                        <mapper from="test-sys-prop.*" to="*" type="glob"/>
                    </syspropertyset>
                    <formatter type="brief" usefile="false"/>
                    <formatter type="xml"/>
                    <jvmarg line="${endorsed.classpath.cmd.line.arg}"/>
                    <jvmarg value="-ea"/>
                    <jvmarg line="${run.jvmargs}"/>
                </junit>
            </sequential>
        </macrodef>
    </target>

3) Now replace it with following code

    <target name="-init-macrodef-junit">
        <macrodef name="junit" uri="http://www.netbeans.org/ns/j2se-project/3">
            <attribute default="${includes}" name="includes"/>
            <attribute default="${excludes}" name="excludes"/>
            <attribute default="**" name="testincludes"/>
            <sequential>
                <fileset id="mixed.tests" dir="${test.src.dir}" excludes="@{excludes},${excludes}" includes="@{includes}">
                    <filename name="@{testincludes}"/>
                </fileset>
                <taskdef name="testng" classname="org.testng.TestNGAntTask" classpath="${run.test.classpath}"/>
                <testng mode="MIXED" classfilesetref="mixed.tests" workingDir="${basedir}" failureProperty="tests.failed" outputdir="${build.test.results.dir}">
                    <classpath>
                        <pathelement path="${run.test.classpath}"/>
                        <pathelement path="${j2ee.platform.classpath}"/>
                        <pathelement path="${build.test.classes.dir}"/>
                    </classpath>
                    <propertyset>
                        <propertyref prefix="test-sys-prop."/>
                        <mapper from="test-sys-prop.*" to="*" type="glob"/>
                    </propertyset>
                    <jvmarg line="${run.jvmargs.prop}"/>
                    <jvmarg line="${endorsed.classpath.cmd.line.arg}"/>
                </testng>
            </sequential>
        </macrodef>
    </target>

4) And as the last thing - create some simple TestNG test, Test the project and see what happens:

compile-test:
[TestNG] Running:
  Ant suite

JUnit 4
PASSED: hello on hello(test.JUnitTest)

===============================================
    test.JUnitTest
    Tests run: 1, Failures: 0, Skips: 0
===============================================

TestNG
PASSED: aTest

===============================================
    Ant test
    Tests run: 1, Failures: 0, Skips: 0
===============================================


===============================================
Ant suite
Total tests run: 2, Failures: 0, Skips: 0
===============================================

[TestNG] Time taken by org.testng.reporters.jq.Main@67006d75: 43 ms
[TestNG] Time taken by org.testng.reporters.JUnitReportReporter@27ce2dd4: 6 ms
[TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@aaf8358: 8 ms
[TestNG] Time taken by org.testng.reporters.EmailableReporter@10b61fd1: 1 ms
[TestNG] Time taken by [TestListenerAdapter] Passed:0 Failed:0 Skipped:0]: 0 ms
[TestNG] Time taken by org.testng.reporters.XMLReporter@33c1b02: 3 ms
test-report:
test:
BUILD SUCCESSFUL (total time: 0 seconds)


Now we can test the project and all JUnit tests as well as new TestNG tests will run both at the same time using TestNG.

Known issues

  • posible ClassNotFountError from org.testng.internal.ClassHelper - this class in IJUnitTestRunner createTestRunner(TestRunner runner) method instantiates classes depending on JUnit binary and if it is not found Error is thrown, to avoid/fix this 'Class.forName("Test/TestCase")' needs to be called before '(IJUnitTestRunner) ClassHelper.forName(JUNIT_TESTRUNNER).newInstance()' - this will allow TestNG to recover from the state when "-junit" (or "-mixed") is used and no JUnit is found on the classpath - FIXED (44c68b0)
  • abstract class with suite method is not recognized as valid JUnit test

ChangeLog

This part discribes all differences between official TestNG distribution (as of v. 6.4beta from February 2, 2012) and modified TestNG library included as part of TestNG plugin for NetBeans

Full diff between both repositories: testng-6.4beta.diff.zip

Particular changes:

  • ad72beb - unrelated - a fix for building TestNG distribution to contain additional artifacts in a common location
  • 89f3b06 - update JUnit library used for building TestNG to JUnit 4 and support for running JUnit 4 tests - this affects current junit="true" option in a way that it is able to run also JUnit 4 tests
  • f5c4b35 - provide two versions of testng.jar - one with rebundled JUnit classes and second without JUnit classes
  • 1522307 - fixing compilation errors after pulling from official repository
  • 71126c8 - initial checking of mixed mode
  • f258209 - separation of JUnit3 and JUnit4 runners and improved test results reporting
  • 1876355 - previous 2 changesets (71126c8 and f258209) merged into one for better review
  • bf04c20 - (empty) merge
  • 543fc9c - a typo
  • 77de21b - making sure that TestNG runs with JUnit on its classpath as well as without it without any errors
  • efa11aa - merge with official repo as of February 2, 2012
  • 2e805d9 - fixing compilation errors after pulling from official repository
  • 44c68b0 - avoid ClassNotFoundError (as mentioned under #Known issues)
Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo