DevFaqUsingSimpletests
How do I test a 6.5M2 or later platform-based application?
NOTE: This is a work in progress and will be expanded/improved as I experiment more with the new testing setup. Correctness is more important than being thorough.
Background
In versions of NetBeans prior to 6.5M2, comprehensive testing of platform-based applications was done using the XTest Framework. XTest added a number of build scripts, utilities and testing libraries plus some configuration to categorize and group tests together. The framework was originally developed for testing the IDE itself, and although using it to test other platform-based applications was possible, it was somewhat difficult to set up these tests.
During development of the 6.5 series, an effort was made to improve testing and XTest was replaced by inherent support for tests in NetBeans build scripts (also known as the build harness). Therefore, unit and functional tests for platform applications are now supported out of the box in NetBeans 6.5M2 and later. This effort is sometimes called "simpletests" because it greatly simplifies the work necessary to do such tests on your application. More information about simpletests can be found in the Wiki.
JUnit Version
While XTest supported only JUnit 3.x, NetBeans 6.5 now supports JUnit 4.x, via the JUnit 4 module in the platform9 cluster. If you're not familiar with the difference, this helpful article explains the changes from JUnit 3 and JUnit 4. Put simply, the main difference from a user's point of view is is one of syntax and style. The tests you've already written against 3.x will continue to run under 4.x. And although you could continue writing new tests using the 3.x syntax if you wanted, it's much easier to have NetBeans generate the test stubs for you (Tools -> Create JUnit Tests or Ctrl+Shift+U). Starting with NetBeans 6.5, any new tests will be generated using the JUnit 4.x style.
Converting Tests for Applications Which Previously Used XTest
Essentially, you need only remove the XTest build and configuration files from your modules and replace them with a few lines of code in your tests' suite() method. More information is available in the XTest Replacement Cookbook
Learn by example
Before: See PaintApplicationWithXTest_DevFaqUsingSimpletests.zip (Using XTest) works only with NB 6.5M1 and earlier and requires XTest After: See PaintApplicationWithSimpletests_DevFaqUsingSimpletests.zip (Ported to use simpletests); works only with NB 6.5M2
Code Coverage
See Code Coverage.
Setting up functional tests for a Platform Application
The testing libraries are included as modules in the build harness, so you will need to include the harness cluster in your application before you can support tests. This is easily done through the IDE:
- Right click your suite's node in the project view
- Choose "Properties"
- Click "Libraries" on the left
- Check the "harness" box and add the entire harness cluster.
- Now click the platform n cluster (currently platform 10)
- Check the boxes against the MultiView Windows and Visual Library API.
Now you must set up the structure under your module:
- Go to the 'file' view, and expand your modules node
- In the test folder, create a folder named 'qa-functional'. Underneate that, create the folder src.
- Under the test dir you should now have two folders, unit and qa-functional. Under each of those should be an src folder.
- Restart the IDE, there should now be two new Nodes in that Module’s Project Tab: Functional Test Packages and Functional Test Libraries
- Right-click the Functional Test Libraries –> Add Functional Test Dependencies, then add jemmy, nbjunit, jellytools platform, jellytools and junit4
It should now be possible to run a class that extends JellyTestCase, and for the IDE to display this correctly.
TODO
- Describe how to set up a new unit test from scratch
- Describe how to set up a new functional test from scratch
- Describe adding support for external testing libraries (e.g. FIT, TestNG or marathon)
- Describe how to set up additional test types (e.g. performance)
- Describe how to configure suite build.xml to remove harness from ZIP distribution
- Describe how to run tests under automated build using Hudson
NOTES
MockLookup and other classes mentioned on the Useful Test Classes in Modules are not available in the platform.
- You can add additional libraries needed for unit tests by setting the test.unit.cp.extra property (e.g. in the module's project.properties file) to point to those libraries. It does not matter where these libraries (JAR files) reside, and you can refer to them using a hardcoded path (BAD) or via a relative path -- even using $suite.dir as a starting point in case they're used by multiple modules in the suite. In the latter case, you can define the test.unit.cp.extra property in the suite's platform.properties (NOT project.properties) file. Be aware that this may limit your ability to define additional things at the individual module level, so it might be better to define a standard suite-wide property like standard.unit.test.libs to point to things used throughout the suite, then have individual modules define test.unit.cp.extra to include this plus potentially some other things. Items defined in the test.unit.cp.extra property are used at both compile time and runtime, it seems.
- I'd guess that the test.unit.functional.extra (or maybe test.unit.qa-functional.extra) property works the same way for functional tests, but I have not tried it. It did work under XTest though.
- There is a property extra.test.libs.dir tersely described in the harness README that makes it sound like a great way to define a place for extra testing libraries that will be picked up automatically during tests, but I could not get it to work. Or at least things in that directory were not found at compile time.
