Generic Support for Unit Tests in the NetBeans IDE

Overview

The main goal of the generic support for unit testing is to provide an API that would allow to easily add support for various specific types of unit test frameworks (JUnit, PHPUnit, J2MEUnit, Ruby tests, …) into various project types (J2SE projects, NB modules, WebApps, C/C++, Ruby, …) into the IDE.

The most important changes from the current state:

• support for unit testing is extracted from the core projects infrastructure to a separate module (generic support for unit tests)
• it is not required for a project type to provide support for any particular unit test framework
• each project type may be extended with a support for one or more unit testing frameworks by external modules

One external module can add support for multiple specific testing frameworks (but will usually add support for only one).

It is not necessary for one external module to add support to all project types – it can add support only to a small number of project types, while another module adds support for the same testing framework to other project types.

Motivation (Problems To Be Solved)

So far, the NetBeans support for unit testing is spread across several modules as follows:

• the JUnit module can create simple JUnit tests in Java projects
  – but it does not know how to create J2MEUnit tests for J2ME projects.
  There is an unofficial (friend) API for this – JUnitPlugin.java
• the JUnit module knows how to navigate between a source class and the corresponding test class
  – but it does not know how to navigate between Ruby source code and the corresponding tests
  There is another unofficial (friend) API for this, in a separate module ("Navigate To Test", directory "gototest")
• the JUnit module knows how to parse output that is generated when JUnit tests are executed via Ant (JUnitAntLogger, JUnitOutputReader)
  – but it does not know how to parse and display Ruby unit tests, TestNG tests, Maven tests, …
  There is no API for this, either official or unofficial; the respective modules simply copy the code from the JUnit module and accomodate it according to their needs. Some 80 % of the code remains unchanged, but in several copies.
• the JUnit module does not initiate execution of tests (it is handled by the projects infrastructure and its actions) so it does not know that tests have been started until it detects some output from Ant's <junit> task

One of the consequences is that every time the test execution mechanism needs to be changed (e.g. make Ant-based projects generate XML report files), the developer who wants this change must make the change not only in the JUnit module, but also in modules defining web projects, NetBeans module projects, several sub-types of J2SE projects, J2ME projects, etc. For each type of project, the change requires a change of a (typically) quite large build file and possibly also some changes in the Java code. As more and more project types are supported, this is becoming a maintenance nightmare.

Goals

These are the expected results of the transition to a generic support for unit tests:

• pluggable support for unit testing frameworks
• shared actions for creation and running of unit tests and for navigation between tests and the tested code
• shared implementation of a test results window with a public API
  (this may be, and probably will be, done later)

Architecture

In the description below, the term "testing framework" refers to the generic unit testing framework being described by this document.

Module Projects UI does not know anything about unit testing. Module Generic Unit Testing Framework provides all the tests-related actions currently provided by module Projects UI. It depends on the projects Modules providing support for project types may not know anything about unit testing either.

Features provided by the testing framework:

• tests-related actions currently provided by module Projects UI
• tests-related actions currently provided by the JUnit module (Go to Test/Tested Class, Create Unit Tests, etc.)
• all the tests-related Ant targets in projects' build.xml files
• API for registration of modules providing support for individual unit testing frameworks (JUnit, TestNG, etc.)
• API for queries about test roots for various individual testing frameworks
  (e.g. Give me test roots for TestNG tests in this project.)
• API for incremental display of test results (in future)

API changes required in the projects infrastructure:

• It must be possible to query a project about its type, such that the plugin providing support for a specific testign framework knows whether it should plug into it or not.

SPI of modules providing support for individual unit testing frameworks

• provide information about test roots, or even create the test roots (TBD)

Open Issues

The description so far only assumed that all projects are Ant-based. But this was a wrong assumption. So we should either make it more general (such that it works with projects not based not Ant, for example with C/C++ projects) or declare that it is only for Ant-based projects.

Other Related Changes

Change of Shortcuts

Shortcuts for execution of unit tests should be changed as follows: