TS 72 UnitTesting

JUnit Test Specification

Author: Martin Skurla
Last update: 5.6.2012
Introduction: This document contains test specifications for JUnit and TestNG modules.
Comments: JUnit module is an extension to JUnit framework and TestNG module is an extension to TestNG modules. Both serve in NetBeans IDE as support for creating unit tests of individual classes or whole packages. They generate skeletons of those unit tests and allows to run tests directly from tested classes or packages.
Note: This Specification describes testing of J2SE, J2ME, J2EE and Maven projects. It is recommended to use the Full IDE distribution with all modules.
Installation note: JUnit module will be installed as part of the first JUnit usage. TestNG Ant and Maven modules are installed as part of NetBeans installation.
Sample project to work with: TS_68_JUnit/JUnitSampleProject_TS_68_JUnit.zip

Contents


Test suite: Test libraries

Purpose: Verify proper usage of Test Libraries in different projects
Setup: Start IDE with clear userdir.

  1. "J2SE Project"
    1. Create new Java Application project with default setting
    2. Select main class (e.g. JavaApplication1.java) node in Project tab and try to create test for this class (CTRL + SHIFT + U [or Command + SHIFT+ U on OSX])
    • EXPECTED RESULT: JUnit version support was not set yet, so you can see JUnit version chooser. Now you are able to choose JUnit 4.x or JUnit3.x support.
    1. Press Cancel
    • EXPECTED RESULT: Test Libraries folder appears, but it is empty.
    1. Select JavaApplication1.java node again and select 'Tools > Create JUnit Tests' from the content menu
    2. Select JUnit4, and press Select
    • EXPECTED RESULT: Dialog appears prompting download of JUnit.
    1. Click Download and Install JUnit. Complete the steps in the Wizard to download and install the JUnit plugin.
    • EXPECTED RESULT: Create Tests dialog appears.
    1. Press Cancel
    • EXPECTED RESULT: Test Libraries folder now contains JUnit 4.
    1. Try to create JUnit test for JavaApplication1.java again
    2. Leave all settings unchanged and press the OK button
    • EXPECTED RESULT: Test class was created. There was no "Select JUnit Version" dialog.
    1. Delete 'JavaApplication1Test.java'
    2. Remove the JUnit 4 test library from Test Libraries
    3. Try to create JUnit test for JavaApplication1.java again
    • EXPECTED RESULT: The version select dialog appears again. If you choose on of the versions and create new test, corresponding test library is added to Test Libraries folder.
    1. Delete everything in Test Packages folder
    2. Select Test Libraries and select 'Add Library...' from its content menu
    3. Add the JUnit library that is currently missing (it depends on what version you have selected in previous step), so both JUnit libraries are present
    4. Open project's properties (right-click on the project's root node and select 'Properties')
    5. In category Sources change "Source Level" listbox value to "1.4" and press "OK" button
    6. Try to create JUnit test for JavaApplication1.java
    • EXPECTED RESULT: "Select JUnit Version" dialog appears again. However, you are not able to choose JUnit 4.x, because it requires Java 1.5+.
    1. Leave JUnit 3.x selected and create the test with default setting
    • EXPECTED RESULT: Test "JavaApplication1Test.java" is successfully created in the "Test Packages" and package with the same name as the package containing tested file. JUnit4.x library was automatically removed from the Test Libraries folder
  2. "Java Web Application"
    1. Create new Web Application (File - New Project - Java Web - Web Application) with default setting
    2. Create new package "newpackage" in Source Packages folder
    3. In this package create new Java Class "NewClass.java"
    4. Add some method to this class, e.g. public void foo() {}
    • Note: Tested class must have at least one testable method, otherwise the test cannot be created
    1. Select NewClass.java node and select 'Tools > Create JUnit Tests' from the content menu
    2. Choose one of the JUnit versions and create a test with default setting
    • EXPECTED RESULT: Test was successfully created. Test Libraries folder is now visible, with the appropriate JUnit version in it
    1. Verify that libraries behave the same way as in "J2SE Project" testcase (try removing/adding libraries and changing source level of the project)
    • EXPECTED RESULT: Results are same as in the previous testcase
  3. "Mobile Application"
    • Note: Mobile Applications use JMUnit support for creating and running unit tests
    1. Create new Mobile Application (File - New Project - Java ME - Mobile Application) with default setting
    2. Expand Resources folder
    • EXPECTED RESULT: There is "HelloMIDlet.java" in "hello" package. Recources folder is empty by now
    1. In "hello" package create new Java Class "NewClass.java"
    2. Add some method to this class, e.g. public void foo() {}
    3. Create new test for this class (either by pressing Ctrl + Shift + U or selecting 'Tools > Create JUnit Tests')
    • EXPECTED RESULT: Test "NewClassTest.java" is successfully created in "hello" package. JMUnit4CLDC10 library was automatically added to the "Resources" folder
    1. Right-click on "Resources" folder and select 'Add Library'
    2. Select JMUnit4CLDC11 from the list and press 'Add Library' button
    3. Delete "NewClassTest.java"
    4. Create new test for "NewClass.java" (leave all settings unchanged)
    • EXPECTED RESULT: Test "NewClassTest.java" is successfully created. You are NOT prompted to choose between the test libraries. JMUnit4CLDC10 is used by default.


Test suite: Test and Testsuite files

Purpose: Test verifies automatically generated code in unit tests and test suites
Setup: Start IDE with clear userdir. Download and open TS_68_JUnit/JUnitSampleProject_TS_68_JUnit.zip

  1. "Test files"
    1. Select "classtypes" package and press Crtl+Shift+U
    2. Ensure that all checkboxes in the "Create Test" dialog are checked
    3. Choose JUnit
    4. Select JUnit version
    5. Press Select
    • Note: Go through all the tests from this suite with the same JUnit version and then repeat it with the other version
    • JUnit EXPECTED RESULT: Package "classtypes" is created in the "Test Packages" folder. There are four files present: three tests and one suite. Test filenames are "[TestedClassName]Test.java", suite filename is "[PackageName]Suite.java". Files are created without errors.
  2. "Test files without suite"
    1. Delete the package in "Test Packages"
    2. Select "classtypes" (in "Source Packages") and invoke 'Tools > Create Tests'
    3. Uncheck 'Generate Test Suites' checkbox. Other checkboxes should be checked
    4. Press OK
    • EXPECTED RESULT: Package "classtypes" is created in the "Test Packages" folder. There are three test files present. There is no suite file. Files are created without errors.
  3. "Class Types"
    1. Delete the package in "Test Packages"
    2. Select "classtypes" (in "Source Packages") and invoke 'Tools > Create Tests'
    3. Uncheck some checkboxes under 'Class Types' (Package Private Classes, Abstract Classes, Exception Classes)
    4. Press OK
    • EXPECTED RESULT: Test for appropriate class type is NOT created. If you uncheck all three checkboxes, no test is created.
  4. "Method Access Level"
    1. Select "Methods.java" in "methodaccess" package and open it in editor
    2. Invoke 'Tools > Create Tests' from the main menu
    3. Uncheck some checkboxes under 'Method Access Level' (Public, Protected, Package Private)
    4. Press OK
    • EXPECTED RESULT: Test file "MethodsTest.java" is created and opened in editor. Only tests for methods with selected access level are created (e.g. "testPublicMethod()" if 'Public' checkbox was checked). If you uncheck all checkboxes, placeholder method "testSomeMethod" will be created.
  5. "Generated Code"
    1. Delete the "MethodsTest.java" test file
    2. Select "Methods.java"
    3. Invoke 'Tools > Create Tests' from the main menu
    4. Uncheck the following checkboxes under 'Generated Code' (Test Initializer, Test Finalizer, Default Method Bodies)
    5. Check all other checkboxes (some were unchecked in previous steps)
    6. Press OK
    • EXPECTED RESULT: Test file "MethodsTest.java" is created and opened in editor. Test methods are empty, test initializer and test finalizer are not created.
      JUnit4.x note: Static methods setUpClass() and tearDownClass() are still present
  6. "Generated Comments"
    1. Delete the "MethodsTest.java" test file
    2. Select "Methods.java"
    3. Invoke 'Tools > Create Tests' from the main menu
    4. Uncheck all checkboxes under 'Generated Comments' (Javadoc Comments, Source Code Hints)
    5. Check all other checkboxes (some were unchecked in previous steps)
    6. Press OK
    • EXPECTED RESULT: Test file "MethodsTest.java" is created and opened in editor. Test methods are created without javadoc comments and without "//TODO" notes.
      JUnit4.x note: Annotations like @Test, @BeforeClass etc. are still present
  7. "Test names"
    1. Delete the "MethodsTest.java" test file
    2. Select "Methods.java" right click and select 'Tools -> Create Tests' to create JUnit test
    3. Change the class name to "methodaccess.MyMethodsTest"
    4. Change the class name to "mymethodaccess.MethodsTest"
    • EXPECTED RESULT: The same warning is displayed in both cases ("You will not be able to navigate..."). The test still can be created (OK button is enabled). In the second case test will not be compilable because of visibility modifiers.
  8. "Test names II"
    1. Change the class name to "methodaccess.MethodsTestXXX" (so the class name is not ending with "...Test")
    • EXPECTED RESULT: Different warning is displayed than in previous case ("The projects infrastructure..."). The test cannot be created (OK button is disabled)
  9. "Test Package"
    1. Close the "Create Tests" dialog
    2. Select project root node and select 'Properties' from its content menu
    3. In category Sources - Test Package Folders press Add Folder button
    4. Create new folder in your project directory named "test2" and select it (Note: you may have to create this folder externally)
    • EXPECTED RESULT: "test2" is added to Source Packages Folder on the same level as Test packages. Its label is set to "test2"
    1. Change the label to "My Test Package"
    2. Press OK button
    3. Try to create new test for some file or package
    • EXPECTED RESULT: "My Test Package" can be selected as 'Location:' in the "Create Tests" dialog
  10. "Create New JUnit Test"
    1. Select "Test Packages" and invoke 'File - New File (Ctrl+N)'
    2. Choose "Unit Tests - JUnit Test"
    3. press 'Next >'
    4. Type "mypackage" for 'Package:' and leave other setting unchanged
    5. Press Finish
    • EXPECTED RESULT: New test file with no test methods is created in "mypackage" package
  11. "Create New Test for Existing Class"
    1. Select "Test Packages" and select 'New > Other...' from its content menu
    2. Choose "Unit Tests - Test for Existing Class"
    3. press 'Next >'
    4. Use the Browse button to select some class from the project
    5. Press OK
    6. Uncheck some checkboxes (under Method Access Level, Generated Code, Generated Comments) and press Finish
    • EXPECTED RESULT: New test file "[TestedClassName]Test.java" is created. Package name corresponds to the package, where the tested class is. Source code of the test is generated according to the settings made in the last step.
  12. "Create New Test Suite"
    1. Select "Test Packages" and select 'New > Other...' from its content menu
    2. Choose "Unit Tests - Test Suite"
    3. press 'Next >'
    4. Choose "mypackage" from the 'Package:' drop down menu
    5. Uncheck some checkboxes (under Generated Code, Generated Comments) and press Finish
    • EXPECTED RESULT: New test suite is created in "mypackage" package. Source code of the suite is generated according to the settings made in the last step.


Test suite: Running and Debugging

Purpose: Test verifies different approaches how to run and debug unit tests.
Setup: Open the sample project from previous suite. Empty the "Test Packages" folder (if not already empty)

  1. "First run"
    1. Select "teststorun" package and select 'Tools > Create Tests'
    2. Ensure that all checkboxes in the "Create Tests" dialog are checked
    3. Press OK
    4. Run all the tests by selecting project root node and invoke 'Test (Alt+F6)' ('Test ^F6' on OSX) from its content menu
    • EXPECTED RESULT: Test Results window is displayed with results of all tests. 9 tests are shown as FAILED and 1 test caused an error (+ reason of failures and error).
      Red progress bar shows 0.0%
  2. "Modify the tests"
    1. In "CalculatorTest.java" remove or comment all lines with "fail("The test case is a prototype.");"
    2. Modify the values for x, y and expResult in both test methods, so that testAdd() method shall pass the test (e.g. x=10.0, y=20.0, expResult=30.0)
      and testDivide() shall fail (e.g. x=100.0, y=50.0, expResult=10.0)
    3. In "AbstractThingTest.java" remove or comment line with "fail("The test case is a prototype.");" only in testDoSomething() method.
    4. In "StatementsTest.java" remove or comment all lines with "fail("The test case is a prototype.");"
    5. Add following method to "StatementsTest.java":
    • Note: If you are using JUnit4 for testing, add the "@Test" annotation above the method
public void testMyOwn() {
    System.out.println("My Own Method");
    assertTrue (Statements.equalArrays(new int[]{1,2,3}, new int[]{1,2,3}));
    assertTrue (Statements.equalArrays(new int[]{1,2,3}, new int[]{1,2}));
    assertFalse(Statements.equalArrays(new int[]{1,2,3}, new int[]{4,5,6}));
} 
  1. "Rerun"
    1. In the Test Results window press the 'Rerun' button (with green "fast forward" icon)
    • EXPECTED RESULT: All tests are run again. 2 tests passed, 8 tests failed and 1 caused an error.
      Progress bar shows 18.18% and some progress (green bar)
  2. "Run single test file"
    1. Select file "AbstractThing.java" from the (Source Packages) "teststorun" package
    2. Invoke 'Test File (Ctrl+F6)' from its popup menu
    • EXPECTED RESULT: Only three tests from the AbstractThingTest are run.
  3. "Run single test file II"
    1. Select file "AbstractThingTest.java" from the (Test Packages) "teststorun" package
    2. Invoke 'Run File (Shift+F6)' from its popup menu
    • EXPECTED RESULT: The same tests (as in previous step) are run
  4. "Run single test file III"
    1. Select file "AbstractThingTest.java" from the (Test Packages) "teststorun" package again
    2. Invoke 'Test File (Ctrl+F6)' from its popup menu
    • EXPECTED RESULT: The same tests are run again
  5. "Run test suite"
    1. Select file "TestsToRunSuite.java" from the (Test Packages) "teststorun" package
    2. add code "suite.addTest(CalculatorTest.suite());" as part of "suite()" method
    3. Invoke 'Run File (Shift+F6)' from its popup menu
    4. Invoke 'Test File (Ctrl+F6)' from its popup menu
    • EXPECTED RESULT: All test methods are run in both cases. Result is the same as in case of testing the whole project (2 pass, 8 fail, 1 error)
  6. "Debugging"
    1. Open file "AbstractThingTest.java" in the editor and place line breakpoint anywhere to testChange() method
    2. Place another line breakpoint into "CalculatorTest.java" to the testAdd() method
    3. Select file "AbstractThing.java" from the (Source Packages) "teststorun" package
    4. Invoke 'Debug Test File (Ctrl+Shift+F6)' from its popup menu
    • EXPECTED RESULT: Debugging is started. The debugger stops at the breakpoint in the testChange() method.
    • Press the green 'Continue' button (F5), so the debugger finishes
    • EXPECTED RESULT: Debugging is finished. Test Results window is displayed.
  7. "Debugging II"
    1. Select file "AbstractThingTest.java" from the (Test Packages) "teststorun" package
    2. Try debugging the test file by invoking 'Debug File (Ctrl+Shift+F5)' from its popup menu
    3. Then try selecting 'Debug Test File (Ctrl+Shift+F6)' on the same file
    • EXPECTED RESULT: In both cases the debugging is run and it stops at the breakpoint, just like in the previous test case.
  8. "Debugging Test Suite"
    1. Select file "TeststorunSuite.java" from the (Test Packages) "teststorun" package
    2. Invoke 'Debug File (Ctrl+Shift+F5)' from its popup menu
    • EXPECTED RESULT: Debugger is started, it stops in testAdd() method (CalculatorTest) and after pressing the Continue button, it stops testChange() method (AbstractThingTest). Pressing the Continue button once again finishes the debugging and displays Test Results.
    • Note: the order of the previous methods could differ (JDK 7 changed the order of reflection method order and thus the JUnit method order is changed as well)
  9. "Debugging Test Suite II"
    1. Select file "TestsToRunSuite.java" again
    2. Invoke 'Debug Test File (Ctrl+Shift+F6)' from its popup menu
    • EXPECTED RESULT: Debugger is started successfully, it works just like in the previous testcase.


Test suite: Results window actions

Purpose: Testing functionality of Test Results window buttons and popup menu options
Setup: Continue from previous test suite. Run Test of the whole project (Alt+F6), so the Test Results window is opened.

  1. "Output panel"
    1. Check the output in the right side of the splited Test Results window
    • EXPECTED RESULT: It contains the output of all test (it corresponds to the screenshot above)
    • Note: The output of each test is defined using the System.out.println method
    • Note: For standard test output search for output snippets:
------------- Standard Output ---------------
...
------------- ---------------- ---------------
  1. "Tooltips"
    1. Move mouse cursor over the nodes in Test Results window
    • EXPECTED RESULT: Tooltip is displayed with the output of the test method (or all test methods in case of test class node). For example: tooltip for "testChange" node displays "change", tooltip for "teststorun.CalculatorTest" displays "add" and "divide"
  2. "Filter button"
    1. Toggle/untoggle filter buttons (labeled "Show Passed/Failed/Error")
    • EXPECTED RESULT: Only passed/failed/error tests are shown if the button is toggled.
  3. "Navigation - test class"
    1. In Test Results tree doubleclick on "teststorun.AbstractThingTest" node.
    • EXPECTED RESULT: "AbstractThingTest.java" is opened in the editor. Focus is on the line with class definition
  4. "Navigation - test method"
    1. Doubleclick on "testAdd (passed)" node under "teststorun.CalculatorTest".
    • EXPECTED RESULT: "CalculatorTest.java" is opened in the editor and focus is on testAdd() method
  5. "Navigation - cause of a failure"
    1. Expand "testMyOwn (FAILED)" node under "teststorun.StatementsTest".
    2. Doubleclick on "junit.framework.AssertionFailedError"
    • EXPECTED RESULT: "StatementsTest.java" is opened in the editor and focus is on the line that caused the error in testMyOwn() method (second assertTrue statement)
  6. "Go to source"
    1. Repeat previous navigation tests (3., 4., 5.), but instead of doubleclicking on the node use "Go to Source" option from its popup menu
    • EXPECTED RESULT: It will open appropriate test file with focus on class/method/line (same result as double-clicking on the node)
  7. "Previous/Next Failure"
    1. Press the UP or DOWN ARROW button part of the Test Results vertical toolbar
    • EXPECTED RESULT: Previous/Next failure node is highlighted. it is possible to cycle through all failures
  8. "View Difference"
    1. Select a node where the test failed because of different expected and actual value (contains text: "expected<...> but was <...>") e.g. testDivide method node
    2. Select "View Difference" from the node's popup menu
    • EXPECTED RESULT: New window is opened. Both values, Expected and Actual, can be seen
  9. "Run Again"
    1. Select "Run Again" from the "testDivide" node's popup menu
    • EXPECTED RESULT: Only the testDivide() test method is run (NOT the whole CalculatorTest). Results window contains only result of 1 test (failed). Popup menu of the "testDivide" node contains 4 options (Go to Source, Run Again, Debug, View Difference).
  10. "Debug"
    1. Place a line breakpoint on any line in testDivide() method
    2. Ensure there is still the breakpoint in testAdd() method from previous test (if not, create new one)
    3. Select "Debug" from the "testDivide" node's popup menu in the Test Results window
    • EXPECTED RESULT: The debugger is run and it stops at the breakpoint in testDivide() method. I does NOT stop in testAdd() method, as we are not debugging the whole test, but just one method.
  11. Results Window actions consistency
    1. Try using different ways of running tests (test the whole project, test single file, run test suite)
    2. Verify that Test Result window offers still the same options for all nodes and they are working correctly
    • EXPECTED RESULT: All actions described above should be available no matter how the test is run.


Test suite: Different project types

Purpose: It is important to test JUnit functionality for different project types, especially the ability to run and debug tests as well as proper working Test Results window
Setup: Run the IDE with clear userdir

  1. "Project with disabled Compile on Save"
    • Note: There is an issue 161283 describing effect of 'Compile on Save' on Test Results window and its actions. WORKS FOR ME, PLEASE CONFIRM
    1. Open the sample project used in previous test suites
    2. Remove the whole content of Test Packages
    3. Open project's properties, open category 'Build' - 'Compiling' and uncheck the checkbox next to 'Compile on Save'
    4. Press OK
    5. Repeat the steps from "Running and Debugging" test suite
    6. Repeat the steps from "Results window actions" test suite
    • EXPECTED RESULT: The functionality should be all the same. Turning of the CoS should not have impact on handling test files and/or actions in Test Results window.
  2. "Maven project"
    • Note: To switch between JUnit 3 and 4 remove JUnit dependency from pom.xml
    1. Create new Maven Project (choose Maven -> Project from Archetype and type "maven-archetype-quickstart" into Search textbox)
    2. Copy the whole package "teststorun" (from Source Packages) and its content from the sample project used in previous steps to Source Packages of this new Maven project
    3. Repeat the steps from "Running and Debugging" test suite
    4. Repeat the steps from "Results window actions" test suite
    • EXPECTED RESULT: Running and debugging of test files works correctly (except differences described below). The same actions as in case of J2SE project are supported in Test Results window
    • Known differences: 'Run File', 'Debug File' and 'Profile File' cannot be executed on test file or test suite
  3. "Web Application"
    1. Create new Java Web Application with default setting
    2. Copy the whole package "teststorun" (from Source Packages) and its content from the sample project used in previous steps to Source Packages of this new Web Application project
    3. Repeat the steps from "Running and Debugging" test suite
    4. Repeat the steps from "Results window actions" test suite
    • EXPECTED RESULT: Everything works correctly.
  4. "NB Module"
    1. Create new NetBeans module project
    2. Leave all settings to their default values. When prompted for a Code Name Base, type e.g. "org.myorg.mymodule"
    3. Copy the whole package "teststorun" (from Source Packages) and its content from the sample project used in previous steps to Source Packages of this new NetBeans module project
    4. Repeat the steps from "Running and Debugging" test suite
    5. Repeat the steps from "Results window actions" test suite
    • EXPECTED RESULT: Everything works correctly.
    • Known issue: issue 177959
    • Known issue: bug 204918: Wrong JUnit Library selected on NB Module - DOESN'T WORK FOR ME, PLEASE CONFIRM


Test suite: TestNG

Purpose: Verify functionality of TestNG integration
Setup: Start IDE with clear userdir. Create new java se project.

  1. "Templates - TestNG test class"
    1. In new file wizard, create "Unit Tests | TestNG Test case". Leave all checkbox checked, fill name of some package.
    2. After test is created, uncomment test method.
    • EXPECTED RESULT: Testcase file is created, no compilation error are shown.
  1. "Templates - TestNG test suite"
    1. In new file wizard, create "Unit Tests | TestNG Test suite". Fill name of package used in previous test.
    • EXPECTED RESULT: XML test suite is created. By default in includes all test cases in package
  1. "Run test suite"
    1. On Test suite node, invoke "Test file" action.
    • EXPECTED RESULT: All test cases (in this case just one) defined in suite ran. Tests results window shows results.

Repeat the steps from "Running and Debugging" and "Results window actions" test suites with corresponding TestNG equivalents
  • use TestNG instead of JUnit when creating tests
  • Test suite is not a java file, but xml file

Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo