TS 68 JUnit

JUnit Test Specification

Author: Pavel Pribyl
Last update: 24.8. 2009
Introduction: This document contains test specifications for JUnit module.
Comments: JUnit module is an extension to JUnit framework and serves in NetBeans IDE as support for creating unit tests of individual classes or whole packages. It generates skeletons of those unit tests and allows to run tests directly from tested classes or packages.
Note: This Specification describes testing of J2SE, J2ME and J2EE projects. It is recommended to use the Full IDE distribution with all modules.
Sample project to work with: TS_68_JUnit/JUnitSampleProject_TS_68_JUnit.zip

Contents


Test suite: Test libraries

Purpose: Verify proper usage of Test Libraries in different projects
Setup: Start IDE with clear userdir.

  1. "J2SE Project"
    1. Create new Java Application project with default setting
    2. Expand Test Libraries folder
    • EXPECTED RESULT: There are two libraries present: JUnit3.x and JUnit4.x
    1. Select Main.java node in Project tab and try to create test for this class (CTRL + SHIFT + U)
    • EXPECTED RESULT: JUnit version support was not set yet, so you can see JUnit version chooser. Now you are able to choose JUnit 4.x or JUnit3.x support.
    1. Press Cancel
    2. Select one of the libraries in Test Libraries folder and remove it
    3. Select Main.java node again and select 'Tools > Create JUnit Tests' from the content menu
    4. Leave all settings unchanged and press the OK button
    • EXPECTED RESULT: Test class was created. There was no "Select JUnit Version" dialog.
    1. Delete 'MainTest.java'
    2. Remove the remaining test library (so there is none left)
    3. Try to create JUnit test for Main.java again
    • EXPECTED RESULT: The version select dialog appears again. If you choose on of the versions and create new test, corresponding test library is added to Test Libraries folder.
    1. Delete everything in Test Packages folder
    2. Select Test Libraries and select 'Add Library...' from its content menu
    3. Add the JUnit library that is currently missing (it depends on what version you have selected in previous step), so both JUnit libraries are present
    4. Open project's properties (right-click on the project's root node and select 'Properties')
    5. In category Sources change "Source Level" listbox value to "1.4" and press "OK" button
    6. Try to create JUnit test for Main.java
    • EXPECTED RESULT: "Select JUnit Version" dialog appears again. However, you are not able to choose JUnit 4.x, because it requires Java 1.5+.
    1. Leave JUnit 3.x selected and create the test with default setting
    • EXPECTED RESULT: Test "MainTest.java" is successfully created in the "Test Packages" and package with the same name as the package containing tested file. JUnit4.x library was automatically removed from the Test Libraries folder
  2. "Java Web Application"
    1. Create new Web Application (File - New Project - Java Web - Web Application) with default setting
    2. Ensure that there are 2 JUnit libraries in Test Libraries folder
    3. Create new package "newpackage" in Source Packages folder
    4. In this package create new Java Class "NewClass.java"
    5. Add some method to this class, e.g. public void foo() {}
    • Note: Tested class must have at least one testable method, otherwise the test cannot be created
    1. Select NewClass.java node and select 'Tools > Create JUnit Tests' from the content menu
    2. Choose one of the JUnit versions and create a test with default setting
    • EXPECTED RESULT: Test was successfully created. Only one library is is present in Test Libraries folder
    1. Verify, that libraries behave the same way as in "J2SE Project" testcase (try removing/adding libraries and changing source level of the project)
    • EXPECTED RESULT: Results are same as in the previous testcase
  1. "Mobile Application"
    • Note: Mobile Applications use JMUnit support for creating and running unit tests
    1. Create new Mobile Application (File - New Project - Java ME - Mobile Application) with default setting
    2. Expand Resources folder
    • EXPECTED RESULT: There is "HelloMIDlet.java" in "hello" package. Recources folder is empty by now
    1. In "hello" package create new Java Class "NewClass.java"
    2. Add some method to this class, e.g. public void foo() {}
    3. Create new test for this class (either by pressing Ctrl + Shift + U or selecting 'Tools > Create JUnit Tests')
    • EXPECTED RESULT: Test "NewClassTest.java" is successfully created in "hello" package. JMUnit4CLDC10 library was automatically added to the "Resources" folder
    1. Right-click on "Resources" folder and select 'Add Library'
    2. Select JMUnit4CLDC11 from the list and press 'Add Library' button
    3. Delete "NewClassTest.java"
    4. Create new test for "NewClass.java" (leave all settings unchanged)
    • EXPECTED RESULT: Test "NewClassTest.java" is successfully created. You are NOT prompted to choose between the test libraries. JMUnit4CLDC10 is used by default.


Test suite: Test and Testsuite files

Purpose: Test verifies automatically generated code in unit tests and test suites
Setup: Start IDE with clear userdir. Download and open TS_68_JUnit/JUnitSampleProject_TS_68_JUnit.zip

  1. "Test files"
    1. Select "classtypes" package and press Crtl+Shift+U
    2. Choose one of the JUnit versions
    • Note: Go through all the tests from this suite with the same JUnit version and then repeat it with the other version
    1. Ensure that all checkboxes in the "Create Test" dialog are chcecked
    2. Press OK
    • EXPECTED RESULT: Package "classtypes" is created in the "Test Packages" folder. There are four files present: three tests and one suite. Test filenames are "[[[TestedClassName | [TestedClassName]]Test.java, suite filename is "[[[PackageName | [PackageName]]Suite.java. Files are created without errors."
  2. "Test files without suite"
    1. Delete the package in "Test Packages"
    2. Select "classtypes" (in "Source Packages") and invoke 'Tools > Create JUnit Tests'
    3. Uncheck 'Generete Test Suites' checkbox. Other chceckboxes should be checked
    4. Press OK
    • EXPECTED RESULT: Package "classtypes" is created in the "Test Packages" folder. There are three test files present. There is no suite file. Files are created without errors."
  3. "Class Types"
    1. Delete the package in "Test Packages"
    2. Select "classtypes" (in "Source Packages") and invoke 'Tools > Create JUnit Tests'
    3. Uncheck some checkboxes under 'Class Types' (Package Private Classes, Abstract Classes, Exception Classes)
    4. Press OK
    • Note: See issue 170621
    • EXPECTED RESULT: Test for appropriate class type is NOT created. If you uncheck all three checkboxes, no test is created."
  4. "Method Access Level"
    1. Select "Methods.java" in "classtypes" package and open it in editor
    2. Invoke 'Tools > Create JUnit Tests' from the main menu
    3. Uncheck some checkboxes under 'Method Access Level' (Public, Protected, Package Private)
    4. Press OK
    • EXPECTED RESULT: Test file "MethodsTest.java" is created and opened in editor. Only tests for methods with selected access level are created (e.g. "testPublicMethod()" if 'Public' checkbox was chcecked)
  5. "Generated Code"
    1. Delete the "MethodsTest.java" test file
    2. Select "Methods.java"
    3. Invoke 'Tools > Create JUnit Tests' from the main menu
    4. Uncheck all checkboxes under 'Generated Code' (Test Initializer, Test Finalizer, Default Method Bodies)
    5. Check all other checkboxes (some were unchecked in previous steps)
    6. Press OK
    • EXPECTED RESULT: Test file "MethodsTest.java" is created and opened in editor. Test methods are empty, test initializer and test finalizer are not created.
      JUnit4.x note: Static methods setUpClass() and tearDownClass() are still present
  6. "Generated Comments"
    1. Delete the "MethodsTest.java" test file
    2. Select "Methods.java"
    3. Invoke 'Tools > Create JUnit Tests' from the main menu
    4. Uncheck all checkboxes under 'Generated Comments' (Javadoc Comments, Source Code Hints)
    5. Check all other checkboxes (some were unchecked in previous steps)
    6. Press OK
    • EXPECTED RESULT: Test file "MethodsTest.java" is created and opened in editor. Test methods are created without javadoc comments
      and without //TODO notes.
      JUnit4.x note: Annotations like @Test, @BeforeClass etc. are still present
  7. "Test names"
    1. Delete the "MethodsTest.java" test file
    2. Select "Methods.java" and select to create ne JUnit test
    3. Change the class name to "methodaccess.MyMethodsTest"
    4. Change the class name to "mymethodaccess.MethodsTest"
    • EXPECTED RESULT: The same warning is displayed in both cases ("You will not be able to navigate..."). The test still can be created (OK button is enabled)
  8. "Test names II"
    1. Change the class name to "methodaccess.MethodsTestXXX" (so the class name is not ending with "...Test")
    • EXPECTED RESULT: Different warning is displayed than in previous case ("The projects infrastructure..."). The test cannot be created (OK button is disabled)
  9. "Test Package"
    1. Close the "Create Test" dialog
    2. Select project root node and select 'Properties' from its content menu
    3. In category Sources - Test Package Folders press Add Folder button
    4. Create new folder in your project directory named "test2" and select it (Note: you may have to create this folder externally)
    • EXPECTED RESULT: "test2" is added to Test Package Folders. Its label is set to "test2"
    1. Change the label to "My Test Package"
    2. Press OK button
    3. Try to create new test for some file or package
    • EXPECTED RESULT: "My Test Package" can be selected as 'Location:' in the "Create Test" dialog
  10. "Create New JUnit Test"
    1. Select "Test Packages" and invoke 'File - New File (Ctrl+N)'
    2. Choose "Junit - JUnit Test"
    3. press Next >
    4. Type "mypackage" for 'Package:' Leave other setting unchanged
    5. Press Finish
    • EXPECTED RESULT: New test file with no test methods is created in "mapackage" package
  11. "Create New Test for Existing Class"
    1. Select "Test Packages" and select 'New > Other...' from its content menu
    2. Choose "Junit - Test for Existing Class"
    3. press Next >
    4. Use the Browse button to select some class from the project
    5. Press OK
    6. Uncheck some checkboxes (under Method Access Level, Generated Code, Generated Comments) and press Finish
    • EXPECTED RESULT: New test file "[[[TestedClassName | [TestedClassName]]Test.java is created. Package name corresponds to the package, where the tested class is. Source code of the test is generated according to the settings made in the last step.
  12. "Create New Test Suite"
    1. Select "Test Packages" and select 'New > Other...' from its content menu
    2. Choose "Junit - Test Suite"
    3. press Next >
    4. Choose "mypackage" from the 'Package:' drop down menu
    5. Press OK
    6. Uncheck some checkboxes (under Generated Code, Generated Comments) and press Finish
    • EXPECTED RESULT: New test suite is created in "mapackage" package. Source code of the suite is generated according to the settings made in the last step.


Test suite: Running and Debugging

Purpose: Test verifies different approaches how to run and debug unit tests.
Setup: Open the sample project from previous suite. Empty the "Test Packages" folder (if not already empty)

  1. "First run"
    1. Select "teststorun" package and select 'Tools > Create JUnit Tests'
    2. Choose one of the JUnit versions
    3. Ensure that all checkboxes in the "Create Test" dialog are chcecked
    4. Press OK
    • Note : There is an issue 169565 with creating tests for Java Interface. You have to manually change "extends" to "implements" in "IFaceTest.java" on the line containing "public class IfaceImpl extends Iface"
    1. Run all the tests by selecting project root node and invoke 'Test (Alt+F6)' from its content menu
    • EXPECTED RESULT: Test Results window is displayed with results of all tests. All tests are shown as FAILED (+ reason of failure).
      Red progress bar shows 0.0%
  1. "Modify the tests"
    1. In "CalculatorTest.java" remove or comment all lines with "fail("The test case is a prototype.");"
    2. Modify the values for x, y and expResult in both test methods, so that testAdd() method shall pass the test (e.g. x=10.0, y=20.0, expResult=30.0)
      and testDivide() shall fail (e.g. x=100.0, y=50.0, expResult=10.0)
    3. In "AbstractThingTest.java" remove or comment line with "fail("The test case is a prototype.");" only in testDoSomething() method.
    4. In "StatementsTest.java" remove or comment all lines with "fail("The test case is a prototype.");"
    5. Add following method to "StatementsTest.java":
    • Note: If you are using JUnit4 for testing, add the "@test" annotation above the method
    • public void testMyOwn() {
      
System.out.println("My Own Method"); assertTrue(Statements.equalArrays(new int[[ | ]]{1,2,3}, new int[]{1,2,3})); assertTrue(Statements.equalArrays(new int[[ | ]]{1,2,3}, new int[]{1,2})); assertFalse(Statements.equalArrays(new int[[ | ]]{1,2,3}, new int[]{4,5,6})); }
  1. "Rerun"
    1. In the Test Results window press the 'Rerun' button (with green "fast forward" icon)
    • EXPECTED RESULT: All tests are run again. 2 tests passed, 8 tests failed and 1 caused an error.
      Progress bar shows 18.2% and some progress (green bar)
      File:Test Result Window
  2. "Run single test file"
    1. Select file "AbstractThing.java" from the "teststorun" package
    2. Invoke 'Test File (Ctrl+F6)' from its popup menu
    • EXPECTED RESULT: Only three tests from the AbstractThingTest are run.
  3. "Run single test file II"
    1. Select file "AbstractThingTest.java" from the (Test Packages) "teststorun" package
    2. Invoke 'Run File (Shift+F6)' from its popup menu
    • EXPECTED RESULT: The same tests (as in previous step) are run
  4. "Run single test file III"
    1. Select file "AbstractThingTest.java" from the (Test Packages) "teststorun" package again
    2. Invoke 'Test File (Ctrl+F6)' from its popup menu
    • EXPECTED RESULT: The same tests are run again
  5. "Run test suite"
    1. Select file "TestsToRunSuite.java" from the (Test Packages) "teststorun" package
    2. Invoke 'Run File (Shift+F6)' from its popup menu
    3. Invoke 'Test File (Ctrl+F6)' from its popup menu
    • EXPECTED RESULT: All test methods are run in both cases. Result is the same as in case of testing the whole project (2 pass, 8 fail, 1 error)
  6. "Debugging"
    1. Open file "AbstractThingTest.java" in the editor and place line breakpoint anywhere to testChange() method
    2. Place another line breakpoint into "CalculatorTest.java" to the testAdd() method
    3. Select file "AbstractThing.java" from the (Source Packages) "teststorun" package
    4. Invoke 'Debug Test File (Ctrl+Shift+F6)' from its popup menu
    • EXPECTED RESULT: Debugging is started. The debugger stops at the breakpoint in the testChange() method.
    • Press the green 'Continue' button (F5), so the debugger finishes
    • EXPECTED RESULT: Debugging is finished. Test Results window is displayed.
  7. "Debugging II"
    1. Select file "AbstractThingTest.java" from the (Test Packages) "teststorun" package
    2. Try debugging the test file by invoking 'Debug File (Ctrl+Shift+F5)' from its popup menu
    3. Then try selecting 'Debug Test File (Ctrl+Shift+F6)' on the same file
    • EXPECTED RESULT: In both cases the debugging is run and it stops at the breakpoint, just like in the previous test case.
  8. "Debugging Test Suite"
    1. Select file "TestsToRunSuite.java" from the (Test Packages) "teststorun" package
    2. Invoke 'Debug File (Ctrl+Shift+F5)' from its popup menu
    • EXPECTED RESULT: Debugger is started, it stops in testAdd() method (CalculatorTest) and after pressing the Continue button, it stops testChange() method (AbstractThingTest). Pressing the Continue button once again finishes the debugging and displays Test Results.
  9. "Debugging Test Suite II"
    1. Select file "TestsToRunSuite.java" again
    2. Invoke 'Debug Test File (Ctrl+Shift+F6)' from its popup menu
    • EXPECTED RESULT: Debugger is started successfully, it works just like in the previous testcase.


Test suite: Results window actions

Purpose: Testing functionality of Test Results window buttons and popup menu options
Setup: Continue from previous test suite. Run Test of the whole project (Alt+F6), so the Test Results window is opened.

  1. "Output panel"
    1. Check the output in the right side of the splited Test Results window
    • EXPECTED RESULT: It contains the output of all test (it corresponds to the screenshot above)
    • Note: The output of each test is defined using the System.out.println method
  2. "Tooltips"
    1. Move mouse cursor over the nodes in Test Results window
    • EXPECTED RESULT: Tooltip is displayed with the output of the test method (or all test methods in case of test class node). For example: tooltip for "testChange" node displays "change", tooltip for "teststorun.CalculatorTest" displays "add" and "divide"
  3. "Filter button"
    1. Toggle/untoggle filter button (labeled "Show Failures Only")
    • EXPECTED RESULT: Only failed tests are shown if the button is toggled.
  4. "Navigation - test class"
    1. In Test Results tree doubleclick on "teststorun.AbstractThingTest" node.
    • EXPECTED RESULT: "AbstractThingTest.java" is opened in the editor. Focus is on the line with class definition
  5. "Navigation - test method"
    1. Doubleclick on "testAdd (passed)" node under "teststorun.CalculatorTest".
    • EXPECTED RESULT: "CalculatorTest.java" is opened in the editor and focus is on testAdd() method
  6. "Navigation - cause of a failure"
    1. Expand "testMyOwn (FAILED)" node under "teststorun.StatementsTest".
    2. Doubleclick on "junit.framework.AssertionFailed"
    • EXPECTED RESULT: "StatementsTest.java" is opened in the editor and focus is on the line that caused the error in testMyOwn() method (second AssertTrue statement)
  7. "Go to source"
    1. Repeat previous navigation tests, but instead of doubleclicking on the node use "Go to Source" option from its popup menu
    • EXPECTED RESULT: It will open appropriate test file with focus on class/method/line (same result as double-clicking on the node)
  8. "Previous/Next Failure"
    1. Press the UP or DOWN ARROW button
    • EXPECTED RESULT: Previous/Next failure node is highlighted. it is possible to cycle through all failures
  9. "View Difference"
    1. Select a node where the test failed because of different expected and actual value (contains text: "expected<...> but was <...>")
    2. Select "View Difference" from the node's popup menu
    • EXPECTED RESULT: New window is opened. Both values, Expected and Actual, can be seen
  10. "Run Again"
    1. Select "Run Again" from the "testDivide" node's popup menu
    • EXPECTED RESULT: Only the testDivide() test method is run (NOT the whole CalculatorTest). Results window contains only result of 1 test (failed). Popup menu of the "testDivide" node contains 4 options (Go to Source, Run Again, Debug, View Difference).
  11. "Debug"
    1. Place a line breakpoint on any line in testDivide() method
    2. Ensure there is still the breakpoint in testAdd() method from previous test (if not, create new one)
    3. Select "Debug" from the "testDivide" node's popup menu in the Test Results window
    • EXPECTED RESULT: The debugger is run and it stops at the breakpoint in testDivide() method. I does NOT stop in testAdd() method, as we are not debugging the whole test, but just one method.
  12. Results Window actions consistency
    1. Try using different ways of running tests (test the whole project, test single file, run test suite)
    2. Verify that Test Result window offers still the same options for all nodes and they are working correctly
    • EXPECTED RESULT: All actions described above should be available no matter how the test is run.
    • Note: There is an issue 171050 describing different behavior of Test Results window


Test suite: Different project types

Purpose: It is important to test JUnit functionality for different project types, especially the ability to run and debug tests as well as proper working Test Results window
Setup: Run the IDE with clear userdir

  1. "Project with disabled Compile on Save"
    • Note: There is an issue 161283 describing effect of 'Compile on Save' on Test Results window and its actions.
    1. Open the sample project used in previous test suites
    2. Remove the whole content of Test Packages
    3. Open project's properties, open category 'Build' - 'Compiling' and uncheck the checkbox next to 'Compile on Save'
    4. Press OK
    5. Repeat the steps from "Running and Debugging" test suite
    6. Repeat the steps from "Results window actions" test suite
    • EXPECTED RESULT: The functionality should be all the same. Turning of the CoS should not have impact on handling test files and/or actions in Test Results window.
  2. "Maven project"
    1. Create new Maven Project (choose Maven Quickstart Archetype 1.0)
    2. Copy the whole package "teststorun" (from Source Packages) and its content from the sample project used in previous steps to Source Packages of this new Maven project
    3. Repeat the steps from "Running and Debugging" test suite
    4. Repeat the steps from "Results window actions" test suite
    • EXPECTED RESULT: Running and debugging of test files works correctly (except differences described below). The same actions as in case of J2SE project are supported in Test Results window
    • Known differences: 'Run File' and 'Debug File' cannot be invoked on test file
    • Known issues (please check before posting new one): issue 171057
  3. "Web Application"
    1. Create new Java Web Application with default setting
    2. Copy the whole package "teststorun" (from Source Packages) and its content from the sample project used in previous steps to Source Packages of this new Web Application project
    3. Repeat the steps from "Running and Debugging" test suite
    4. Repeat the steps from "Results window actions" test suite
    • EXPECTED RESULT: Everything works correctly.
    • Known issues: issue 171153: "Run Again" and "Debug" actions are not available in Results window.
  4. "NB Module"
    1. Create new NetBeans module project
    2. Leave all settings to their default values. When prompted for a Code Name Base, type e.g. "org.myorg.mymodule"
    3. Copy the whole package "teststorun" (from Source Packages) and its content from the sample project used in previous steps to Source Packages of this new NetBeans module project
    4. Repeat the steps from "Running and Debugging" test suite
    5. Repeat the steps from "Results window actions" test suite
    • EXPECTED RESULT: Everything works correctly.
    • Known issues: "Run Again" and "Debug" actions are not available in Results window.


Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo