---

Can we run performance tests without XTest?

Done!

• The main testing infrastructure simplification has been integrated to main repository. It is ready to be included for 6.5M2.

Shall we?

• Yes, because xtest is not integrated with the NetBeans API Support
• Yes, Xtest contains tons of magical build scripts
• Yes, it has too much configuration files, while its setup is not easy
• Yes, did anyone tried to profile an XTest test? For regular tests this works quite easy.

Deliverables

• P1: run commit validation without XTest
• P1: modify API support to support this setup
• P1: allow binary test distribution to run all tests without XTest
• P3: make everything work equivalently for external modules

Repo

http://hg.netbeans.org/simpletests/

hg clone -r d6fb81940cd5 main simpletests
hg -R simpletests pull http://hg.netbeans.org/simpletests/

Continuous build

simpletests on Hudson

Migration Guide

Running out of memory when executing your tests? Try:

--- a/o.n.core/nbproject/project.properties     Wed Jun 25 16:18:57 2008 -0400
+++ b/o.n.core/nbproject/project.properties     Wed Jun 25 16:33:04 2008 -0400
@@ -50,4 +50,5 @@ test.excludes=\
     org/netbeans/core/projects/data/
-
+# Otherwise ValidateLayerConsistencyTest can fail:
+test.run.args=-ea -XX:MaxPermSize=200m

This is the sample change that was needed in order to execute the web.structs module in the IDE mode in the new way:

diff -r 59a0ec8e6882 web.struts/nbproject/project.xml
--- a/web.struts/nbproject/project.xml  Mon May 19 14:47:53 2008 +0200
+++ b/web.struts/nbproject/project.xml  Mon May 19 14:48:13 2008 +0200
@@ -339,6 +339,7 @@ made subject to such option by the copyr
                     </test-dependency>
                     <test-dependency>
                         <code-name-base>org.netbeans.modules.nbjunit</code-name-base>
+                        <recursive/>
                         <compile-dependency/>
                     </test-dependency>
                 </test-type>
diff -r 59a0ec8e6882 web.struts/test/qa-functional/src/test/EndToEndTest.java
--- a/web.struts/test/qa-functional/src/test/EndToEndTest.java  Mon May 19 14:47:53 2008 +0200
+++ b/web.struts/test/qa-functional/src/test/EndToEndTest.java  Mon May 19 14:48:13 2008 +0200
@@ -48,6 +48,7 @@ import java.net.URLConnection;
 import java.net.URLConnection;
 import java.util.Properties;
 import javax.swing.JTextField;
+import junit.framework.Test;
 import org.netbeans.jellytools.Bundle;
 import org.netbeans.jellytools.EditorOperator;
 import org.netbeans.jellytools.NewFileWizardOperator;
@@ -60,7 +61,6 @@ import org.netbeans.jemmy.Waitable;
 import org.netbeans.jemmy.Waitable;
 import org.netbeans.jemmy.Waiter;
 import org.netbeans.jemmy.operators.JButtonOperator;
-import org.netbeans.junit.NbTestSuite;
 import org.netbeans.jellytools.JellyTestCase;
 import org.netbeans.jellytools.NbDialogOperator;
 import org.netbeans.jellytools.NewFileNameLocationStepOperator;
@@ -79,6 +79,7 @@ import org.netbeans.jemmy.operators.JTex
 import org.netbeans.jemmy.operators.JTextFieldOperator;
 import org.netbeans.jemmy.operators.JTreeOperator;
 import org.netbeans.jemmy.operators.Operator.DefaultStringComparator;
+import org.netbeans.junit.NbModuleSuite;
 import org.netbeans.junit.ide.ProjectSupport;

 /** End-to-end scenario test based on
@@ -96,27 +97,22 @@ public class EndToEndTest extends JellyT
     }

     /** Creates suite from particular test cases. You can define order of testcases here. */
-    public static NbTestSuite suite() {
-        NbTestSuite suite = new NbTestSuite();
-        suite.addTest(new EndToEndTest("testSetupStrutsProject"));
-        suite.addTest(new EndToEndTest("testCreateLoginPage"));
-        suite.addTest(new EndToEndTest("testCreateLoginBean"));
-        suite.addTest(new EndToEndTest("testCreateLoginAction"));
-        suite.addTest(new EndToEndTest("testCreateSecurityManager"));
-        suite.addTest(new EndToEndTest("testCreateForward"));
-        suite.addTest(new EndToEndTest("testCreateShopPage"));
-        suite.addTest(new EndToEndTest("testCreateLogoutPage"));
-        suite.addTest(new EndToEndTest("testCreateForwardInclude"));
-        suite.addTest(new EndToEndTest("testRunApplication"));
-        return suite;
-    }
-
-    /* Method allowing test execution directly from the IDE. */
-    public static void main(java.lang.String[] args) {
-        // run whole suite
-        junit.textui.TestRunner.run(suite());
-        // run only selected test case
-        //junit.textui.TestRunner.run(new EndToEndTest("test1"));
+    public static Test suite() {
+        return NbModuleSuite.create(
+            NbModuleSuite.createConfiguration(EndToEndTest.class).addTest(
+              "testSetupStrutsProject", "testCreateLoginPage", "testCreateLoginBean",
+              "testCreateLoginAction", "testCreateSecurityManager", "testCreateForward",
+              "testCreateShopPage", "testCreateLogoutPage", "testCreateForwardInclude",
+              "testRunApplication"
+            )
+            .enableModules(".*")
+            .clusters(".*")
+        );
     }

     /** Called before every test case. */

Quite easy, almost one to one mapping, one would say. More info is also available directly from the users, the quality engineers.

QUESTION (from Tom Wheeler): The XTestReplacementCookBook link above points to a non-public Sun server. I would be very interested in reading it -- and anything else that has to do with testing platform applications. Is there a chance you could post a copy somewhere that non-Sun people could get to it?

If you want to create config with name e.g. commit, go to $MODULENAME/nbproject/project.properties and add property which classes you want to run:

test.config.commit.includes=\
    org/netbeans/test/ide/IDECommitValidationTest.class,\
    org/netbeans/test/editor/MultiviewEditorReflectionTest.class

This example comes from ide.kit module. You can use also wildcards:

test.config.commit.includes=**/Validate*Test.class 

This example comes from core.windows module.

These tests are executable from $MODULENAME directory with this command.

ant test-qa-functional -Dtest.config=commit

Controlling execution order of tests

If you have multiple tests that are supposed to run in the same VM, in the specified order. Then you can either place them into one Test class and have correct order of its methods, or, if you have them in multiple classes, you can use the ordering abilities of NbModuleSuite:

public static Test suite() {
  return NbModuleSuite.create(
    NbModuleSuite.emptyConfiguration()
    .addTest(FirstTest.class)
    .addTest(SecondTest.class)
    .addTest(ThirdTest.class)
  );
}

This will guarantee that all testXYZ methods in the FirstTest class are executed first, and then the SecondTest, etc. You can also explicitly list the individual test method names, in case you want to control that order as well by: addTest(FirstTest.class, "testM1", "testM2");

Selecting the right tests for various configs

Imagine you have one test case that contains a lot of tests, but you'd like to run them in various configurations: stable, all, really stable, etc. This is the recommended structure for your tests

public class CheckIDEWorks extends NbTestCase {
  public CheckIDEWorks(String n) { super(n); }

  public void testBasics() { ... }
  public void testMore() { ... }
  public void testRest() { ... }
}

The previous example define a regular test case with a set of methods that define behaviour of all your tests. However, it is not executed by the infrastructure by default, as its name does not end with "Test", and as such it is not recognized as test. You can do that, if you want, but suppose you want to create three different configurations for these tests. Then you can define own classes:

public class Stable { 
  public static Test suite() {
    return NbModuleSuite.create(
      NbModuleSuite.emptyConfiguration()
        .addTest(CheckIDEWorks.class, "testBasics", "testMore")
    );
  }
}
public class ReallyStable { 
  public static Test suite() {
    return NbModuleSuite.create(
      NbModuleSuite.emptyConfiguration()
        .addTest(CheckIDEWorks.class, "testBasics")
    );
  }
}
public class AllTest { 
  public static Test suite() {
    return NbModuleSuite.create(
      NbModuleSuite.emptyConfiguration()
        .addTest(CheckIDEWorks.class) // without a list of testcases, all are executed
    );
  }
}

Now, you can just predefine various configs that you wish to be executed. So you can have a test config for stable, as well as commit set of tests in your nbproject/project.properties:

test.config.stable.includes=**/Stable.class 
test.config.commit.includes=**/ReallyStable.class 

You may, but need not specify config for all tests, by default every class with name that ends with Test is included, which in our example means just AllTest. With a style like this you can create as many tests as you want, group them into execution units according to their stability and just by changing properties specify which execution units shall be executed when.

How to set up and run Tomcat, Glassfish, ...

To get an auto access to servers(Tomcat, Glassfish, JBoss) you should extend J2eeTestCase instead of JellyTestCase. It provides methods

 Configuration addServerTests(Configuration, String...)
 Configuration addServerTests(Server server, Configuration, String...)

Server

is an enum of actually supported servers. The methods are trying to find server using system properties and register it into the IDE. If there is a required server, test's are added into configuration, if there is no required server emptyTest is added so that the configuration would be runnable but no failure appeared.

If no server is required, no server is registered in IDE.

You can also use methods isRegistered(Server server), getServerNode(Server server) these are supposed to by used in test runtime.

If you need to run your test on all servers use addServerTests(...) and start test with different server settings each time.

To set server path start test using ant command ant -Dtest-qa-functional-sys-prop.tomcat.home=/instalace/tomcat test or set tomcat.home, jboss.home, glassfish.home properties in different way.

To be done

• (13ad8e27dcf0) do not fail whole build when modules.list is set to some module, which have e.g. only unit tests. E.g.

ant -Dnetbeans.dest.dir=/space/simpletests/netbeans \
  -Dmodules.list=java2/org-netbeans-modules-ant-grammar

fails with

BUILD FAILED
/space/simpletests/build.xml:47: The following error occurred while executing this line:
/space/simpletests/all-tests.xml:61:
Invalid file: /space/simpletests/qa-functional/java2/org-netbeans-modules-ant-grammar

• (WONTFIX, feature of hudson itself) setting JDK for execution (already tracked) - serious for us
• (ecf494ffc72b) Have an option to run several testcases within the same suite without NetBeans restarts
• (8572baa87847) remove usage of modules.patches.* in NbModuleSuite (package-private access from tests to their modules is not going to be possible anyway), and test that tests can access impl classes with the public access modifier in any modules or tests declared in tests deps
• (P3) make running tests from test distribution work for external modules which are not included in the target platform and thus have no module configs
• (P3) consider providing support for JUnit 4-style tests (using annotations) in NbTestCase
• (6ca3fcb52d31) classLoader doesn't load all classes needed in form module

To be done by QA after merge

• (P1) There are a lot of functional tests that will need a new suite method: hg loc -r tip '*/test/qa-functional/src/**Test.java' Some already have a suite method, which will need to be rewritten.
  • As soon as we migrate performance tests, Marian and his guys will take care of the rest
• (P1) a number of functional tests cannot run because deps on nbjunit or java.j2seproject are missing <recursive/> and thus cause NoClassDefFoundErrors at runtime (e.g. on o.n.insane or projectui classes)
• (P3) update nbbuild/hudson/trunk to run all C/V tests under new regime
  • to start with, ant commit-validation must not use XTest
  • check up on functional tests relying on Glassfish or WTK
• (P3) delete */test/build*.xml and */test/cfg-*.xml
  • In many cases (e.g. uml.kit) these files are the only source of info about the classpath of functional tests (nbproject/project.* were never correctly updated).
• (P3) xml/test/qa-lib should probably be moved to a normal place (in which case try reenabling tests in xml.text, xml.tools, and xml.tools.java). Generally there are some odd test dirs: ls -1d */test/*/src | egrep -v 'unit|qa-functional'
• (P3) fix tests that do something silly with the data dir: fgrep xtest.data */nbproject/project.properties

After merge, should also update TestDistribution.

Binary test distribution without X

QA is going to migrate to new test infrastructure and run there tests without X. New BTD (without X) is a little bit different (different targets, properties,..) List of properties we are going to use when we migrate to new BTD:

• test.types - to determinate test type we want to run (previously xtest.testtype)
• test.modules - to specify list of modules for which we want to run test (previously xtest.modules.list) e.g. org.netbeans.modules.ant.grammar
• test.clusters - does not need to be specified when you specify test.modules; Use in case you want to run test for whole cluster, e.g. test.clusters=platform*
• test.config - to distinguish between various test configurations - defined in nbproject/project.properties (previously done by configuring cfg xml file)
• test.disable.fails - we have to set this property while running on Hudson in order to distinguish failure of test vs failure of test run.

Some notes

XTest features

Many modules have both unit and qa-functional test dirs:

ls -1d `hg loc -r tip '*/test/qa-functional/src/**Test.java' \
  | perl -pi -e 's!/qa-functional/src/.*!/unit/src!g' | sort | uniq`

jemmy.log gets correctly written to test workdir. Seems that screen.png screenshots get saved correctly too (JellyTestCase.runBare). Will need some way to collect workdirs and publish them. Minimally, include e.g. */build/test/work/ in Hudson artifacts. Nicer would be to have a plugin which collects them and links to them from test results.

Binary test distribution creates complete dist for all clusters (easy to pick apart manually if desired).

Related issues

test.unit.cp calculation ignores Class-Path extensions (now fixed in branch)

Cannot set bigger -mx for tests (fixed in branch)

Do not include nbjunit in test CP without explicit decl (fixed in branch, so long as you specify at least dep on junit)

openide/loaders QAFunctional tests are marked as uncompilable (seems fine in branch)

Implement qa-functional tests node (now fixed in branch)

Errors in execution of binary tests distribution should be passed to parent ant task

Wrong class path order set while running JUnit test via NB default ant scripts

Support for Emma coverage in external NBM projects

Jirka's comments

• xtest and results: screenshot, jemmy.log, timeout - OK
• xtest and results: keeping whole userdir - it lays around, Hudson needs to pick it up
• xtest and config: no configs right now

Tom Wheeler's comments

We develop several platform-based applications and do a lot of testing at work. I make the following suggestions based on our experience:

• There should be support for various (and potentially arbitrary) types of tests such as unit, functional, performance and so on. XTest currently supports this although the default testing support of the IDE/harness supports only unit tests at present. JesseGlick: only unit and qa-functional test roots will be supported.

• It should be easy -- or at least possible -- for us to add support for additional testing libraries. While we use JUnit, Jemmy and Jellytools, we also use other libraries such as Fit. Ideally this could be supported by giving us some mechanism to add our own JAR files to the test classpath and a corresponding hook into an Ant task for running such tests. JesseGlick: just add them to CP.

• Currently XTest does not consider the project.xml when defining the classpath used for tests, but instead we have to set properties like test.unit.run.cp.extra and test.qa-functional.compile.cp.extra that point to specific JARs for modules within the platform. This should be automatic based on our modules' dependencies. JesseGlick: should no longer be a problem in branch.

• We build against a known binary of the platform checked out from source control; likewise we test against known libraries from source control. In other words, we don't use the testing libraries installed into the IDE; we set xtest.home, jemmy.home and jellytools.home to point to libraries that are external to the IDE. This is very important to us as we need repeatable results and because our developers use a variety of IDEs. JesseGlick: new harness just uses JARs in the platform, not an issue.

• There should be only one way to run unit tests from Ant or the IDE. Currently there are two ways when you have a module with XTest support (via the harness' support for unit tests and also through XTest support for unit tests). This is confusing for users, and worse yet, the same unit tests run under the two different mechanisms can give different results. JesseGlick: fixed in branch.

• Since the IDE now encourages you to create JUnit 4 tests, there should be support for running them

Tomas Musil's comments to testdistribution (related to build#29)

• P1 do not fail whole build when modules.list is set to some module, which have e.g. only unit tests. E.g.

ant -Dnetbeans.dest.dir=/space/simpletests/netbeans \
  -Dmodules.list=java2/org-netbeans-modules-ant-grammar

fails with

BUILD FAILED
/space/simpletests/build.xml:47: The following error occurred while executing this line:
/space/simpletests/all-tests.xml:61:
Invalid file: /space/simpletests/qa-functional/java2/org-netbeans-modules-ant-grammar

JesseGlick: should be easy enough to fix.

• seems like BUG - provide a possibility not fail whole build if some tests fail - we need to distinguish states "test ran with failures" vs. "test did not ran for some reason" i.e. omit target failtests - not sure what exactly test.disable.fails affects, but it does not seem to control only if failure of test causes failure of build. E.g. when i run

ant -Dnetbeans.dest.dir=/space/simpletests/netbeans \
  -Dmodules.list=ide9/org-netbeans-modules-utilities -Dtest.disable.fails=true

test gets stuck and ends with some o.n.jemmy.TimeoutExpiredException, while running only

ant -Dnetbeans.dest.dir=/space/simpletests/netbeans \
  -Dmodules.list=ide9/org-netbeans-modules-utilities

ends with normal test error and build failure. JesseGlick: should be easy to fix with JUnitReportSubant.

• P3 nice to have: some kind of clean target which would remove results from previous testrun JesseGlick: ant clean does not work?

• P4 mechanism of failure detection is not good - it does not respect results from current test run. I.e. $RESULTS/failed-test-var can be from previous testrun, my current testrun does not have any error/failure, but build fails with message that there were some failures.

• P1 setting JDK for execution (already tracked) - serious for us

Petr Zajac's comments

We run tests for Instant JChem in our private hudson by using junit harness. Hudson was not able to kill tree of processes. Look at discussion thread. Xtest is little more reliable in killing processes. JesseGlick: submit a patch for Hudson.

Oleg Khokhlov's comments

* P1 tests fail if you add more then one class to suite:

        ...
        public static Test suite() {
             NbTestSuite s = new NbTestSuite("UI Responsiveness J2SE Menus suite");
             s.addTest(NbModuleSuite.create(MainMenu.class, ".*", ".*"));
             s.addTest(NbModuleSuite.create(MainSubMenus.class, ".*", ".*"));
             ...

IZ bug: 134525

Tom Wheeler's feedback using this for a platform app:

These are my first impressions after nominally getting simpletests working for a platform app.

• Cannot run if Java3D installed (patch supplied)

• Removing a module from the suite (via the suite customizer dialog) does not remove it as a functional test dependency for modules in that suite. JesseGlick: can file as a low-priority bug in apisupport/project.

• Adding additional test types (such as performance or acceptance), while possible, was not as easy as it probably could be. JesseGlick: not supported.

• It would be great if there was a well-known directory (like suite/testlibs) in which we could place JARs that will automatically be visible in the test classpath(s). Otherwise, you have to define properties -- although this is not hard, it's bad for those who make platforms because Ant properties are immutable and downstream app developers cannot (easily) override or append to a property set upstream. JesseGlick: no plans to implement. Add to each module as needed.

• It seems that test libraries are modules and to get functional testing to work at all, you have to include them in the suite. Those modules have dependencies and those dependencies also cascade down, leaving me to select several dozen more modules than I need to. This wastes disk space, bandwidth, memory and clutters the app with things the developer does not want or need. Hopefully, my impression about how that part works is simply wrong (awaiting help on dev@openide), otherwise, this is a big enough problem that it will probably force us to stick with NB 6.1 and XTest until we find a solution. JesseGlick: minimize dependencies of those libraries as needed.

Attachments