FitnessTestsWithoutX

Can we run performance tests without XTest?

Done!

  • The main testing infrastructure simplification has been integrated to main repository. It is part of NetBeans 6.5.

Shall we?

  • Yes, because xtest is not integrated with the NetBeans API Support
  • Yes, Xtest contains tons of magical build scripts
  • Yes, it has too much configuration files, while its setup is not easy
  • Yes, has anyone tried to profile an XTest test? For regular tests this works quite easy.

Deliverables

  • P1: run commit validation without XTest
  • P1: modify API support to support this setup
  • P1: allow binary test distribution to run all tests without XTest
  • P3: make everything work equivalently for external modules

Repo

http://hg.netbeans.org/simpletests/

hg clone -r d6fb81940cd5 main simpletests
hg -R simpletests pull http://hg.netbeans.org/simpletests/

Continuous build

simpletests on Hudson

Migration Guide

Running out of memory when executing your tests? Try:

--- a/o.n.core/nbproject/project.properties     Wed Jun 25 16:18:57 2008 -0400
+++ b/o.n.core/nbproject/project.properties     Wed Jun 25 16:33:04 2008 -0400
@@ -50,4 +50,5 @@ test.excludes=\
     org/netbeans/core/projects/data/
-
+# Otherwise ValidateLayerConsistencyTest can fail:
+test.run.args=-ea -XX:MaxPermSize=200m


This is the sample change that was needed in order to execute the web.structs module in the IDE mode in the new way:

diff -r 59a0ec8e6882 web.struts/nbproject/project.xml
--- a/web.struts/nbproject/project.xml  Mon May 19 14:47:53 2008 +0200
+++ b/web.struts/nbproject/project.xml  Mon May 19 14:48:13 2008 +0200
@@ -339,6 +339,7 @@ made subject to such option by the copyr
                     </test-dependency>
                     <test-dependency>
                         <code-name-base>org.netbeans.modules.nbjunit</code-name-base>
+                        <recursive/>
                         <compile-dependency/>
                     </test-dependency>
                 </test-type>
diff -r 59a0ec8e6882 web.struts/test/qa-functional/src/test/EndToEndTest.java
--- a/web.struts/test/qa-functional/src/test/EndToEndTest.java  Mon May 19 14:47:53 2008 +0200
+++ b/web.struts/test/qa-functional/src/test/EndToEndTest.java  Mon May 19 14:48:13 2008 +0200
@@ -48,6 +48,7 @@ import java.net.URLConnection;
 import java.net.URLConnection;
 import java.util.Properties;
 import javax.swing.JTextField;
+import junit.framework.Test;
 import org.netbeans.jellytools.Bundle;
 import org.netbeans.jellytools.EditorOperator;
 import org.netbeans.jellytools.NewFileWizardOperator;
@@ -60,7 +61,6 @@ import org.netbeans.jemmy.Waitable;
 import org.netbeans.jemmy.Waitable;
 import org.netbeans.jemmy.Waiter;
 import org.netbeans.jemmy.operators.JButtonOperator;
-import org.netbeans.junit.NbTestSuite;
 import org.netbeans.jellytools.JellyTestCase;
 import org.netbeans.jellytools.NbDialogOperator;
 import org.netbeans.jellytools.NewFileNameLocationStepOperator;
@@ -79,6 +79,7 @@ import org.netbeans.jemmy.operators.JTex
 import org.netbeans.jemmy.operators.JTextFieldOperator;
 import org.netbeans.jemmy.operators.JTreeOperator;
 import org.netbeans.jemmy.operators.Operator.DefaultStringComparator;
+import org.netbeans.junit.NbModuleSuite;
 import org.netbeans.junit.ide.ProjectSupport;

 /** End-to-end scenario test based on
@@ -96,27 +97,22 @@ public class EndToEndTest extends JellyT
     }

     /** Creates suite from particular test cases. You can define order of testcases here. */
-    public static NbTestSuite suite() {
-        NbTestSuite suite = new NbTestSuite();
-        suite.addTest(new EndToEndTest("testSetupStrutsProject"));
-        suite.addTest(new EndToEndTest("testCreateLoginPage"));
-        suite.addTest(new EndToEndTest("testCreateLoginBean"));
-        suite.addTest(new EndToEndTest("testCreateLoginAction"));
-        suite.addTest(new EndToEndTest("testCreateSecurityManager"));
-        suite.addTest(new EndToEndTest("testCreateForward"));
-        suite.addTest(new EndToEndTest("testCreateShopPage"));
-        suite.addTest(new EndToEndTest("testCreateLogoutPage"));
-        suite.addTest(new EndToEndTest("testCreateForwardInclude"));
-        suite.addTest(new EndToEndTest("testRunApplication"));
-        return suite;
-    }
-
-    /* Method allowing test execution directly from the IDE. */
-    public static void main(java.lang.String[] args) {
-        // run whole suite
-        junit.textui.TestRunner.run(suite());
-        // run only selected test case
-        //junit.textui.TestRunner.run(new EndToEndTest("test1"));
+    public static Test suite() {
+        return NbModuleSuite.create(
+            NbModuleSuite.createConfiguration(EndToEndTest.class).addTest(
+              "testSetupStrutsProject", "testCreateLoginPage", "testCreateLoginBean",
+              "testCreateLoginAction", "testCreateSecurityManager", "testCreateForward",
+              "testCreateShopPage", "testCreateLogoutPage", "testCreateForwardInclude",
+              "testRunApplication"
+            )
+            .enableModules(".*")
+            .clusters(".*")
+        );
     }

     /** Called before every test case. */

Quite easy, almost one to one mapping, one would say. More info is also available directly from the XTest Replacement Cook Book.

If you want to create config with name e.g. commit, go to $MODULENAME/nbproject/project.properties and add property which classes you want to run:

test.config.commit.includes=\
    org/netbeans/test/ide/IDECommitValidationTest.class,\
    org/netbeans/test/editor/MultiviewEditorReflectionTest.class

This example comes from ide.kit module. You can use also wildcards:

test.config.commit.includes=**/Validate*Test.class 

This example comes from core.windows module.

These tests are executable from $MODULENAME directory with this command.

ant test-qa-functional -Dtest.config=commit

Controlling execution order of tests

If you have multiple tests that are supposed to run in the same VM, in the specified order. Then you can either place them into one Test class and have correct order of its methods, or, if you have them in multiple classes, you can use the ordering abilities of NbModuleSuite:

public static Test suite() {
  return NbModuleSuite.create(
    NbModuleSuite.emptyConfiguration()
    .addTest(FirstTest.class)
    .addTest(SecondTest.class)
    .addTest(ThirdTest.class)
  );
}

This will guarantee that all testXYZ methods in the FirstTest class are executed first, and then the SecondTest, etc. You can also explicitly list the individual test method names, in case you want to control that order as well by: addTest(FirstTest.class, "testM1", "testM2");

Selecting the right tests for various configs

Imagine you have one test case that contains a lot of tests, but you'd like to run them in various configurations: stable, all, really stable, etc. This is the recommended structure for your tests

public class CheckIDEWorks extends NbTestCase {
  public CheckIDEWorks(String n) { super(n); }

  public void testBasics() { ... }
  public void testMore() { ... }
  public void testRest() { ... }
}

The previous example define a regular test case with a set of methods that define behaviour of all your tests. However, it is not executed by the infrastructure by default, as its name does not end with "Test", and as such it is not recognized as test. You can do that, if you want, but suppose you want to create three different configurations for these tests. Then you can define own classes:

public class Stable { 
  public static Test suite() {
    return NbModuleSuite.create(
      NbModuleSuite.emptyConfiguration()
        .addTest(CheckIDEWorks.class, "testBasics", "testMore")
    );
  }
}
public class ReallyStable { 
  public static Test suite() {
    return NbModuleSuite.create(
      NbModuleSuite.emptyConfiguration()
        .addTest(CheckIDEWorks.class, "testBasics")
    );
  }
}
public class AllTest { 
  public static Test suite() {
    return NbModuleSuite.create(
      NbModuleSuite.emptyConfiguration()
        .addTest(CheckIDEWorks.class) // without a list of testcases, all are executed
    );
  }
}

Now, you can just predefine various configs that you wish to be executed. So you can have a test config for stable, as well as commit set of tests in your nbproject/project.properties:

test.config.stable.includes=**/Stable.class 
test.config.commit.includes=**/ReallyStable.class 

You may, but need not specify config for all tests, by default every class with name that ends with Test is included, which in our example means just AllTest. With a style like this you can create as many tests as you want, group them into execution units according to their stability and just by changing properties specify which execution units shall be executed when.

How to set up and run Tomcat, Glassfish, ...

To get an auto access to servers(Tomcat, Glassfish, JBoss) you should extend J2eeTestCase instead of JellyTestCase. It provides methods

 Configuration addServerTests(Configuration, String...)
 Configuration addServerTests(Server server, Configuration, String...)
Server
is an enum of actually supported servers. The methods are trying to find server using system properties and register it into the IDE. If there is a required server, test's are added into configuration, if there is no required server {emptyTest} is added so that the configuration would be runnable but no failure appeared.

If no server is required, no server is registered in IDE.

You can also use methods
isRegistered(Server server), getServerNode(Server server) 
these are supposed to by used in test runtime. If you need to run your test on all servers use
addServerTests(...)
and start test with different server settings each time. To set server path start test using ant command
 ant -Dtest-qa-functional-sys-prop.tomcat.home=/instalace/tomcat test 
or set {tomcat.home}, jboss.home, glassfish.home properties in different way.

To be done

  • (13ad8e27dcf0) do not fail whole build when modules.list is set to some module, which have e.g. only unit tests. E.g.
ant -Dnetbeans.dest.dir=/space/simpletests/netbeans \
  -Dmodules.list=java2/org-netbeans-modules-ant-grammar

fails with

BUILD FAILED
/space/simpletests/build.xml:47: The following error occurred while executing this line:
/space/simpletests/all-tests.xml:61:
Invalid file: /space/simpletests/qa-functional/java2/org-netbeans-modules-ant-grammar
  • (WONTFIX, feature of hudson itself) setting JDK for execution (already tracked) - serious for us
  • (ecf494ffc72b) Have an option to run several testcases within the same suite without NetBeans restarts
  • (8572baa87847) remove usage of modules.patches.* in NbModuleSuite (package-private access from tests to their modules is not going to be possible anyway), and test that tests can access impl classes with the public access modifier in any modules or tests declared in tests deps
  • (P3) make running tests from test distribution work for external modules which are not included in the target platform and thus have no module configs
  • (P3) consider providing support for JUnit 4-style tests (using annotations) in NbTestCase
  • (6ca3fcb52d31) classLoader doesn't load all classes needed in form module

To be done by QA after merge

  • (P1) There are a lot of functional tests that will need a new suite method:
  hg loc -r tip '*/test/qa-functional/src/**Test.java'
  
 Some already have a suite method, which will need to be rewritten.
  • As soon as we migrate performance tests, Marian and his guys will take care of the rest
  • (P1) a number of functional tests cannot run because deps on nbjunit or java.j2seproject
 are missing <recursive/> and thus cause NoClassDefFoundErrors at runtime
 (e.g. on o.n.insane or projectui classes)
  • (P3) update nbbuild/hudson/trunk to run all C/V tests under new regime
  • to start with, ant commit-validation must not use XTest
  • check up on functional tests relying on Glassfish or WTK
  • (P3) delete */test/build*.xml and */test/cfg-*.xml
  • In many cases (e.g. uml.kit) these files are the only source of info
  about the classpath of functional tests
  (nbproject/project.* were never correctly updated).
  • (P3) xml/test/qa-lib should probably be moved to a normal place (in which case try reenabling tests in xml.text, xml.tools, and xml.tools.java).
 Generally there are some odd test dirs:
  ls -1d */test/*/src | egrep -v 'unit|qa-functional'
  
  • (P3) fix tests that do something silly with the data dir:
  fgrep xtest.data */nbproject/project.properties
  

After merge, should also update TestDistribution.

Binary test distribution without X

QA is going to migrate to new test infrastructure and run there tests without X. New BTD (without X) is a little bit different (different targets, properties,..) List of properties we are going to use when we migrate to new BTD:

  • test.types - to determinate test type we want to run (previously xtest.testtype)
  • test.modules - to specify list of modules for which we want to run test (previously xtest.modules.list) e.g. org.netbeans.modules.ant.grammar
  • test.clusters - does not need to be specified when you specify test.modules; Use in case you want to run test for whole cluster, e.g. test.clusters=platform*
  • test.config - to distinguish between various test configurations - defined in nbproject/project.properties (previously done by configuring cfg xml file)
  • test.disable.fails - we have to set this property while running on Hudson in order to distinguish failure of test vs failure of test run.


Some notes

XTest features

Many modules have both unit and qa-functional test dirs:

ls -1d `hg loc -r tip '*/test/qa-functional/src/**Test.java' \
  | perl -pi -e 's!/qa-functional/src/.*!/unit/src!g' | sort | uniq`

jemmy.log gets correctly written to test workdir. Seems that screen.png screenshots get saved correctly too (JellyTestCase.runBare). Will need some way to collect workdirs and publish them. Minimally, include e.g. */build/test/work/ in Hudson artifacts. Nicer would be to have a plugin which collects them and links to them from test results.

Binary test distribution creates complete dist for all clusters (easy to pick apart manually if desired).

Related issues

test.unit.cp calculation ignores Class-Path extensions (now fixed in branch)

Cannot set bigger -mx for tests (fixed in branch)

Do not include nbjunit in test CP without explicit decl (fixed in branch, so long as you specify at least dep on junit)

openide/loaders QAFunctional tests are marked as uncompilable (seems fine in branch)

Implement qa-functional tests node (now fixed in branch)

Errors in execution of binary tests distribution should be passed to parent ant task

Wrong class path order set while running JUnit test via NB default ant scripts

Support for Emma coverage in external NBM projects

Jirka's comments

  • xtest and results: screenshot, jemmy.log, timeout - OK
  • xtest and results: keeping whole userdir - it lays around, Hudson needs to pick it up
  • xtest and config: no configs right now

Tom Wheeler's comments

We develop several platform-based applications and do a lot of testing at work. I make the following suggestions based on our experience:

  • There should be support for various (and potentially arbitrary) types of tests such as unit, functional, performance and so on. XTest currently supports this although the default testing support of the IDE/harness supports only unit tests at present.
  JesseGlick: only unit and qa-functional test roots will be supported.
  TomWheeler: It is still possible to do this anyway, I will document how elsewhere in the Wiki
  • It should be easy -- or at least possible -- for us to add support for additional testing libraries. While we use JUnit, Jemmy and Jellytools, we also use other libraries such as Fit. Ideally this could be supported by giving us some mechanism to add our own JAR files to the test classpath and a corresponding hook into an Ant task for running such tests.
  JesseGlick: just add them to CP.

Tomas Musil's comments to testdistribution (related to build#29)

  • P1 do not fail whole build when modules.list is set to some module, which have e.g. only unit tests. E.g.
ant -Dnetbeans.dest.dir=/space/simpletests/netbeans \
  -Dmodules.list=java2/org-netbeans-modules-ant-grammar

fails with

BUILD FAILED
/space/simpletests/build.xml:47: The following error occurred while executing this line:
/space/simpletests/all-tests.xml:61:
Invalid file: /space/simpletests/qa-functional/java2/org-netbeans-modules-ant-grammar
  JesseGlick: should be easy enough to fix.
  • seems like BUG - provide a possibility not fail whole build if some tests fail - we need to distinguish states "test ran with failures" vs. "test did not ran for some reason" i.e. omit target failtests - not sure what exactly test.disable.fails affects, but it does not seem to control only if failure of test causes failure of build. E.g. when i run
ant -Dnetbeans.dest.dir=/space/simpletests/netbeans \
  -Dmodules.list=ide9/org-netbeans-modules-utilities -Dtest.disable.fails=true

test gets stuck and ends with some o.n.jemmy.TimeoutExpiredException, while running only

ant -Dnetbeans.dest.dir=/space/simpletests/netbeans \
  -Dmodules.list=ide9/org-netbeans-modules-utilities

ends with normal test error and build failure.

  JesseGlick: should be easy to fix with JUnitReportSubant.
  • P3 nice to have: some kind of clean target which would remove results from previous testrun
  JesseGlick: ant clean does not work?
  • P4 mechanism of failure detection is not good - it does not respect results from current test run. I.e. $RESULTS/failed-test-var can be from previous testrun, my current testrun does not have any error/failure, but build fails with message that there were some failures.
  • P1 setting JDK for execution (already tracked) - serious for us

Petr Zajac's comments

We run tests for Instant JChem in our private hudson by using junit harness. Hudson was not able to kill tree of processes. Look at discussion thread. Xtest is little more reliable in killing processes.
  JesseGlick: submit a patch for Hudson.
  

Oleg Khokhlov's comments

* P1 tests fail if you add more then one class to suite:
        ...
        public static Test suite() {
             NbTestSuite s = new NbTestSuite("UI Responsiveness J2SE Menus suite");
             s.addTest(NbModuleSuite.create(MainMenu.class, ".*", ".*"));
             s.addTest(NbModuleSuite.create(MainSubMenus.class, ".*", ".*"));
             ...

IZ bug: 134525


Tom Wheeler's feedback using this for a platform app:

These are my first impressions after nominally getting simpletests working for a platform app.

  • Removing a module from the suite (via the suite customizer dialog) does not remove it as a functional test dependency for modules in that suite.
  JesseGlick: can file as a low-priority bug in apisupport/project.
  • It would be great if there was a well-known directory (like suite/testlibs) in which we could place JARs that will automatically be visible in the test classpath(s). Otherwise, you have to define properties -- although this is not hard, it's bad for those who make platforms because Ant properties are immutable and downstream app developers cannot (easily) override or append to a property set upstream.
  JesseGlick: no plans to implement. Add to each module as needed.
  TomWheeler: If I were to implement, would you be interested in including this in the harness?
  JesseGlick: probably not, unless there is a clearer justification for the complexity. There is already a configurable test classpath per module.
Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo