UI Commit Validation Tests for Continuous NetBeans Build
This is description of UI functional tests which run as part of !NetBeans continuous build. These tests should satisfy that new build is not corrupted. For every cluster exists a suite of tests which cover basic vital functionality. Each responsible engineer from the list below should be subscribed to broken_builds@netbeans.org mailing list. Test failures are reported by emails with subject "Hudson build became unstable: trunk #". Engineer should look at simple results or full XTest results and respond to the mailing list. For hints how to evaluate test results look at this document.
Clusters
CndAnchor
EnterpriseAnchor
IdeAnchor
IdentityAnchor
| Cluster | identity |
| Description | Identity cluster. |
| Status | ? Need to assign responsible engineer and create tests. |
| Location | |
| How to run | |
| Results | |
| Responsible Engineer | |
| Responsible Manager | |
JtwoeeAnchor
MobilityAnchor
ProfilerAnchor
RubyAnchor
UmlAnchor
VisualwebAnchor
<!-- possible statuses
ok OK
warning !
question mark ?
red X X
-->
Infrastructure
NetBeans continuous build runs on Hudson server. Build is controlled by nbbuild/hudson/trunk script. Tests are run by XTest which uses
master-config.xml. To add tests for a new cluster you have to modify nbbuild/hudson/trunk. To add test for existing cluster you can modify master-config.xml or appropriate cfg-qa-functional.xml file. Similarly you can disable tests if you comment it out or remove it.
Special setup
- GlassFish - for J2EE tests there is needed GlassFish application server. It is automatically installed and cleaned before every test run. It is controlled by nbbuild/hudson/initAppserver.sh script. There must exist an installer jar corresponding to AS_BINARY property in this script.
- WTK - for mobility tests it is needed WTK. wtk.dir property set in nbbuild/hudson/trunk script points to /hudson. Zipped WTK must be in this directory and it has to be named exactly wtk22_linux.zip because it is hard coded in mobility.project/test/build-qa-functional.xml.
Run all tests at once
To run alltest at once you can use the following command:
ant -f nbbuild/build.xml commit-validation
-D"xtest.userdata|com.sun.aas.installRoot"=<glassfish install dir>
-Dwtk.dir=<path to wtk22_win.zip or wtk22_linux.zip>
-Dxtest.config=commit-validation-all
Test results history
To be able to track history of test failures, there is the
testsHistory project. It runs after every successful
trunk build. It copies results of tests into testsHistory workspace and prepares results to be processed by XTest PES. To setup testsHistory project do the following in your NetBeans working repository:
cd xtest
ant makedist
ant makedist-pes
Copy xtest/xtest-distribution.zip, xtest/pes/xtest-pes-distribution.zip, build.xml
, pes-config.xml
and xtest-pes.sh
to /hudson/workdir/jobs/testsHistory and do the following on deadlock.netbeans.org:
cd /hudson/workdir/jobs/testsHistory
mkdir xtest-distribution
unzip -d xtest-distribution xtest-distribution.zip
mkdir xtest-pes
unzip -d xtest-pes xtest-pes-distribution.zip
rm *.zip
cp build.xml pes-config.xml xtest-pes
cp xtest-pes.sh xtest-pes/bin
Then configure Hudson project. You can see configuration config.xml
. Also set to build testsHistory project after trunk project.
Test sandbox
Project http://deadlock.netbeans.org/hudson/job/testsSandbox can be used to try new tests before you add them into official suite.
Important links and files