---

UI Commit Validation Tests for Continuous NetBeans Build

This is description of UI functional tests which run as part of !NetBeans continuous build. These tests should satisfy that new build is not corrupted. For every cluster exists a suite of tests which cover basic vital functionality. Each responsible engineer from the list below should be subscribed to broken_builds@netbeans.org mailing list. Test failures are reported by emails with subject "Hudson build became unstable: trunk #". Engineer should look at simple results or full XTest results and respond to the mailing list. For hints how to evaluate test results look at this document.

Clusters

• cnd OK
• enterprise OK
• ide OK
• identity ?
• j2ee OK
• mobility OK
• profiler !
• ruby OK
• uml OK
• visualweb X

Cluster	cnd
Description	CND cluster
Status	OK Running on deadlock.
Location	ide.kit/test/qa-functional/src/org/netbeans/test/cnd/CNDValidation.java
How to run	ant -f xtest/instance/build.xml -Dxtest.config=commit-validation-cnd runtests
Results	http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/artifact/xtest/instance/results/index.html
Responsible Engineer	Maria Tishkova
Responsible Manager	Julie Guzzetta

Cluster	enterprise (soa, xml)
Description	Enterprise cluster including soa and xml
Status	OK Running on deadlock.
Location	xml.schema/test/qa-functional/src/org/netbeans/test/xml/schema/AcceptanceTestCase.java soa.kit/test/org/netbeans/test/qa-functional/src/xslt/AcceptanceTestCase.java enterprise/bpel/test/qa-functional/src/org/netbeans/test/bpel/verification/tests/VerificationTest.java
How to run	ant -f xtest/instance/build.xml -Dxtest.config=commit-validation-enterprise runtests
Results	http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/artifact/xtest/instance/results/index.html
Responsible Engineer	Andrei Chistiakov
Responsible Manager	Hong Lin

Cluster	ide (java, nb, platform)
Description	Standard !NetBeans IDE including ide, java, nb and platform clusters.
Status	OK Running on deadlock.
Location	ide.kit/test/qa-functional/src/org/netbeans/test/ide/IDEValidation.java
How to run	ant -f xtest/instance/build.xml -Dxtest.config=commit-validation-nb runtests
Results	http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/artifact/xtest/instance/results/index.html
Responsible Engineer	Ivan Sidorkin
Responsible Manager	Petr Blaha

Cluster	identity
Description	Identity cluster.
Status	? Need to assign responsible engineer and create tests.
Location
How to run
Results
Responsible Engineer
Responsible Manager

Cluster	j2ee
Description	J2EE cluster
Status	OK Running on deadlock.
Location	ide.kit/test/qa-functional/src/org/netbeans/test/j2ee/J2EEValidation.java
How to run	ant -f xtest/instance/build.xml -Dxtest.config=commit-validation-j2ee -D"xtest.userdata¦com.sun.aas.installRoot"=/<your_workdir>/glassfish runtests
Results	http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/artifact/xtest/instance/results/index.html
Responsible Engineer	Ivan Sidorkin
Responsible Manager	Petr Blaha

Cluster	mobility
Description	Mobility cluster
Status	OK Running on deadlock.
Location	mobility.project/test/qa-functional/src/org/netbeans/test/mobility/NewProjectFileTest.java
How to run	ant -f xtest/instance/build.xml -Dxtest.config=commit-validation-mobility -Dwtk.dir=<path_to_wtk> runtests
Results	http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/artifact/xtest/instance/results/index.html
Responsible Engineer	Lukas Hasik
Responsible Manager	Martin Balin

Cluster	profiler
Description	Profiler cluster
Status	! Only two basic test cases are running on deadlock because profiling freezes IDE.
Location	ide.kit/test/qa-functional/src/org/netbeans/test/profiler/ProfilerValidation.java
How to run	ant -f xtest/instance/build.xml -Dxtest.config=commit-validation-profiler runtests
Results	http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/artifact/xtest/instance/results/index.html
Responsible Engineer	Alexandr Scherbatiy
Responsible Manager	Mikhail Kondratyev

Cluster	ruby
Description	Ruby cluster
Status	OK Running on deadlock.
Location	ide.kit/test/qa-functional/src/org/netbeans/test/ruby/RubyValidation.java
How to run	ant -f xtest/instance/build.xml -Dxtest.config=commit-validation-ruby runtests
Results	http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/artifact/xtest/instance/results/index.html
Responsible Engineer	Ivan Sidorkin
Responsible Manager	Petr Blaha

Cluster	uml
Description	UML cluster
Status	OK Running on deadlock.
Location	uml.kit/test/qa-functional/src/org/netbeans/test/uml/validation/UMLValidationTests.java
How to run	ant -f xtest/instance/build.xml -Dxtest.config=commit-validation-uml runtests
Results	http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/artifact/xtest/instance/results/index.html
Responsible Engineer	Sherry Zhou
Responsible Manager	Tammera Krone

Cluster	visualweb
Description	Visual web cluster
Status	X Tests are failing and disabled on deadlock.
Location	visualweb.kit/test/test/qa-functional/src/org/netbeans/modules/visualweb/test/sanity/SanityTests.java
How to run	ant -f visualweb/ravebuild/build.xml build-test-tools commit-validation (Detailed instructions)
Results	http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/artifact/xtest/instance/results/vw/index.html
Responsible Engineer	Roman Mostyka
Responsible Manager	Petr Blaha

Infrastructure

NetBeans continuous build runs on Hudson server. Build is controlled by nbbuild/hudson/trunk script. Tests are run by XTest which uses master-config.xml. To add tests for a new cluster you have to modify nbbuild/hudson/trunk. To add test for existing cluster you can modify master-config.xml or appropriate cfg-qa-functional.xml file. Similarly you can disable tests if you comment it out or remove it.

Special setup

• GlassFish - for J2EE tests there is needed GlassFish application server. It is automatically installed and cleaned before every test run. It is controlled by nbbuild/hudson/initAppserver.sh script. There must exist an installer jar corresponding to AS_BINARY property in this script.
• WTK - for mobility tests it is needed WTK. wtk.dir property set in nbbuild/hudson/trunk script points to /hudson. Zipped WTK must be in this directory and it has to be named exactly wtk22_linux.zip because it is hard coded in mobility.project/test/build-qa-functional.xml.

Run all tests at once

To run alltest at once you can use the following command:

ant -f nbbuild/build.xml commit-validation
    -D"xtest.userdata|com.sun.aas.installRoot"=<glassfish install dir>
    -Dwtk.dir=<path to wtk22_win.zip or wtk22_linux.zip>
    -Dxtest.config=commit-validation-all

Test results history

To be able to track history of test failures, there is the testsHistory project. It runs after every successful trunk build. It copies results of tests into testsHistory workspace and prepares results to be processed by XTest PES. To setup testsHistory project do the following in your NetBeans working repository:

cd xtest
ant makedist
ant makedist-pes

Copy xtest/xtest-distribution.zip, xtest/pes/xtest-pes-distribution.zip, build.xml, pes-config.xml and xtest-pes.sh to /hudson/workdir/jobs/testsHistory and do the following on deadlock.netbeans.org:

cd /hudson/workdir/jobs/testsHistory
mkdir xtest-distribution
unzip -d xtest-distribution xtest-distribution.zip
mkdir xtest-pes
unzip -d xtest-pes xtest-pes-distribution.zip
rm *.zip
cp build.xml pes-config.xml xtest-pes
cp xtest-pes.sh xtest-pes/bin

Then configure Hudson project. You can see configuration config.xml. Also set to build testsHistory project after trunk project.

Test sandbox

Project http://deadlock.netbeans.org/hudson/job/testsSandbox can be used to try new tests before you add them into official suite.

Important links and files

• http://deadlock.netbeans.org/hudson/job/trunk/ - NetBeans continuous build project
• nbbuild/hudson/trunk - shell script which controls continuous build
• nbbuild/hudson/initAppserver.sh - shell script to init GlassFish application server for tests
• nbbuild/hudson/run-vw-sanity.sh - shell script to run tests for visualweb cluster
• http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/testReport/ - simple test report from last build
• http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/artifact/xtest/instance/results/index.html - XTest results from last build
• failures history matrix - history matrix with failures for last several builds
• test evaluation - hints how to evaluate tests results
• xtest/instance/master-config.xml - XTest master config file for configuring suites to run

Attachments