UI Commit Validation Tests for Continuous NetBeans Build

This is description of UI functional tests which run as part of NetBeans continuous build. These tests should satisfy that new build is not corrupted. For every cluster exists a suite of tests which cover basic vital functionality. Each responsible engineer from the list below should be subscribed to broken_builds@netbeans.org mailing list. Test failures are reported by emails with subject "Hudson build became unstable: trunk #". Engineer should look at simple results and respond to the mailing list. For hints how to evaluate test results look at this document.

Test Suite Execution Rules

• All developers have to execute commit validation test suite before committing their changes
• If tests fail developers have to analyze the results and decide whether problem is in his code or in commit validation tests
• If developer thinks the problem is in tests he has to file P1 issue for failing tests (use component: ide, subcomponent: commit validation) and wait for QE response. The current owner of commit validation tests is Ivan Sidorkin.
• Responsible QE engineer has to respond during 24 hours (or on Monday in case test fails on Friday) and approve/disapprove commit
• If QE does not respond in 24 hours developer can disable the test and commit his changes, and don't forget to email Ivan Sidorkin, so he can fix the test and enable it soon.

Clusters

• cnd OK
• enterprise OK
• ide OK
• identity ?
• j2ee OK
• mobility OK
• profiler !
• ruby OK
• uml X
• visualweb X
• php OK

Infrastructure

NetBeans continuous build runs on Hudson server. Build is controlled by nbbuild/hudson/trunk script. To add tests for a new cluster you have to modify nbbuild/hudson/trunk. To add test for existing cluster you can modify nbbuild/project.properties file in your module. Similarly you can disable tests if you comment it out or remove it.

Special setup

• GlassFish - for J2EE tests there is needed GlassFish application server. It is automatically installed and cleaned before every test run. It is controlled by nbbuild/hudson/initAppserver.sh script. There must exist an installer jar corresponding to AS_BINARY property in this script.
• WTK - for mobility tests it is needed WTK. wtk.dir property set in nbbuild/hudson/trunk script points to /hudson. Zipped WTK must be in this directory and it has to be named exactly wtk22_linux.zip because it is hard coded in mobility.project/test/build-qa-functional.xml.

Run all tests at once

To run all tests at once you can use the following command:

ant commit-validation

To run only unit tests you can use the following command:

ant -Dtest.type="test-unit" commit-validation

Test results history

OBSOLETE, needs to be corrected.

To be able to track history of test failures, there is the testsHistory project. It runs after every successful trunk build. It copies results of tests into testsHistory workspace and prepares results to be processed by XTest PES. To setup testsHistory project do the following in your NetBeans working repository:

cd xtest
ant makedist
ant makedist-pes

Copy xtest/xtest-distribution.zip, xtest/pes/xtest-pes-distribution.zip, build.xml, pes-config.xml and xtest-pes.sh to /hudson/workdir/jobs/testsHistory and do the following on deadlock.netbeans.org:

cd /hudson/workdir/jobs/testsHistory
mkdir xtest-distribution
unzip -d xtest-distribution xtest-distribution.zip
mkdir xtest-pes
unzip -d xtest-pes xtest-pes-distribution.zip
rm *.zip
cp build.xml pes-config.xml xtest-pes
cp xtest-pes.sh xtest-pes/bin

Then configure Hudson project. You can see configuration config.xml. Also set to build testsHistory project after trunk project.

Test sandbox

OBSOLETE, needs to be corrected.

Project http://deadlock.netbeans.org/hudson/job/testsSandbox can be used to try new tests before you add them into official suite.

Important links and files

• http://deadlock.netbeans.org/hudson/job/trunk/ - NetBeans continuous build project
• nbbuild/hudson/trunk - shell script which controls continuous build
• nbbuild/hudson/initAppserver.sh - shell script to init GlassFish application server for tests
• nbbuild/hudson/run-vw-sanity.sh - shell script to run tests for visualweb cluster
• http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/testReport/ - test report from last build
• test evaluation - hints how to evaluate tests results (partly obsolete)

Attachments

build.xml		4242 bytes
config.xml		1719 bytes
pes-config.xml		690 bytes
xtest-pes.sh		6310 bytes