UICommitValidation
UI Commit Validation Tests for Continuous NetBeans Build
This is description of UI functional tests which run as part of NetBeans continuous build. These tests should satisfy that new build is not corrupted. For every cluster exists a suite of tests which cover basic vital functionality. Each responsible engineer from the list below should be subscribed to broken_builds@netbeans.org mailing list. Test failures are reported by emails with subject "Hudson build became unstable: trunk #". Engineer should look at simple results and respond to the mailing list. For hints how to evaluate test results look at this document.
Test Suite Execution Rules
- All developers have to execute commit validation test suite before committing their changes
- If tests fail developers have to analyze the results and decide whether problem is in his code or in commit validation tests
- If developer thinks the problem is in tests he has to file P1 issue for failing tests (use component: ide, subcomponent: commit validation) and wait for QE response. The current owner of commit validation tests is Ivan Sidorkin.
- Responsible QE engineer has to respond during 24 hours (or on Monday in case test fails on Friday) and approve/disapprove commit
- If QE does not respond in 24 hours developer can disable the test and commit his changes, and don't forget to email Ivan Sidorkin, so he can fix the test and enable it soon.
Clusters
- cnd OK
- enterprise OK
- ide OK
- identity ?
- j2ee OK</div>
- mobility OK
- profiler !</div>
- ruby OK
- uml X</div>
- visualweb X
- php OK</div>
CndAnchor
</span>
| Cluster | cnd |
|---|---|
| Description | CND cluster |
| Status | OK</div> Running on deadlock. |
| Location | cnd.kit/test/qa-functional/src/org/netbeans/test/cnd/CNDValidation.java |
| How to run | ant -f cnd.kit/build_UICommitValidation.xml -Dtest.config=commit test |
| Results | http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/testReport/org.netbeans.test.cnd/ |
| Responsible Engineer | Alexander Pepin |
| Responsible Manager | Julie Guzzetta |
EnterpriseAnchor
</span>
| Cluster | enterprise (soa, xml) |
|---|---|
| Description | Enterprise cluster including soa and xml |
| Status | OK</div> Running on deadlock. |
| Location | xml.schema/test/qa-functional/src/org/netbeans/test/xml/schema/AcceptanceTestCase.java soa.kit/test/org/netbeans/test/qa-functional/src/xslt/AcceptanceTestCase.java |
| How to run | ant -f soa.kit/build_UICommitValidation.xml -Dtest.config=uicommit -Dcontinue.after.failing.tests=true test ant -f xml.schema/build_UICommitValidation.xml -Dtest.config=uicommit -Dcontinue.after.failing.tests=true test |
| Results | http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/testReport/org.netbeans.test.xml.schema/ http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/testReport/org.netbeans.test.xslt/ |
| Responsible Engineer | XML: Michael Nazarov |
| Responsible Manager | XML: Martin Balin |
IdeAnchor
</span>
| Cluster | ide (java, nb, platform) |
|---|---|
| Description | Standard !NetBeans IDE including ide, java, nb and platform clusters. |
| Status | OK</div> Running on deadlock. |
| Location | java.kit/test/qa-functional/src/org/netbeans/test/ide/IDEValidation.java |
| How to run | ant -f java.kit/build_UICommitValidation.xml -Dtest.config=commit -Dcontinue.after.failing.tests=true test |
| Results | http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/testReport |
| Responsible Engineer | Ivan Sidorkin |
| Responsible Manager | Petr Blaha |
IdentityAnchor
</span>
| Cluster | identity |
|---|---|
| Description | Identity cluster. |
| Status | ?</div> Need to assign responsible engineer and create tests. |
| Location | |
| How to run | |
| Results | |
| Responsible Engineer | |
| Responsible Manager | |
JtwoeeAnchor
</span>
| Cluster | j2ee |
|---|---|
| Description | J2EE cluster |
| Status | OK</div> Running on deadlock. |
| Location | j2ee.kit/test/qa-functional/src/org/netbeans/test/j2ee/J2EEValidation.java |
| How to run | ant -f j2ee.kit/build_UICommitValidation.xml -Dtest.config=j2ee -Dcontinue.after.failing.tests=true -Dtest-qa-functional-sys-prop.com.sun.aas.installRoot=/<your_workdir>/glassfish test |
| Results | http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/testReport/org.netbeans.test.j2ee/ |
| Responsible Engineer | Ivan Sidorkin |
| Responsible Manager | Petr Blaha |
MobilityAnchor
</span>
| Cluster | mobility |
|---|---|
| Description | Mobility cluster |
| Status | OK</div> Running on deadlock. |
| Location | mobility.project/test/qa-functional/src/org/netbeans/test/mobility/ValidationTest.java |
| How to run | ant -f mobility.project/build_UICommitValidation.xml -Dtest.config=uicommit -Dcontinue.after.failing.tests=true -Dwtk.dir=<path_to_wtk> test |
| Results | http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/testReport/org.netbeans.test.mobility/ |
| Responsible Engineer | Ivan Sidorkin |
| Responsible Manager | Marian Mirilovic |
ProfilerAnchor
</span>
| Cluster | profiler |
|---|---|
| Description | Profiler cluster |
| Status | !</div> Only two basic test cases are running on deadlock. |
| Location | profiler/test/qa-functional/src/org/netbeans/test/profiler/ProfilerValidation.java |
| How to run | ant -f profiler/build_UICommitValidation.xml -Dtest.config=commit -Dcontinue.after.failing.tests=true test |
| Results | http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/testReport/org.netbeans.test.profiler/ |
| Responsible Engineer | Ivan Sidorkin |
| Responsible Manager | Marian Mirilovic |
RubyAnchor
</span>
| Cluster | ruby |
|---|---|
| Description | Ruby cluster |
| Status | OK</div> Running on deadlock. |
| Location | ruby.kit/test/qa-functional/src/org/netbeans/test/ruby/RubyValidation.java |
| How to run | ant -f ruby.kit/build_UICommitValidation.xml -Dtest.config=ruby -Dcontinue.after.failing.tests=true test |
| Results | http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/testReport/org.netbeans.test.ruby/ |
| Responsible Engineer | Ivan Sidorkin |
| Responsible Manager | Petr Blaha |
UmlAnchor
</span>
| Cluster | uml |
|---|---|
| Description | UML cluster |
| Status | X Not yet migrated. |
| Location | uml.kit/test/qa-functional/src/org/netbeans/test/uml/validation/UMLValidationTests.java |
| How to run | ant -f xtest/instance/build_UICommitValidation.xml -Dxtest.config=commit-validation-uml runtests |
| Results | http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/artifact/xtest/instance/results/index.html |
| Responsible Engineer | - |
| Responsible Manager | - |
VisualwebAnchor
</span>
| Cluster | visualweb |
|---|---|
| Description | Visual web cluster |
| Status | OK Running on deadlock. |
| Location | visualweb.kit/test/qa-functional/src/org/netbeans/test/sanity/SanityTests.java |
| How to run | ant -f visualweb.kit/build_UICommitValidation.xml -Dtest.config=uicommit test (Detailed instructions) |
| Results | http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/testReport |
| Responsible Engineer | - |
| Responsible Manager | Petr Blaha |
PHPAnchor
</span>
| Cluster | php |
|---|---|
| Description | PHP cluster |
| Status | OK Running on deadlock. |
| Location | php.editor/test/qa-functional/src/org/netbeans/test/php/Commit.java |
| How to run | ant -f php.editor/build_UICommitValidation.xml -Dtest.config=commit -Dcontinue.after.failing.tests=true test |
| Results | |
| Responsible Engineer | Michael Nazarov |
| Responsible Manager | Martin Balin |
</span>
Infrastructure
NetBeans continuous build runs on Hudson server. Build is controlled by nbbuild/hudson/trunk script. To add tests for a new cluster you have to modify nbbuild/hudson/trunk. To add test for existing cluster you can modify nbbuild/project.properties file in your module. Similarly you can disable tests if you comment it out or remove it.
Special setup
- GlassFish - for J2EE tests there is needed GlassFish application server. It is automatically installed and cleaned before every test run. It is controlled by nbbuild/hudson/initAppserver.sh script. There must exist an installer jar corresponding to AS_BINARY property in this script.
- WTK - for mobility tests it is needed WTK. wtk.dir property set in nbbuild/hudson/trunk script points to /hudson. Zipped WTK must be in this directory and it has to be named exactly wtk22_linux.zip because it is hard coded in mobility.project/test/build-qa-functional.xml.
Run all tests at once
To run all tests at once you can use the following command:
ant commit-validation
To run only unit tests you can use the following command:
ant -Dtest.type="test-unit" commit-validation
Test results history
OBSOLETE, needs to be corrected.
To be able to track history of test failures, there is the testsHistory project. It runs after every successful trunk build. It copies results of tests into testsHistory workspace and prepares results to be processed by XTest PES. To setup testsHistory project do the following in your NetBeans working repository:
cd xtest ant makedist ant makedist-pes
Copy xtest/xtest-distribution.zip, xtest/pes/xtest-pes-distribution.zip, build_UICommitValidation.xml, pes-config_UICommitValidation.xml and xtest-pes_UICommitValidation.sh to /hudson/workdir/jobs/testsHistory and do the following on deadlock.netbeans.org:
cd /hudson/workdir/jobs/testsHistory mkdir xtest-distribution unzip -d xtest-distribution xtest-distribution.zip mkdir xtest-pes unzip -d xtest-pes xtest-pes-distribution.zip rm *.zip cp build_UICommitValidation.xml pes-config_UICommitValidation.xml xtest-pes cp xtest-pes_UICommitValidation.sh xtest-pes/bin
Then configure Hudson project. You can see configuration config_UICommitValidation.xml. Also set to build testsHistory project after trunk project.
Test sandbox
OBSOLETE, needs to be corrected.
Project http://deadlock.netbeans.org/hudson/job/testsSandbox can be used to try new tests before you add them into official suite.
Important links and files
- http://deadlock.netbeans.org/hudson/job/trunk/ - NetBeans continuous build project
- nbbuild/hudson/trunk - shell script which controls continuous build
- nbbuild/hudson/initAppserver.sh - shell script to init GlassFish application server for tests
- nbbuild/hudson/run-vw-sanity.sh - shell script to run tests for visualweb cluster
- http://deadlock.netbeans.org/hudson/job/trunk/lastSuccessfulBuild/testReport/ - test report from last build
- test evaluation - hints how to evaluate tests results
(partly obsolete)
