Signature tests verify that public and protected members of non-private classes in each module do not change or remain compatible with previous version. These tests are part of the build or the continuous build process. The testing infrastructure is currently based on APITest, an extension and temporary fork of sigtest tool which is used as part of the JCP test compatibility kits to verify that the API exposed by a JSR implementation does not diverge from the reference implementation.
Turned on Nov 7, 2012:
For a while the VersioningPolicy prescribes how to properly version modules and their API changes. However the policy has for a long time only been applied to modules with stable API. From time to time this leads to no such method or class errors like Issue 186233 which need to be evaluated by platform/module system component owner.
Moreover the introduction of incremental binary build AutoUpdateTask requires more proper attitude to versioning. Only then one can be sure that incremental binary build really links properly.
For the above reasons ita consistency check that verifies minimal versioning consistency has been introduced. The HgParallelProjectIntegration runs:
- Check on all modules exposing non-private packages
- Compares the spec version.
The check is integral part of push-* jobs and if it detects a failure, the push job fails. To recover you are supposed to increment spec version of your module whenever you change its API. Increment spec versions of modules that depend on your API change as well (not enforced right now). Run following command to simulate the check locally:
# first of all make sure your system is built OK # Then either: # generate appropriate JUnit testing report for all modules: $ ant check-sigtests-version # # or run the 'version' check for a single module $ ant -f your.module/build.xml check-sigtest -Dsigtest.check.type=versioncheck
Continuous Check for Changes
The continuous build process runs a sigtest build that compares APIs in a new build with APIs of the one of the previous build. Whenever there is a change (either compatible or incompatible) a mail is sent:
- if the module has public packages - to the firstname.lastname@example.org mailing list
- if the module has friends in other clusters - to the email@example.com mailing list
- if the module has friends only in the same cluster - to appropriate cvs@<module>.netbeans.org mailing list
- if the module has neither friends, neither public packages - it is skipped from the check
Appropriate owners of each module and tech leads of each area are expected to review the change and prevent or revert unintentional changes.
The signature golden files are generated by
$ ant -f nbbuild/build.xml gen-sigtests
and compared by
$ ant -f nbbuild/build.xml gen-sigtests
A module can configure sigtest behavior by modifying its project.properties:
# redefine the email where reports are sent firstname.lastname@example.org # don't fail if the generation of the report fails sigtest.gen.fail.on.error=false
At the time of release
When doing a release it is important to record the actual signatures of stable modules (and to some extend also under development ones), so we know what to keep backward compatibility with. To generate these signatures do:
$ hg update -C releaseXY_base $ ant clean $ ant build $ ant gen-sigtests-release $ hg st|cut -c 3-| xargs zip /tmp/sig.zip
The last target will modify a lot of module/nbproject/*.sig files and possibly generate some new. These files shall be integrated into trunk build. For example by:
$ cd <team-repository> $ hg update -c default $ unzip /tmp/sig.zip $ hg add . $ hg ci -m "Signatures as of release X.Y" $ hg push
Check for Incompatible Changes Since Previous Version
There is another check for API signature compliance: A check that the APIs are compatible with a previous released version. This check is available to every module, which can make it part of its build by generating its API snapshot by:
ant netbeans gen-sigtest
This creates a file nbproject/org-netbeans-module-name.sig which can be checked into the version control system as a reference point. This check is part of continuous build as well: every module with its golden file in nbproject/org-netbeans-module-name.sig is checked for backward compatibility and failures are included in continuous build test results. To run the same test for one module on local computer just use:
$ ant netbeans check-sigtest
in your module directory and API of your module will be verified. Of course, only if the signature file exists.
Who needs the Golden Signature File
Not every module with an API needs this golden file; currently only those modules that produced a stable API in release 6.1 have it. The plan is to update these golden files with a new version as soon as it is released, so future checks are done against latest released version.
If an incompatible change is made (discouraged and subject to API review process) the golden file needs to be edited by hand to remove the element that is no longer present in the API.
Checking for all changes from previous release
In case you have the sig file in nbproject directory generated at the time of previour release (all public APIs have it), you can see all changes since that time by:
$ ant -f yourmodule/build.xml check-sigtest -Dsigtest.check.type=strictcheck
This check for incompatibilities is done by NetBeans' own version of an OpenJDK tools. This adds a standard Ant task to check for binary backward compatibility, mutual signature compatibility. The binary compatibility check ignores generic types. Its sources are distributed together with its Mercurial repository and are available under GPL version 2. Builds are available from our hudson builder.