SignatureTest

(Difference between revisions)
(fixed broken link)
(At the time of release)
 
(14 intermediate revisions not shown)
Line 1: Line 1:
__NOTOC__
__NOTOC__
-
Signature tests verify that public and protected members of non-private classes in each module do not change or remain compatible with previous version. These tests are part of the build or the [http://deadlock.netbeans.org/hudson/ continuous build process]. The testing infrastructure is currently based on [[APITest | APITest]], an extension and temporary fork of [https://sigtest.dev.java.net/ sigtest tool] which is used as part of the [http://www.jcp.org/ JCP] test compatibility kits to verify that the API exposed by a JSR implementation does not diverge from the reference implementation.
+
Signature tests verify that public and protected members of non-private classes in each module do not change or remain compatible with previous version. These tests are part of the build or the [http://deadlock.netbeans.org/hudson/ continuous build process]. The testing infrastructure is currently based on [[APITest | APITest]], an extension and temporary fork of [http://sigtest.java.net/ sigtest tool] which is used as part of the [http://www.jcp.org/ JCP] test compatibility kits to verify that the API exposed by a JSR implementation does not diverge from the reference implementation.
 +
 
 +
== Use from Maven ==
 +
 
 +
See [[SigTest]] for the details.
== Consistency Check ==
== Consistency Check ==
-
'''Proposal May 18, 2010''':
+
'''Turned on Nov 7, 2012''':
-
For a while the [[VersioningPolicy]] prescribes how to properly version modules and their [[API]] changes. However the policy has for a long time only been applied to modules with stable [[API]]. From time to time this leads to ''no such method or class'' errors like {{iz|186233}}.
+
For a while the [[VersioningPolicy]] prescribes how to properly version modules and their [[API]] changes. However the policy has for a long time only been applied to modules with stable [[API]]. From time to time this leads to ''no such method or class'' errors like {{iz|186233}} which need to be evaluated by ''platform/module system'' component owner.  
-
Also the introduction of incremental binary build [[AutoUpdateTask]] requires more proper attitude to versioning. Only then one can be sure that incremental binary build really links properly.
+
Moreover the introduction of incremental binary build [[AutoUpdateTask]] requires more proper attitude to versioning. Only then one can be sure that incremental binary build really links properly.
-
For the above reasons it is time to introduce a consistency check that verifies, sooner than a changeset propagates to mail-silver repository via the [[HgParallelProjectIntegration]] that it is properly versioned:
+
For the above reasons ita consistency check that verifies minimal versioning consistency has been introduced. The [[HgParallelProjectIntegration]] runs:
-
* The check will run on all modules non-private exposing packages  
+
* Check on all modules exposing non-private packages  
-
* It will compare the spec version.
+
* Compares the spec version.
-
:* If the new spec is the same as previous, the check will fail if there is some [[API]] change (method or class)
+
:* If the new spec is the same as previous, the check fails if there is some [[API]] change (method or class)
-
:* If the new spec is higher, the check will pass if there is new method added or new class added
+
:* If the new spec is higher, the check will pass regardless of any [[API]] changes.
 +
 
 +
The check is integral part of push-* jobs and if it detects a failure, the push job fails. '''To recover''' you are supposed to increment spec version of your module whenever you change its API. Increment spec versions of modules that depend on your [[API]] change as well (not enforced right now). Run following command to simulate the check locally:
 +
 
 +
<source lang="bash">
 +
# first of all make sure your system is built OK
 +
# Then either:
 +
# generate appropriate JUnit testing report for all modules:
 +
$ ant check-sigtests-version
 +
#
 +
# or run the 'version' check for a single module
 +
$ ant -f your.module/build.xml check-sigtest -Dsigtest.check.type=versioncheck
 +
</source>
-
The check will be part of push-* jobs and if it detects a failure, the push job will fail.
+
<!--
 +
In case there is a strong need to disable this kind of version checking for an individual module (provided that the owners know how to version properly), one can add following into own ''nbproject/project.properties'' file: ''sigtest.check.type=check''. This is usefull mostly for modules with split packages.
 +
-->
==Continuous Check for Changes==
==Continuous Check for Changes==
Line 46: Line 64:
== At the time of release ==
== At the time of release ==
-
When doing a release it is important to record the actual signatures of ''stable'' modules (and to some extend also ''under development'' ones), so we know what to keep [[apidesign::BackwardCompatibility|backward compatibility]] with. To generate these signatures do:
+
When doing a release it is important to record the actual signatures of ''stable'' modules (and to some extend also ''under development'' ones), so we know what to keep [[apidesign::BackwardCompatibility|backward compatibility]] with. To generate these signatures we used to do following, but since [https://github.com/apache/incubator-netbeans/pull/1064 Apache NetBeans is in Git], it is better to follow the [https://github.com/apache/incubator-netbeans/pull/1064 PR-1064 steps]. Anyway here is the old recipe:
<source lang="bash">
<source lang="bash">
-
$ hg update -C releaseXY_base
+
$ hg update -C releaseXY_fcs
$ ant clean
$ ant clean
$ ant build
$ ant build
$ ant gen-sigtests-release
$ ant gen-sigtests-release
 +
$ hg st|cut -c 3-| xargs zip /tmp/sig.zip
</source>
</source>
Line 58: Line 77:
<source lang="bash">
<source lang="bash">
-
$ hg update tip
+
$ cd <team-repository>
 +
$ hg update -c default
 +
$ unzip /tmp/sig.zip
$ hg add .
$ hg add .
$ hg ci -m "Signatures as of release X.Y"
$ hg ci -m "Signatures as of release X.Y"

Current revision as of 04:42, 31 December 2018

Signature tests verify that public and protected members of non-private classes in each module do not change or remain compatible with previous version. These tests are part of the build or the continuous build process. The testing infrastructure is currently based on APITest, an extension and temporary fork of sigtest tool which is used as part of the JCP test compatibility kits to verify that the API exposed by a JSR implementation does not diverge from the reference implementation.

Use from Maven

See SigTest for the details.

Consistency Check

Turned on Nov 7, 2012:

For a while the VersioningPolicy prescribes how to properly version modules and their API changes. However the policy has for a long time only been applied to modules with stable API. From time to time this leads to no such method or class errors like Issue 186233 which need to be evaluated by platform/module system component owner.

Moreover the introduction of incremental binary build AutoUpdateTask requires more proper attitude to versioning. Only then one can be sure that incremental binary build really links properly.

For the above reasons ita consistency check that verifies minimal versioning consistency has been introduced. The HgParallelProjectIntegration runs:

  • Check on all modules exposing non-private packages
  • Compares the spec version.
  • If the new spec is the same as previous, the check fails if there is some API change (method or class)
  • If the new spec is higher, the check will pass regardless of any API changes.

The check is integral part of push-* jobs and if it detects a failure, the push job fails. To recover you are supposed to increment spec version of your module whenever you change its API. Increment spec versions of modules that depend on your API change as well (not enforced right now). Run following command to simulate the check locally:

# first of all make sure your system is built OK
# Then either:
# generate appropriate JUnit testing report for all modules:
$ ant check-sigtests-version 
#
# or run the 'version' check for a single module
$ ant -f your.module/build.xml check-sigtest -Dsigtest.check.type=versioncheck


Continuous Check for Changes

The continuous build process runs a sigtest build that compares APIs in a new build with APIs of the one of the previous build. Whenever there is a change (either compatible or incompatible) a mail is sent:

  • if the module has public packages - to the api-changes@netbeans.org mailing list
  • if the module has friends in other clusters - to the api-changes@netbeans.org mailing list
  • if the module has friends only in the same cluster - to appropriate cvs@<module>.netbeans.org mailing list
  • if the module has neither friends, neither public packages - it is skipped from the check

Appropriate owners of each module and tech leads of each area are expected to review the change and prevent or revert unintentional changes.

The signature golden files are generated by

$ ant -f nbbuild/build.xml gen-sigtests

and compared by

$ ant -f nbbuild/build.xml gen-sigtests

A module can configure sigtest behavior by modifying its project.properties:

# redefine the email where reports are sent
sigtest.mail=your.name@your.org
# don't fail if the generation of the report fails
sigtest.gen.fail.on.error=false

At the time of release

When doing a release it is important to record the actual signatures of stable modules (and to some extend also under development ones), so we know what to keep backward compatibility with. To generate these signatures we used to do following, but since Apache NetBeans is in Git, it is better to follow the PR-1064 steps. Anyway here is the old recipe:

$ hg update -C releaseXY_fcs
$ ant clean
$ ant build
$ ant gen-sigtests-release
$ hg st|cut -c 3-| xargs zip /tmp/sig.zip

The last target will modify a lot of module/nbproject/*.sig files and possibly generate some new. These files shall be integrated into trunk build. For example by:

$ cd <team-repository>
$ hg update -c default
$ unzip /tmp/sig.zip
$ hg add .
$ hg ci -m "Signatures as of release X.Y"
$ hg push

Check for Incompatible Changes Since Previous Version

There is another check for API signature compliance: A check that the APIs are compatible with a previous released version. This check is available to every module, which can make it part of its build by generating its API snapshot by:

ant netbeans gen-sigtest

This creates a file nbproject/org-netbeans-module-name.sig which can be checked into the version control system as a reference point. This check is part of continuous build as well: every module with its golden file in nbproject/org-netbeans-module-name.sig is checked for backward compatibility and failures are included in continuous build test results. To run the same test for one module on local computer just use:

$ ant netbeans check-sigtest

in your module directory and API of your module will be verified. Of course, only if the signature file exists.

Who needs the Golden Signature File

Not every module with an API needs this golden file; currently only those modules that produced a stable API in release 6.1 have it. The plan is to update these golden files with a new version as soon as it is released, so future checks are done against latest released version.

If an incompatible change is made (discouraged and subject to API review process) the golden file needs to be edited by hand to remove the element that is no longer present in the API.

Checking for all changes from previous release

In case you have the sig file in nbproject directory generated at the time of previour release (all public APIs have it), you can see all changes since that time by:

$ ant -f yourmodule/build.xml check-sigtest -Dsigtest.check.type=strictcheck


apitest tool

This check for incompatibilities is done by NetBeans' own version of an OpenJDK tools. This adds a standard Ant task to check for binary backward compatibility, mutual signature compatibility. The binary compatibility check ignores generic types. Its sources are distributed together with its Mercurial repository and are available under GPL version 2. Builds are available from our hudson builder.

Other NetBeans Tests

Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo