NbmPackageStability

Revision as of 16:38, 13 December 2011 by Jglick (Talk | contribs)

Contents

Introduction

Overview of current state

A module may export a certain list of public packages to any caller, or certain friend packages to an enumerated list of callers only, or no packages at all. Both Ant- and Maven-based build harnesses verify at build time that a calling module has permission to access packages from an API module via a regular "specification version" dependency, and the same check is done at runtime by the module system.

A spec dep is on a given specification version or newer, together with a major release version or (occasionally) range. The major release version is not related in any way to the specification version, but must be a natural number (if omitted it is typically considered to be zero). When compiling against m/1 2.3, the build harness will by default create a dep "m/1 > 2.3" matching any version of m with major equal to 1 and spec equal to 2.3 or higher. It is possible to explicitly request a major range like "m/1-2 > 2.3" matching m with major 1 and spec 2.3+, or m with major 2 and any spec. The expectation is that compatible API changes increment the spec version (somehow); whereas incompatible changes increment the major version (and perhaps also change the spec version): CompatibilityPolicy.

An implementation dep is on an exact implementation version of a module. By default that is generated by the build harness, e.g. as the current date, but it may be given explicitly, typically as a natural number (unrelated to either the major or spec versions). When using an impl dep, a public type in any package may be accessed (at build time and runtime).

Problems with current state

Having three different kinds of versions that may control module-to-module dependencies - specification, major release, and implementation - is rather confusing. It is not obvious, for example, that 1.1 -> 2.0 in spec version is not semantically any different than 1.1 -> 1.2; or that "m/1-2 > 1.3" will match m/2 in version 1.0. Creating OSGi equivalents of module dependency declarations is particular challenging when the major release version is involved, resulting in hacks like translating NetBeans 1.1 to OSGi 101.1.

Using implementation dependencies is awkward in the best of circumstances. When the exporting module is cooperative, it will declare a numeric implementation version, then use spec.version.base to ensure that changes to that version automatically change the spec version too. (See DevFaqImplementationDependency for background.) Then the importer must declare an impl dep, which is difficult in the Maven harness. And the importer might accidentally begin using packages which the exporter did not mean to make available even to this importer. Implementation dependencies are also not available in OSGi mode, so modules with an integral impl version are simply treated as exporting everything.

Declaring a list of friends for a module prevents you from also declaring public packages. (This is essentially because a module has just one version and we would not want to mark a major release version increment when only changing the friend packages.) Since it is commonplace for an API module to need to export a special SPI to certain callers, but which should not be part of the general API, there are various workarounds known:

  1. Add a public package with a name implying implementation and hope only the right modules call it. If an incompatible change needs to be made in it, make a compatible-style increment to the spec version and just hope the calling modules are updated in tandem (since the module system will not understand). (e.g.: org.openide.util.lookup.implspi)
  2. Like #1, but override module.javadoc.packages so it does not appear in API documentation (but will still appear in code completion etc.). (e.g.: org.netbeans.modules.progress.spi)
  3. Like #1, but use ad-hoc techniques at runtime to prevent calls from unexpected places, like checking the name of a subclass or checking a stack trace. (e.g.: org.openide.util.lookup.implspi.AbstractServiceProviderProcessor)
  4. Try to factor the friend API code into a third module which both the public API module and SPI implementors will depend upon. Not always straightforward, and rarely done in practice.
  5. Declare an impl dep. This is forbidden in the platform cluster, especially due to the problems interoperating with OSGi, but still common in other clusters. (e.g.: org.netbeans.modules.project.uiapi)

External module developers frequently need to call some semi-API which is either not officially exported, or exported only to some friends. This is very common for IDE plugins, since plenty of widely used modules offer only friend packages: XML Multiview, Test Runner, Maven, and so on. The developer is usually implored to either ask for friend status from the maintainer of the API module, or to initiate a review of the module to make it public. Either way, this advice is useless for building against the current NetBeans release, and it discourages people from experimenting with potentially useful APIs which might be improved by their feedback. The problem is especially grave for API modules which are not actively maintained, both because getting changes made to the export metadata is hard, and because the API is probably not changing much from release to release anyway. The usual workarounds are:

  1. Declaring an implementation dependency, and shipping a different copy of the importing module for each NetBeans release, even when (in the usual case) its code is unchanged. Makes it difficult to build the product in a single job when using the Maven harness, and essentially impossible when using the Ant harness.
  2. Like #1 but not bothering to ship different versions; users of a different NetBeans release must recompile the module from sources.
  3. Using reflection to hack into the NB module system and convince it that the importer is a "friend".
  4. Calling the entire API using reflection.

Most incompatible API changes are quite small in scope and are likely to affect very few clients; if they were more important, effort would be put into avoiding them. Yet the module system has only a binary notion "compatible" vs. "incompatible" change. So the API developer who needs to make a slightly incompatible change is forced to choose between

  1. Make the API change and just mark a regular specification version increase. Increment the versions of any callers which are known to need updating and hope that users receive the updates. Hope that there are no other callers needing updates out there.
  2. Increment the major release version. Forces those callers which really needed updates to be updated, which is good. But forces the great majority of callers to declare a dep on the new version (or a range) and release a new binary, which is potentially very disruptive.

Proposal: package stability classifications

Rather than forcing a module to declare a single stability level for its entire exported surface, permit individual packages to be separately marked as stable, good enough for friends but not the general public, or not stable at all. Enforce a uniform, intuitive version range model that matches these expectations; but provide a graceful way to deal with the relatively rare case of a dependency falling outside that range.

Marking stability of export

API_Design#Life-cycle_of_an_API lists various stability classifications for APIs. We should permit a module developer to explicitly indicate on each package their expectations for its stability over time.

One attractive option is to mark this right in the source code, as suggested in MNBMODULE-148. The advantage is that the metadata stays close to the sources, and (with @Documented) becomes part of any generated Javadoc. For example, in package-info.java:

@Exported(Stability.STABLE)
package org.netbeans.api.something;
import org.openide.modules.Exported;
import org.openide.modules.Stability;

where Stability could also be (say) FRIEND, or PRIVATE (the default if unmarked).

The build harness might need to collect stability classifications for the packages in the module and summarize them in the generated manifest; that depends on how they are consumed by importers and the module system (below).

TBD how to deal with binary packages included in a library wrapper module, which obviously could not be annotated in this way. Perhaps these could be listed manually in the source manifest in some way, or passed to the build harness.

Marking stability of import

The next step is to determine what kinds of APIs a module is importing. Assume that major release versions and implementation versions are both deprecated (OpenIDE-Module-Build-Version taking over the informational role in the log file sometimes carried by OpenIDE-Module-Implementation-Version today), leaving the specification version as the sole determiner of API compatibility. (It is already the sole indication of an "upgrade" from the standpoint of Auto Update.)

For each exporter, i.e. module being compiled against, collect a list of packages actually being used. (*) Pick the least stable such package, and include a dependency clause in the generated manifest according to the exporter's name, the exporter's current spec version (see policy below), and that minimum stability. Unlike current module dependencies, the stability would result in an implicit or explicit closed-open version range. Support the exporter is m 1.1 (~ 1.1.0); then

  1. stable -> [1.1,2.0)
  2. friend -> [1.1,1.2)
  3. private -> [1.1,1.1.1)

Thus a use of a stable (or "official") API would have a similar effect as a spec version dependency today (with no declared major release version range): 2.0 would be considered to break compatibility for the official API. This is similar to the conventions used in Semantic Versioning.

For a friend API, routine trunk development could include incompatible changes if convenient, so long as the spec version is updated and friends are updated to match; patches made to a release branch would be assumed compatible for friends, so only the edited module needs to be published on Auto Update. In the case of private packages, it is assumed that any specification version change, even patch updates, might involve incompatible changes.

Note that there is no explicit friend list, so any external module developer is potentially on equal footing as far as using APIs which are not official but are in practice stable enough to be useful.

It is TBD what form this new dependency metadata would take. The most conservative approach is to keep the current header but permit the dependency clause to be richer, perhaps using a range syntax:

OpenIDE-Module-Module-Dependencies: org.openide.nodes ~ [7.2,8.0)

Another possibility is to begin using the OSGi header, which already supports this:

Require-Bundle: org.openide.nodes;bundle-version="[7.2,8.0)"

(*) This could be done at the source level, but more probably in bytecode after compilation. The only significant difference is that compile-time constants - static final fields of primitive or String type whose initializers follow certain restrictions - are inlined by current versions of javac, losing information about the source dependency. It may be possible to detect these in source code; or force the field to not be initializable as a constant, so the caller's bytecode must refer to it. But it may not matter anyway, since if the constant is inlined there will be no runtime reference either. The only case of interest is when an API developer wishes to change a constant, which is typically an incompatible change; but in most such cases other parts of the API signature will be changed too.

Handling possibly broken dependencies

(In this section we assume that an API developer updates the spec version of a module in accordance with the stability level of the package being changed; see below.)

Since incompatible API changes do happen, and on occasion an importer would actually be broken by the change, we want to ensure that a user is never offered a new or updated module which begins throwing linkage errors when it is loaded. (*) Conversely, since most incompatible changes do not affect most callers, we do not want to gratuitously prevent a user from loading a module which in fact would work fine just because the version numbers do not match.

The solution is to degrade gracefully. If the (spec) version of the exporter is within the importer's requested version range, permit it to be loaded as now without any further ado - trusting that versions have been updated in a compliant way. If it is older than the lower end of the range, mark the importer as unloadable, as now - the user should look for a newer version (usually one would have been offered by Auto Update anyway, or they are simply using too old a platform).

If the exporter's version is newer than the upper end of the importer's range, rather than immediately marking the importer as unloadable, go through all the classes in its module and try to resolve them. If any have linkage errors, mark the importer as incompatible as we do today and refuse to enable it; if not, go ahead and load it, while logging a warning. There are a few subtleties here:

  1. While this situation can be expected to be relatively uncommon, forcing resolution of every class in the importer during every startup is probably too expensive. Could be solved by caching the "known-good" state, with the cache to be invalidated on a subsequent startup if either the exporter or importer is changed.
  2. Once a class is resolved, it remains referenced by its class loader for the rest of the session, consuming valuable PermGen heap space. It may be feasible to use a special "throwaway" loader for doing these checks, so that no extra memory is permanently consumed.
  3. The JVM's binary compatibility specification perversely considers it compatible for an implementer of an interface if a method is added to the interface. Since this will still result in NoSuchMethodError's at runtime if called, which are just as unacceptable as any other linkage error, the compatibility checker would need to explicitly ensure that all methods from all nominally implemented interfaces are in fact implemented.
  4. It may be necessary to check the existence of Java members referenced by XML layers, such as the instanceCreate attribute.
  5. Usages of compile-time constants would not be detected unless worked around as above.

The result is the best of both worlds: the same startup performance as today for the usual case that everything is compatible, but the ability to load potentially incompatible plugins at a small cost if they are not really broken.

Note that this system only covers incompatibilities at the level of Java signatures; it does not handle incompatible changes to the behavior of methods, or to extralinguistic contracts such as XML layer locations or system properties. On the other hand, this may not be so bad:

  1. Most of the ways in which modules communicate with one another is through regular Java binary interfaces.
  2. Changes to behavior can be accompanied by changes in signature, which makes it clear at the language level what is different. In fact it is often possible and desirable to expose complex semantics in the Java signature, where it can be managed using common tools like Javadoc, Find Usages, and so on.
  3. If such contracts do get broken, the consequences are generally not as dire as a linkage error. The plugin might not work well or at all, but it is unlikely to ruin the whole application by throwing constant exceptions or the like.

(*) You might think that a broken module would just throw one such error and then "stop", but this is not so. A module may have many functional entry points, and some may be invoked very frequently, not necessarily in response to an explicit user gesture. For example, an editor hint which made use of an unlinkable utility class could throw a fresh exception every time the editor pauses for a rescan!

Policies

Recording baseline version

#70917 in the Ant harness would mean source projects would no longer need to explicitly state the version of each dependency, which is very often wrong (usually too old) anyway. In the case of the Maven harness, the exporter is picked out of the repository by the given version and compiled against, so it makes sense to just declare that as the base version.

Permitted dependencies inside netbeans.org sources

The netbeans.org build harness (projectized.xml) should simply prohibit any dependencies on private packages. This means that the generic harness should have some configurable minimum stability level for all dependencies, below which package usage results in a build error.

The nbms-and-javadoc Hudson job already publishes a "golden" file [1] listing friend dependencies. The same format could track dependencies referring to friend packages.

Making incompatible changes

XXX delta to CompatibilityPolicy

A build harness could try to enforce change policy using signature tests. Currently the Hudson jobs permit any compatible change, sending a report from nbms-and-javadoc; and forbid any incompatible change relative to the last release. For purposes of this proposal, it would be better to permit any compatible change so long as the specification version is incremented, and also any incompatible change so long as the spec version is incremented in the appropriate digit (according to the package's declared stability level), reporting either kind of change in nbms-and-javadoc. Doing this from the Maven harness would be easy enough (since older releases of a plugin are available in the repository), but it is trickier in an Ant build since you would need to somehow retain the signature file from the previous version.

Usage warnings

XXX for external modules, warn about private deps XXX consider using whitelist API

Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo