NbmPackageStability

Revision as of 20:25, 13 December 2011 by Jglick (Talk | contribs)

Contents

Introduction

Overview of current state

A module may export a certain list of public packages to any caller, or certain friend packages to an enumerated list of callers only, or no packages at all. Both Ant- and Maven-based build harnesses verify at build time that a calling module has permission to access packages from an API module via a regular "specification version" dependency, and the same check is done at runtime by the module system.

A spec dep is on a given specification version or newer, together with a major release version or (occasionally) range. The major release version is not related in any way to the specification version, but must be a natural number (if omitted it is typically considered to be zero). When compiling against m/1 2.3, the build harness will by default create a dep "m/1 > 2.3" matching any version of m with major equal to 1 and spec equal to 2.3 or higher. It is possible to explicitly request a major range like "m/1-2 > 2.3" matching m with major 1 and spec 2.3+, or m with major 2 and any spec. The expectation is that compatible API changes increment the spec version (somehow); whereas incompatible changes increment the major version (and perhaps also change the spec version): CompatibilityPolicy.

An implementation dep is on an exact implementation version of a module. By default that is generated by the build harness, e.g. as the current date, but it may be given explicitly, typically as a natural number (unrelated to either the major or spec versions). When using an impl dep, a public type in any package may be accessed (at build time and runtime).

Problems with current state

Usage from external modules

The most serious problem: external module developers frequently need to call some semi-API which is either not officially exported, or exported only to some friends. This is very common for IDE plugins, since plenty of widely used modules offer only friend packages: xml.multiview, gsf.testrunner, jumpto, and so on. The developer is usually implored to either ask for friend status from the maintainer of the API module, or to initiate a review of the module to make it public. Either way, this advice is useless for building against the current NetBeans release, and it discourages people from experimenting with potentially useful APIs which might be improved by their feedback. The problem is especially notable for API modules which are not actively maintained, both because getting changes made to the export metadata is hard, and because the API is probably not changing much from release to release anyway.

The usual workarounds are:

  1. Declaring an implementation dependency, and shipping a different copy of the importing module for each NetBeans release, even when (in the usual case) its code is unchanged. Makes it difficult to build the product in a single job when using the Maven harness, and essentially impossible when using the Ant harness.
  2. Like #1 but not bothering to ship different versions; users of a different NetBeans release must recompile the module from sources.
  3. Calling the entire API using reflection.
  4. Using reflection to hack into the NB module system and convince it that the importer is a "friend". This has become the preferred technique for those who know about it, but of course it subverts the original intent of protecting a user from an unlinkable module - and introduces an unannounced dependency on internal implementation classes in the module system.

Version madness

Having three different kinds of versions that may control module-to-module dependencies - specification, major release, and implementation - is rather confusing. It is not obvious, for example, that upgrading from 1.1 -> 2.0 in spec version is not essentially any different than 1.1 -> 1.2; or that "m/1-2 > 1.3" will match m/2 in version 1.0.

Another minor point of confusion is that "m/1-2 > 1.3" matches 1.3 itself, i.e. '>' should really be '≥'.

Difficulties of implementation dependencies

Using impl deps is awkward in the best of circumstances. When the exporting module is cooperative, it will declare a numeric implementation version, then use spec.version.base to ensure that changes to that version automatically change the spec version too. (See DevFaqImplementationDependency for background.) Then the importer must declare an impl dep, which is difficult in the Maven harness (requires special metadata).

The importer might accidentally begin using packages which the exporter did not mean to make available even to this importer.

When publishing a numeric impl version, it is all too easy to forget to increment it when making changes, so some dependencies are often stuck on the same impl dep for years - even while the actual API has changed considerably.

Mixing public and friend packages

Declaring a list of friends for a module prevents you from also declaring public packages. (This is essentially because a module has just one version and we would not want to mark a major release version increment when only changing the friend packages.) Since it is commonplace for an API module to need to export a special SPI to certain callers, but which should not be part of the general API, there are various workarounds known:

  1. Add a public package with a name implying implementation and hope only the right modules call it. If an incompatible change needs to be made in it, make a compatible-style increment to the spec version and just hope the calling modules are updated in tandem (since the module system will not understand). (e.g.: org.openide.util.lookup.implspi)
  2. Like #1, but override module.javadoc.packages so it does not appear in API documentation (but will still appear in code completion etc.). (e.g.: org.netbeans.modules.progress.spi)
  3. Like #1, but use ad-hoc techniques at runtime to prevent calls from unexpected places, like checking the name of a subclass or checking a stack trace. (e.g.: org.openide.util.lookup.implspi.AbstractServiceProviderProcessor)
  4. Try to factor the friend API code into a third module which both the public API module and SPI implementors will depend upon. Not always straightforward, and rarely done in practice.
  5. Declare an impl dep. This is forbidden in the platform cluster, especially due to the problems interoperating with OSGi, but still common in other clusters. (e.g.: org.netbeans.modules.project.uiapi)

Minor incompatible changes

Most incompatible API changes are quite small in scope and are likely to affect very few clients; if they were more important, effort would be put into avoiding them. Yet the module system has only a binary notion "compatible" vs. "incompatible" change. So the API developer who needs to make a slightly incompatible change is forced to choose between

  1. Make the API change and just mark a regular specification version increase. Increment the versions of any callers which are known to need updating and hope that users receive the updates. Hope that there are no other callers needing updates out there.
  2. Increment the major release version. Forces those callers which really needed updates to be updated, which is good. But forces the great majority of callers to declare a dep on the new version (or a range) and release a new binary, which is potentially very disruptive.

Long-lost friends

OpenIDE-Module-Friends lists can easily include obsolete entries, since there is no check for unused friends. The entries which are netbeans.org modules are useless, since you can just look them up using the golden file; those which are external modules are potentially useful if you want to confer with external module developers about possibly incompatible changes, but it is not necessarily clear whether such friends still exist, or even where they live. For example, org.netbeans.modules.scala.project has been moved around, org.netbeans.modules.fortress.project is essentially dead, and org.netbeans.modules.javafx.dataloader is gone, so the declared friend list for org.netbeans.modules.gsf is no longer very useful.

OSGi translation

Creating OSGi equivalents of module dependency declarations is challenging when the major release version is involved, resulting in hacks like translating NetBeans 1.1 to OSGi 101.1.

Friend packages are not possible in OSGi mode; they are treated as public.

Implementation dependencies are also not available in OSGi mode, so modules with an integral impl version are simply treated as exporting everything.

Proposal: package stability classifications

Rather than forcing a module to declare a single stability level for its entire exported surface, permit individual packages to be separately marked as stable, good enough for friends but not the general public, or not stable at all. Drop the explicit list of friends. Enforce a uniform, intuitive version range model that matches these expectations; but provide a graceful way to deal with the relatively rare case of a dependency falling outside that range.

Marking stability of export

API_Design#Life-cycle_of_an_API lists various stability classifications for APIs. We should permit a module developer to explicitly indicate on each package their expectations for its stability over time.

One attractive option is to mark this right in the source code, as suggested in MNBMODULE-148. The advantage is that the metadata stays close to the sources, and (with @Documented) becomes part of any generated Javadoc. For example, in package-info.java:

@Exported(Stability.STABLE)
package org.netbeans.api.something;
import org.openide.modules.Exported;
import org.openide.modules.Stability;

where Stability could also be (say) FRIEND, or PRIVATE (the default if unmarked).

The build harness might need to collect stability classifications for the packages in the module and summarize them in the generated manifest; that depends on how they are consumed by importers and the module system (below).

TBD how to deal with binary packages included in a library wrapper module, which obviously could not be annotated in this way. Perhaps these could be listed manually in the source manifest in some way, or passed to the build harness.

A more conservative change would be to keep OpenIDE-Module-Public-Packages with its current meaning (assuming OpenIDE-Module-Friends is deprecated); and add OpenIDE-Module-Friend-Packages. A module could export public packages, friend packages, or both (but they may not overlap).

Marking stability of import

The next step is to determine what kinds of APIs a module is importing. Assume that major release versions and implementation versions are both deprecated (OpenIDE-Module-Build-Version taking over the informational role in the log file sometimes carried by OpenIDE-Module-Implementation-Version today), leaving the specification version as the sole determiner of API compatibility. (It is already the sole indication of an "upgrade" from the standpoint of Auto Update.)

For each exporter, i.e. module being compiled against, collect a list of packages actually being used. (See below about constants.) Pick the least stable such package, and include a dependency clause in the generated manifest according to the exporter's name, the exporter's current spec version (see policy below), and that minimum stability. Unlike current module dependencies, the stability would result in an implicit or explicit closed-open version range. Support the exporter is m 1.1 (~ 1.1.0); then

  1. stable -> [1.1,2.0)
  2. friend -> [1.1,1.2)
  3. private -> [1.1,1.1.1)

Thus a use of a stable (or "official") API would have a similar effect as a spec version dependency today (with no declared major release version range): 2.0 would be considered to break compatibility for the official API. This is similar to the conventions used in Semantic Versioning.

For a friend API, routine trunk development could include incompatible changes if convenient, so long as the spec version is updated and friends are updated to match; patches made to a release branch would be assumed compatible for friends, so only the edited module needs to be published on Auto Update. In the case of private packages, it is assumed that any specification version change, even patch updates, might involve incompatible changes.

Note that there is no explicit friend list, so any external module developer is potentially on equal footing as far as using APIs which are not official but are in practice stable enough to be useful.

It is TBD what form this new dependency metadata would take. The most conservative approach is to keep the current header but permit the dependency clause to be richer, perhaps using a range syntax:

OpenIDE-Module-Module-Dependencies: org.openide.nodes ~ [7.2,8.0)

Another possibility is to begin using the OSGi header, which already supports this:

Require-Bundle: org.openide.nodes;bundle-version="[7.2,8.0)"

Handling possibly broken dependencies

(In this section we assume that an API developer updates the spec version of a module in accordance with the stability level of the package being changed; see below.)

Since incompatible API changes do happen, and on occasion an importer would actually be broken by the change, we want to ensure that a user is never offered a new or updated module which begins throwing linkage errors when it is loaded.

(You might think that a broken module would just throw one such error and then "stop", but this is not so. A module may have many functional entry points, and some may be invoked very frequently, not necessarily in response to an explicit user gesture. For example, an editor hint which made use of an unlinkable utility class could throw a fresh exception every time the editor pauses for a rescan!)

Conversely, since most incompatible changes do not affect most callers, we do not want to gratuitously prevent a user from loading a module which in fact would work fine just because the version numbers do not match.

The solution is to degrade gracefully. If the (spec) version of the exporter is within the importer's requested version range, permit it to be loaded as now without any further ado - trusting that versions have been updated in a compliant way. If it is older than the lower end of the range, mark the importer as unloadable, as now - the user should look for a newer version (usually one would have been offered by Auto Update anyway, or they are simply using too old a platform).

If the exporter's version is newer than the upper end of the importer's range, rather than immediately marking the importer as unloadable, go through all the classes in its module and try to resolve them. If any have linkage errors, mark the importer as incompatible as we do today and refuse to enable it; if not, go ahead and load it, while logging a warning. There are a few subtleties here:

  1. While this situation can be expected to be relatively uncommon, forcing resolution of every class in the importer during every startup is probably too expensive. Could be solved by caching the "known-good" state, with the cache to be invalidated on a subsequent startup if either the exporter or importer is changed.
  2. Once a class is resolved, it remains referenced by its class loader for the rest of the session, consuming valuable PermGen heap space. It may be feasible to use a special "throwaway" loader for doing these checks, so that no extra memory is permanently consumed.
  3. The JVM's binary compatibility specification perversely considers it compatible for an implementer of an interface if a method is added to the interface. Since this will still result in NoSuchMethodError's at runtime if called, which are just as unacceptable as any other linkage error, the compatibility checker would need to explicitly ensure that all methods from all nominally implemented interfaces are in fact implemented.
  4. It may be necessary to check the existence of Java members referenced by XML layers, such as the instanceCreate attribute.
  5. Usages of compile-time constants would not be detected unless worked around; see below.

The result is the best of both worlds: the same startup performance as today for the usual case that everything is compatible, but the ability to load potentially incompatible plugins at a small cost if they are not really broken.

Behavioral API changes

This system only covers incompatibilities at the level of Java signatures; it does not handle incompatible changes to the behavior of methods, or to extralinguistic contracts such as XML layer locations or system properties. On the other hand, this may not be so bad:

  1. Most of the ways in which modules communicate with one another is through regular Java binary interfaces.
  2. Changes to behavior can be accompanied by changes in signature, which makes it clear at the language level what is different. In fact it is often possible and desirable to expose complex semantics in the Java signature, where it can be managed using common tools like Javadoc, Find Usages, and so on.
  3. If such contracts do get broken, the consequences are generally not as dire as a linkage error. The plugin might not work well or at all, but it is unlikely to ruin the whole application by throwing constant exceptions or the like.

Compile-time constants

Detecting packages in use by a build-time dependency could be done at the source level, but much more easily in bytecode after compilation. The main difference is that compile-time constants - static final fields of primitive or String type whose initializers follow certain restrictions - are inlined by current versions of javac, losing information about the source dependency. Uses of constants would also not be visible to a linkage checker.

Possible workarounds:

  1. Detect usages of constants in source code. Does not help with the linkage checker unless these are also recorded somewhere in the JAR.
  2. Force the field to not be initializable as a constant, so the caller's bytecode must refer to it. Easy enough to do with a utility method that escapes javac's constant analysis, though you then lose the ability to see the value of constant in Javadoc.
  3. Try to get javac itself to stop inlining constants - perhaps a request for JDK 8, but not viable in the short term.

This issue may not matter much anyway, since if the constant is inlined there will be no runtime reference either. The case of interest is when an API developer wishes to change a constant, which is typically an incompatible change that ought to be treated as such. In many such cases other parts of the API signature will be changed incompatibly as well.

Policies

Transitioning

TBD what the best way would be to transition from the current state. The module system should still be able to handle legacy release and implementation versions, and both impl and legacy (non-range) spec dependencies, and the generic build harnesses (both Ant and Maven) should continue to support legacy mode; but we would want to begin using the new convention for all netbeans.org modules and their interdependencies.

For modules declaring no major release version today, probably this is easy: deps on "m > 1.30" just switch quietly to (e.g.) "m ~ [1.30,2)" and keep similar semantics. OpenIDE-Module-Friends is dropped from the manifest, and the entries in OpenIDE-Module-Public-Packages are marked either stable or friend accordingly.

For modules declaring a major release version today, it is TBD whether that version should be dropped, or retained for compatibility with old external modules but ignored when creating new-style dependencies.

The relatively rare major release range dependencies can simply be replaced with a plain module dep (i.e. on the current version).

OpenIDE-Module-Implementation-Version would be removed from all modules explicitly declaring one (probably this should be made into a build error in projectized.xml if present), and all impl deps converted to plain module deps after ensuring that the packages actually being used are marked as friend stability. spec.version.base=1.5.0 would be converted back to plain OpenIDE-Module-Specification-Version: 1.6 (and again its usage should be made a build error for netbeans.org modules).

Recording baseline version

#70917 in the Ant harness would mean source projects would no longer need to explicitly state the version of each dependency, which is very often wrong (usually too old) anyway. In the case of the Maven harness, the exporter is picked out of the repository by the given version and compiled against, so it makes sense to just declare that as the base version.

CompatibilityPolicy#Versioning_impact could be simplified this way: when making an incompatible change, you would simply update the first component of the affected module's spec version, and that is all. Other modules in the same source tree would automatically begin to request the new version, so no patch to them would be needed, nor would they need to have been "prepared" using major release version ranges. Externally hosted modules would incur the one-time cost of a compatibility check unless and until recompiled against your new release.

Permitted dependencies inside netbeans.org sources

The netbeans.org build harness (projectized.xml) should simply prohibit any dependencies on private packages. This means that the generic harness should have some configurable minimum stability level for all dependencies, below which package usage results in a build error.

The nbms-and-javadoc Hudson job already publishes a "golden" file [1] listing friend dependencies. The same format could track dependencies referring to friend packages.

Signature testing

A build harness could try to enforce change policy using signature tests. Currently the Hudson jobs permit any compatible change, sending a report from nbms-and-javadoc; and forbid any incompatible change relative to the last release. For purposes of this proposal, it would be better to permit any compatible change so long as the specification version is incremented, and also any incompatible change so long as the spec version is incremented in the appropriate digit (according to the package's declared stability level), reporting either kind of change in nbms-and-javadoc. Doing this from the Maven harness would be easy enough (since older releases of a plugin are available in the repository), but it is trickier in an Ant build since you would need to somehow retain the signature file from the previous version.

Usage warnings

When building an external module, i.e. the harness default when not used in netbeans.org sources, deps on private packages should be permitted but issue a warning. Deps on friend packages might merit an informational note but not a warning.

The Whitelist API, originally created for cloud-based servers offering restricted Java Platform APIs, could probably be used to warn the developer about usage of private APIs right in the Java editor: bug #201453.

The current Add Dependency dialog for the Ant harness differentiates "API" from "Non-API" modules based on the existence of public packages, or friend packages for which the proposed importer is in fact a friend. This would probably need to be relaxed so that any module offering either stable- or friend-stability packages would be offered in the "API" list: by using one of the friend packages, you become a friend ex post facto. TBD what should be changed in the Maven plugin; see: MavenNBM4#Dependency_management_and_project_structure

OSGi impact

When using NetBeansInOSGi, the MakeOSGi task would be able to directly translate NetBeans version range dependencies to Require-Bundle entries. An OSGi container would simply lack the graceful fallback for out-of-range dependencies. This would help align NetBeans and OSGi dependencies more closely.

OSGiAndNetBeans#Runtime could be simplified in the same way; unnatural dependencies like [2.1,100) would become the natural [2.1,3.0).

Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo