We have long had a general VersioningPolicy describing how to maintain APIs and make compatible changes. There was also a note on making incompatible changes but it did not cover any real policy.
Types of incompatible change
There are several basic grades of incompatible API changes, from mild to severe in effects.
Sometimes an entire class can be deprecated so that no one need refer to it any more when using the new recommended API. This is generally possible when the class contains either static utility code or an old SPI, but is not directly referred to from other (non-deprecated) API classes. We can call such an API separable and changes to such APIs are relatively straighforward.
When a whole class is deprecated, it is best to move this class to a deprecated module. Over time we can collect deprecated classes and safely package them up. A deprecated module is marked with OpenIDE-Module-Deprecated: true and can include a localizable OpenIDE-Module-Deprecation-Message as well.
You can use the module refactoring facility to retain binary compatibility for old client modules still referring to the class in its original module. They can migrate on their own schedule. The deprecated module can be made an autoload so it is not enabled in a standard IDE distribution, and eventually we can move it to the Update Center and out of the standard codebase.
Non-Java-language APIs can sometimes fall into this category as well. For example, a deprecated style of object registration could in some cases be supported only when a deprecated module is enabled, without cluttering the code which handles the new style of registration.
Many deprecated APIs cannot be easily separated without causing an incompatible API change and breaking old clients at some point. For example, a central class like FileObject cannot realistically be replaced with something else: far too many other APIs refer directly to FileObject in method signatures, and published NetBeans Platform documentation makes frequent reference to this class. However, certain methods like getPackageName (meaningless as of NB 4.0) can be deprecated and eventually deleted once everyone has had ample time to stop using them.
To quote the Java guide, deprecation is used when an API
- is insecure, buggy, or highly inefficient;
- is going away in a future release;
- or encourages bad coding practices.
Most deprecations will involve the actual @Deprecated annotation. (The @deprecated Javadoc tag should be used as well, to give instructions on what to do instead.) In some cases, an API involves a non-Java-language construct yet needs to be deprecated; for example, placement of XML layer entries into the wrong folder. In these cases, platform code should issue a warning into the log file noting the problem, the source (culpable module), and suggested fix.
Prior to this writing (NB 6.5) we have not had a general policy on incremental incompatible changes. As a result we were afraid of breaking anything by doing incompatible changes, which often resulted in the extreme position of never deprecating old, obsoleted ways of doing certain tasks. The result was that:
- New API users can become overwhelmed trying to find what code patterns are actually encouraged, making the Platform seem overly complex and hard to learn. (getLookup? getCookie? getCookieSet? ...)
- Nobody fixes obsoleted API usages even in our own codebase, which results in additional accumulation of usages of such APIs due to infamous, yet common, "copy-paste programming".
- Old usages of obsoleted APIs are not reported to developers when they migrated to a new version of the platform.
This resulted in the accumulation of old debris in many central APIs (some of it formally marked @Deprecated, some not). Moreover, even if such APIs became formally deprecated, the general injunction of not breaking backward compatibility resulted in keeping this debris around, some of it likely unused for years. The presence of this much deprecated code in our APIs has several ill effects for users of the NetBeans Platform, as well as NetBeans API developers:
- Modules are bloated by unused bytecode, and module projects by unused source code.
- In many cases non-deprecated code has to explicitly accommodate the existence of deprecated code, increasing complexity and thus maintenance burden and likelihood of bugs.
Of course, these ill effects have to be balanced in each case by the benefit to the maintainer of a client module in having the API left untouched. For example, an isolated static utility method of small size does not do much harm to the containing API, whereas if it was at one time heavily used, deleting the method could cause a lot of trouble for third-party module developers.
When deprecating an inseparable API in a phased manner, the author of the replacement API is responsible for making sure that the usage of obsoleted predecessor is eliminated from the NetBeans.org code base in a timely manner, or properly justifying why that need not happen.
It is proposed that after an inseparable API (class with outside references, method, ...) has been officially deprecated in a major NetBeans release, and was not in fact used by any modules present in that release, that it is permitted to be hidden from newly compiled sources (via PatchedPublic annotation) in the next release and then deleted completely in a subsequent one. Note that "major release" in this context does not refer to the version of NetBeans (e.g. NetBeans 5.5, 5.5.1, 6.0, etc.) but rather a change of enough signficance to warrant incrementing the cluster's version number (e.g. platform8, platform9, etc.).
The simplest variant is that the API can be hidden from newly compiled clients in the very next major release after its deprecation: a one-release grace source compatibility period. This may be considered too aggressive, however this is balanced by keeping the binary compatibility for yet another release: a two-release binary compatibility grace period, meaning the API must must exist for linkage (not compilation and unused) for at least two major releases before it can be deleted in a third. Of course, it is important to announce the expected time of hiding/deletion before the grace period begins.
The rest of this section describes the details of how such a policy could be implemented. Feedback from community members (mainly third-party module developers) is very much needed. This process also needs to be integrated with BackwardCompatibilityTesting.
Technical procedure for making an incompatible phased change
- Introduce the replacement API which is intended to cover all use cases covered by the old API.
- Mark the old API as deprecated. Properly document the intended replacement.
- Whenever possible, create an editor hint to identify usages of the deprecated API; and, in case there is a mechanical correspondence with the new API, also offer an automated fix for converting to the new idiom.
- Take responsibility for replacing all usages of the old API in modules hosted on netbeans.org. (Include the main and contrib repositories at least.)
- Directly fix everything you can safely and confidently change yourself.
- File P2 defects for what you cannot fix yourself, because the surrounding code is too subtle and unfamiliar. Be ready to provide advice to the assignee of the bug. Keep track of all such bugs using Issuezilla dependencies.
- Try to finish the transition within a single development cycle. (If you cannot do so, consider seriously whether you can expect third-party module developers to do the same!)
- If you find that the proposed replacement does not cover every use case after all, you need to either fix that immediately or defer the deprecation until you can.
- Increment the major release version of the API module, unless this has already been done for another phased change in the same module in the same release cycle.
- Announce timing of the expected deletion in your apichanges.xml entry. (You must have finished the replacement in netbeans.org modules first.)
- In next major release, make the source-incompatible, yet binary compatible change. (Make the method/field/class private and annotate it with @org.openide.modules.PatchedPublic. Any client wishing to recompile cannot use the method any more.)
- In the next major release, remove the private API.
The module system supports ranges in the major release version used in a module dependency. For example, org.openide.filesystems/1-2 > 7.42 means that this module should be compatible with org.openide.filesystems/1 in version 7.42 or later, or any version of org.openide.filesystems/2, but perhaps not with org.openide.filesystems/3 or higher. If we had a consistent expectation for the minimum grace period used for incompatible changes, then we could use ranges to good effect to preserve binary compatibility during the grace period.
For example, assuming a one-release grace period, any module which compiled without deprecation warnings against org.openide.filesystems/1 (and which produced no runtime warnings in the log) could safely declare org.openide.filesystems/1-2 > ... as its dependency. The module would then be usable without recompilation in the subsequent NetBeans release, making it easier to evaluate possible migration to a new release, and relieving the maintainer of the need to supply an update to users the moment the new release came out.
TBD whether it makes sense for the module development support to automatically introduce ranges like this when adding a module dependency to a project. The use of the range makes sense only if the developer is really committed to avoiding all usage of deprecated elements from the API.
The minimum grace period before deletion acceptable to the community needs to be determined. So far:
- one major release for deprecating and remove usage from netbeans.org modules
- one major release for source-incompatible, yet binary-compatible change via PatchedPublic annotation
- removal in subsequent major release
apichanges.xml needs a new syntax for an incompatible phased change with expected removal date/release. The existing incompatible option is a bit misleading here.
IDE/Platform release notes should link to this policy (or a summary of it), as well as the API change list, and should be reviewed for readability and accuracy by a qualified documenter (gwielenga comes to mind).
On occasion some part of an API, or a whole API, simply needs to be replaced with something quite different. In such a case there needs to be a "flag day" when all usages of the old API are replaced with the new API (or simply removed from the build or commented out).
For example, there was no plausible way to migrate smoothly from the 3.6 Filesystems-as-classpath paradigm to the 4.0 project system; nor from the MDR/JMI/Javamodel system to the "Retouche"/javac system.
Clearly such a major change has to be planned and communicated well in advance. Anyone wishing to migrate to the new version of NetBeans has to adapt their code.
If there is a policy of using major release version ranges to accommodate planned deletions of deprecated APIs (see above), then abrupt incompatible changes would need to be accompanied by increments of the major release version of the API module by 2 or more (i.e., one more than the minimum grace period length).