---

Authors: MilosKleint, DavidKonecny, JanRojcek

Status: version 0.1

Out-of-the-Box Versionability Proposal

This document first lists current problems and then suggests a solution.

Usecases

Couple of high level use cases which illustrate problems of NetBeans 6.0.

UC1: IDE-provided library

Start NetBeans IDE on JDK 1.5; create new J2SE project; create new JPanel in the project.

Problem(s): This results into project with references to three IDE-provided libraries: Swing Layout Extentions, Junit 3.8.1 and JUnit 4.1. These libraries are referenced by name. If the project is send to non-NetBeans user (via VCS or ZIP file) the project Ant script will fail because library references cannot be resolved.

Alternative scenario resulting into the same problem is: create Web Application project with a framework - the frawework is added to project as IDE-provided library referenced by name.

UC2: user-defined library

Create user defined Library "jakarta-commons" containing jars, javadoc and sources from http://commons.apache.org/collections/ and http://commons.apache.org/lang/ and http://commons.apache.org/logging/; create new J2SE project; add "jakarta-commons" library to compilation classpath; create a Java file with call to the library.

Problem(s): This results into project with reference to user defined library. If the project is send to non-NetBeans user (via VCS or ZIP file) the Ant script will fail because library reference cannot be resolved. If the project is send to other NetBeans user (via VCS or ZIP file) the situation depends on whether this user already created user defined library with the name "jakarta-commons" or not. Most probably they did not and in such a case project opening marks project as broken and project compilation fails until the problem is resolved.

UC3: raw jar

Download http://commons.apache.org/lang/; create J2SE project in NetBeans; add commons-lang-X.X.jar to compilation classpath; create a Java file with call to the jar.

Problem(s): Because of AlwaysRelativeCollocationQuery the reference to commons-lang-X.X.jar is always relative regardless whether it make sense or not. If the project is send to NetBeans or non-NetBeans user, via VCS or ZIP file, it may or may not work depending on whether commons-lang-X.X.jar file is part of VCS/ZIP or not.

Alternative scenario resulting into the same problem is: create J2SE project A and J2SE project B and make A depending on B. If both A and B are in the same VCS/ZIP everything works.

UC4: J2SE platform

This usecase is identical to UC2 but is listened separately because of different solution.

Start NetBeans IDE on JDK 1.6; in Java Platform Manager create new platform for J2SE 1.4; create 'legacy' J2SE project and change its platform to J2SE 1.4 platform.

Problem(s): the same as in UC2.

UC5: J2EE server

Create Web Application with Glassfish server without any framework.

Problem(s): This results into project with name reference to J2EE platform and server classpath being used for project compilation. If the project is send to non-NetBeans user (via VCS or ZIP file) the Ant script will fail because J2EE platform cannot be resolved. If the project is send to other NetBeans user (via VCS or ZIP file) the situation depends on whether the user has a J2EE platform (of project's J2EE version) defined in NetBeans or not. If they do not have one the project opening marks project as broken and project compilation fails until the problem is resolved. If they have one the project opening may succeed but project compilation/deployment may still fail because different J2EE platforms come with different set of jars. For example JSF jars are not included by default when creating Web application with JSF for Glassfish v2 J2EE5 server because they are part of Glassfish server. Sending such a project to user with Tomcat 6 J2EE5 server (or just changing project's server from Glassfish to Tomcat) results into compilation errors because JSF jars will not be available.

UC6: NB6 project upgrade

This is not much of a usecase but rather testcase - open NB6 project and make it sharable one.

Solution

It is simply impossible to guess user's intentions with the project. The UI should unobtrusively ask whether the project is planned to be shared or not and then warn/guide user when something could break sharability. Sharable project will use sharable libraries instead of IDE libraries, that it library definition file and library jars itself are defined in project's file structure or within its proximity. Fixing current disability to attach Javadoc and sources to raw jars should significantly decrease the need for libraries. In long term Javadoc and sources association should be done on demand - editor code completion should allow user to "Attach Javadoc"; Go To source or debugger's Step Into should allow user to "Attach Sources". Giving user control over how files are referenced from project (and also showing these paths in UI instead of always fully qualifying them) should help them make right decision or at least make them aware of potential problems.

Terminology

Library definition - definition of library that includes library's name, classpath, source path, Javadoc, etc. It can be stored in XML element or just set of Java properties.

Library JARs - physical jars of a library; can be also zip files in case of Javadoc or sources.

Libraries definition file - Java properties file (or xml eventually) which may contain zero and more library definitions. The difference from library catalogue is that this file can be stored alongside project sources and shared via VCS/ZIP. When 'shared' is used in relation to this file it always means sharing file via VCS/ZIP with other users. A definition file could be also 'shared' by two projects but verb 'use' will be always used in such a case.

Libraries catalogue - New term coined for the global IDE libraries maintainable via Library Manager (menu Tools->Libraries). The catalogue is a special-case of "Libraries definition file".

General concept

• Project has new attribute which is set by user: is project sharable or not. Based on this value we provide better defaults and guidance in various places.

• Sharable project cannot use libraries catalogue but instead has one (and only one) libraries definition file which is stored alongside of project sources. "Alongside" is defined by CollocationQuery implementations and user preference.

• Non-sharable project behaves as NB6 project - it uses directly libraries catalogue and it cannot use libraries definition file.

• A libraries definition file can be used by one or more projects. It should always be used just by projects that can safely access it by relative path.

• Library JARs shall also be always safely accessed by relative path from libraries definition file. The default location for jars is the same directory or a subdirectory where the definition file resides. Any updates of shared library definition shall check for safe relative path access and warn / offer to copy the bits to the a correct location.

• It is possible to import a library from libraries catalogue to libraries definition file. The result of import is that #1) library definition is copied into shared libraries definition file and #2) library JARs are copied to subfolder of where libraries definition file resides.

• Libraries added to sharable project automatically by the IDE (see for example UC1), results in a copy of library definition and library jars from the libraries catalogue to libraries definition file as described in previous bullet point.

• User can define a new library in libraries catalogue just once and import the definition and jars into the shared projects. Or they can define the library just in shared libraries definition file.

New project wizard

There are two additions:

• sharable checkbox - the initial value can be guessed in some cases, for example if the project is being created under a VCS root. There is open issue listed later in this document whether the option should be by default true of false.

• folder to create libraries definition file in or an existing definition file to use - this option is enabled only for sharable projects. Initial value is "../libs" which suggests to create libraries definition file in libs folder which is at the same level as project folder. For example creating two sharable projects with this default value results into following structure where both projects are using the same libraries definition file from libs folder:

|-NetBeansProjects
  |
  |-Project1
  |
  |-libs
  |
  |-Project2

Illustrative screenshot:

Project Properties customizer

Libraries panel of project properties customizer is enhanced for shared project to look like:

It allows to change libraries definition file. The actual management of libraries is done in Tools/Libraries dialog. The picture above is old/wrong.

Similarly Add Library button from classpath tabbed pane shows dialog listing only shared library definitions:

and allows to create new sharable library definition or import global one.

Libraries definition file can be changed in project customizer but no checks are done regarding whether all currently used library definitions are available in new definition file or not. That is it's user's responsibility to resolve any potential problems and (at least in first version) IDE will not provide any assistance.

Project sharability state can be also changed anytime. It is not expect to be common action and that's why it is not available in project customizer but will be placed in menu as a "Make Project Non-Sharable" action. To make project non-sharable the shared libraries definition file is swapped for global libraries catalogue and again no checks are done - all libraries used in project's classpath must exist in libraries catalogue and it's up to user to resolve any potential problems.

For non-sharable project Libraries panel of project properties customizer looks slightly differently:

It allows user to make project sharable via 'Make Project Sharable' button. Reason for placing the button in the customizer is that there is suitable space in UI and to help user to discover this feature. All other behavior of non-sharable project is identical to NB6.

Newly added Edit button next to classpath items allows to edit association of Javadoc and sources with libraries or jars and is disabled for projects. Classpath item clearly indicates whether it has Javadoc/sources. Editing library shows:

and editing JAR items shows:

There is one more general change visible prominently in project customizer. Any path is displayed the way it is stored. If a JAR reference is stored as relative path then UI should show the relative value. Fully qualified path may be shown in tooltip if needed.

Import of a library from Libraries Catalogue

As mentioned previously Manage Libraries dialog for customization of shared libraries definition file allows to import libraries from global catalogue. Invoking import action shows list of libraries from libraries catalogue:

Selecting library will copy #1) library's definition to shared libraries definition file and #2) library's JARs to subfolder of where libraries definition file resides. The subfolder is named according to library name to prevent jar name collisions. For example adding IDE-provided Struts library to shared project with default location of libraries definition file will result into following diretory structure:

|-NetBeansProjects
  |
  |-Project1
  |
  |-libs
    |
    |-lib.properties
    |
    |-struts
      |
      |-struts.jar
      |
      |-commons-logging.jar
      |
      ...

Make Project Sharable action

User decides to make project shareable and denotes a location for the shareable library definition. UI is a dialog/wizard invoked from project Properties dialog. We help user to choose location for the shareable library definition file. The location is checked and list of all project artifacts (libraries and jars) is shown with suggested actions to make project sharable. The default action is determined according to:

• if no library definitions are present, we copy the definitions of libraries that are used in the project from the global library catalogue and copy the actual jars as well.
• alternatively we can copy only IDE supplied libraries by default and offer user to choose for libraries defined by him/her. That way we manage to handle the case of libraries defined by absolute path (on X:\ drive for example) as per company-wide policy.
• if a library definition file is already present, we only copy definitions that are missing.
• collocated JARs are kept at present location; otherwise copying is suggested

First step in wizard:

Last step in wizard:

Give user control on file references

Adding a file to project or library carry often a problem how to reference the file - via relative or absolute path? Or would it be better to just copy file somewhere else? There will be custom JFileChooser which let user decide. Such a chooser will be used whenever a file is being added to project or library regardless whether it is JAR file, Javadoc or sources ZIP file, etc.

Add library jar illustrative screenshot:

Based on where it is used it could always recommend a default action:

• for jars that have sharable path, the "Use relative path" option is the default
• for jars with non-sharable path, use the "Copy" option as default

Additional warning messages can be displayed in the dialog to make user aware of eventual sharability issues with current selection.

Also:

• warn about sharable/non-sharable path to library descriptor file. Allow absolute paths however as well. That caters to the usercase where company has fixed structure on network path (e.g. X:\ on Windows) and all projects are using fully qualified path references.
• warn about sharable/non-sharable locations of jars in relation to the library definition file. If the definition file is already considered non-sharable, we can loosen the warnings.

Open issues

• how to handle upgrades of IDE packed libraries?

mkleint: upgrades should be generally user's responsibility. if the consequence of non-upgrade mean broken IDE functionality, the affected module needs to check the project's classpath for old jars (eg. by comparing MD5 checksums of binaries known to be old.) and offer an upgrade. Upgrade can mean 1. change of current library definition content, or creating new shareable library definition with distinct new name, removing the old library from project classpath and adding the new library on classpath.

Possibly for future releases we might consider adding some sort of versioning to libraries, to easily recognize the version used and the version required.

• is new project by default sharable or not (in case we do not know for sure, e.g. not under VCS FS)

mkleint: What about remembering the last user's choice?

• copy javadoc and sources during library import or not? should there be different behaviour for IDE-provided and user defined libraries? ANSWER: always copy sources and javadoc files. It is easy for user to omit them from VCS check in or just delete them.

• any special support for inter-project references? we do not want probably to "copy" projects just to make them collocated. ANSWER: no, this is fully up to user

• if project is sharable then all paths (including absolute ones) should be stored in project properties and not in private properties, right? ANSWER: yes.

• Shared libraries management (invoked via "Manage" button in Libraries panel of Project Customizer) may confuse users. Defining new shared library may be understood as adding it to classpath. To avoid this perhaps "Manage" button should not be there. Its only purpose anyway is to allow removal of libraries (addition can be done via "Add Library").

How this address identified problems

Solution for UC1

Adding IDE-provided library to non-sharable project behaves like today.

IDE-provided library, that is library from libraries catalogue cannot be used directly by sharable project. Such a library has to be first imported into sharable libraries definition file. And that will make library sharable via VCS/ZIP. In this usecase IDE automatically imports JUnit or Swing Extension library.

Solution for UC2

The same as Solution for UC1 - the user defined library from libraries catalog need to be imported to project sharable one. User is free to define a library directly in the shareable libraries definition, or reuse the global definition repeatedly.

If library JARs are already stored in alongside of project sources then their copying during import from libraries catalogue may be skipped and only library definition is copied. For example in following structure

|-NetBeansProjects
  |
  |-Project1
  |
  |-libs
  |
  |-commons-lang
     |
     |-commons-lang-2.3.jar
     |
     |-commons-lang-2.3-sources.jar

there is project Project1 with shared libraries definition in libs folder. If user created library in libraries catalogue representing JARs from commons-lang folder and later decides to add this library to Project1 then only library definition will be copied and library JARs will be referenced relatively.

Solution for UC3

Adding raw jar to non-sharable project adds jar with fully qualified path.

Adding raw jar to sharable project is done via customized JFileChooser which gives user options how the file should be referenced from a project.

Solution for UC4

The problem with explicit J2SE platform is that it is not desirable (ignoring the fact it is also difficult) to store J2SE jars/binaries in VCS. From this point of view there is not much what can be done. IDE warns user and as a fallback uses default platform (the one IDE was started in).

Solution for UC5

J2EE server usecase has several solutions but it is not clear which is the best one.

1) One solution is to do nothing and leave it as it is now. This works well if all project members have the same (development time) application server installed. That's quite common scenario. Advantage is that user's do not need to care about what is on server classpath - they just use it and it is the same for all team members. Non-NetBeans users just set property pointing to their installation of the application server. Similarly continual build machine needs to have the application server installed and property set. Disadvantage of this approach is that project has external dependencies and in a sense project build is not reproducible. This may become more apparent problem over the time. To produce maintenance release three years down the track will still require the same version of application server which was used for development. Changing application server to different one may cause project compilation errors as described in UC5.

PH1: At least we should try to display a message reporting missing server in ant output.

PH2: Creation of the maintenance release is more likely about careful project management. Even if you will be able to create the build (intended for old server version) you'll need the required server version for testing. You just need the given server version in any case.

2) Different solution is to copy all server JARs to project. Such a project works well both for NetBeans and non-NetBeans users. It does not require any special tasks to be done for continual build machine and therefore it does not suffer from disadvantage of previous solution. Potential issue might be that server classpath contains lot of server internal JARs. NetBeans Server plugin API allows to filter these out, as for example done for Glassfish, but it is not guaranteed and for example in Tomcat case copying all server JARs may not be desirable. Copying JARs to project make application independent of application server, but using application specific JARs may prevent application from running on different server.

PH1: This is very convenient however we expose user to some legal issues (that can be acceptable if we will warn him somehow) - in fact he is publishing (via mail or VCS) proprietary code possibly without rights to redistribute. Maybe we could make extra checkbox for "Share server libraries" with some warning or possibility to select and place license file to the library location.

PH2: The original solution considered just j2ee platform libraries. However this seems to be idealized. Server integration provide much more things and classpaths to other tools (like web service). Also the user have to select the proper library - things are going to be bit complicated. Even in the case all these aspects will be covered - the "only" benefit will be buildable project (could be enough motivation to do the work). For IDE deployment, testing, configuration you will need the registered instance anyway.

3) Variation of solution #2) is to analyze server classpath in new project wizard and use IDE libraries instead. IDE would bundle different versions of Servlet API, JSP API, JSF API, J2EE API and these would be imported into sharable project instead of copying all server jars. User could use existing mechanisms to exclude library from being packaged in WAR file. mkleint: Ideally the IDE shall mark them as excluded automatically I think. If there is an API JAR which we cannot bundle (for example J2EE 1.4 because of legal reasons) the IDE could import (that is copy) the JAR from server and store it in project's shared libraries definition file location - the imported file would be also automatically excluded from WAR file knowing it is already in server. Or alternativelly user could be asked for location of this file via our customized JFileChooser and decide there how to add the file to project. This solution works only for couple of well known API JARs mentioned above (JSF, JSP, ...). For all other (server specific) JARs user would have to explicitly add them to project classpath as in scenarion #2). This solution is independent on application server so changing server it is not an issue.

PH1: The hard part seems to be "to analyze". I don't see any particular advantage compared to #2) solution.

Solution for UC6

Old NB6 project is by default in non-sharable state and therefore can be turned into sharable invoking 'Make Project Sharable' action in Project Properties and following previously described process for making project sharable.

Plan

ToDoTasksScratchPad

Attachments