BuildSystemDesign

Contents

Build System Design

Version:
October 24, 2006
Author:
Jesse Glick, Jan Rojček
Abstract:
For NetBeans 4.0 a new build system was introduced. An Ant-based design was selected, with principal design goals including architectural simplicity, ease of expansion, robust headless builds, good control by project type of the UI, and an initial feature set adequate for everyday development. This document outlines the fundamental design principles of that system.

Overview from a User Perspective

First, an overview of how the build system works as far as the user is concerned, without regard for its implementation in NetBeans modules. This section describes the currently implemented UI and build behavior.

User development files (e.g Java sources) are grouped into logical units called projects. Generally, a project is a disk folder containing some sources of various kinds; some metadata describing its structure and contents for use by the IDE; an Ant build script; some files (in Java *.properties format) giving parameters needed by Ant as well as the IDE; and build products created by Ant.

Each project has a type which specifies its behavior. For example, a J2SE project is expected to contain some Java sources, perhaps some JUnit-based test sources, perhaps a JAR manifest, etc.; its behaviors include compiling the sources, creating a JAR file, running the program using a main(String[]) method, debugging the program, running tests, generating Javadoc, etc.

(Currently the project type is expected to anticipate and handle all aspects of the project’s behavior, but see [#natures below] for more.)

You can create new projects using wizards in the IDE’s UI and they should be more or less ready to go when they are created—depending on the project type, you may need to first add some source files. You can open or close existing projects to display them in the IDE’s Projects tab. (Whether a project is open or not does not affect much besides the display of this tab and some global actions in the UI that need a list of likely files to work on—you can freely edit files from projects that are not open in the UI, build them, etc.)

Every file on disk may belong to a project (or it may not). But a file may only belong to one project at a time. Normally all files beneath a project directory are considered to belong to it. (A project directory may be contained within another project directory—in this case files by default belong to the nearest containing project directory.) The project which owns a file is able to supply some behaviors for it—for example, a Java source file contained in the Java sources folder of a J2SE project will have editor code completion choices appropriate to that project.

The IDE’s UI supplies a set of actions (e.g. menu items) which perform operations on the project. For example, if you select any file contained in a project and choose Build from the UI, the project will be built. (Some actions are context-sensitive, so that for example with a Java source file selected you can choose Compile File to compile just that one source file.) Each operation is typically performed by just running Ant.

An Ant-Based Project in Detail

So what does a project look like on disk? To start with, for the purposes of this document, we will assume that your projects are kept under version control, e.g. CVS. You can also have a project that is not under version control if you like living dangerously. Unless otherwise stated, everything in the project is considered versionable. See [#vcs below] for more details.

At the top level of your project directory there may be various source folders, according to the project type. There may also be externally located source folders.

There is also a file build.xml, an Ant build script. This script is the entry point for building your project—whenever you invoke a build action from the IDE, a particular target in this script is run. You can edit this script if you want, modifying or even replacing anything in it. But the build.xml as created by the new project wizard is almost empty—it just delegates all actions to an implementation script which it imports (see below).

A subfolder called nbproject holds project metadata and the implementation of the Ant build. A file project.xml holds general project configuration. This includes the type of project, (perhaps) its name, a list of references to subprojects (see below), and some basic structural parameters of the project—e.g. which version of Ant it requires as a minimum, whether the project build specifies a particular Java platform to use instead of the default, etc. The XML file is extensible but may be syntax-checked using XML Schema. It is normally modified via the IDE’s GUI.

nbproject also contains an Ant build script called build-impl.xml. The meat of the build semantics for the project is defined here. You cannot directly edit this script; it is automatically regenerated whenever project.xml changes. (See below for more on how to customize the Ant build by hand despite this.) Although it is generated from project.xml, it should normally be kept under version control to ensure that users of command-line Ant, or other IDEs, can build the project by just checking it out and running it. (More below.)

Also in nbproject is project.properties. This file (in Java properties format) contains most build parameters, such as locations of libraries, javac option settings, etc. It is normally updated by the GUI when you modify project configuration, but you may also edit it by hand, e.g. using the IDE’s text editor.

nbproject/genfiles.properties is used to detect [#upgrades upgrade situations]. It should normally be kept in a VCS, but should not be edited by hand, and if deleted will not cause much harm.

nbproject has a subdirectory called private that contains any per-user customizations for this project. (Technically, this is per checkout of the project rather than per user as such.) This directory is not kept under version control. private.xml may contain transient information, such as files the developer has opened from the project recently. private.properties can set build parameters—any property defined here overrides any definition in project.properties.

Finally, nbproject/private can contain a cache subdirectory for internal use by the IDE. This directory, if it exists, is not versioned. Currently this facility is not used.

Besides project.properties and private.properties, a build script also loads settings from a file named build.properties located in the IDE user directory. Typically this file would give the locations of Java platforms, libraries, or other externally-installed resources. You can keep these links in one place and refer to them by name in particular projects. Typically entries in this file are regenerated from other IDE configuration data in the user directory.

As is normal in properties files used by Ant, properties can refer to other properties that have already been defined—including elsewhere in the same file—using the ${propname} syntax. A build script will typically load private.properties first, then the global build.properties, and finally project.properties. Earlier definitions always take precedence.

Customizing a Project

If you do not know how to write Ant scripts, or just do not wish to be bothered, you can manipulate all commonly required aspects of the project and run common build operations from the IDE’s GUI—you are not required to deal with Ant directly. You can open a project customizer dialog and change the classpath, JAR compression options, common Javadoc flags, etc. Changes are saved to disk and so take effect the next time you run an Ant target.

If you do know how to use Ant, you can customize your project to your heart’s content, all in build.xml. You can add some new targets. You can perform additional steps before or after any predefined target. You can even make a target not delegate to build-impl.xml, and instead do something quite different: for example, if you have very specific requirements for how to generate Javadoc, you can edit build.xml to run the <javadoc> task exactly the way you like it, rather than in the way the IDE set it up. You can use any tasks bundled with Ant, or even tasks you create yourself or get from some external source.

If you make such customizations, the IDE will not “know about” them in any real sense—that section of the project customizer dialog will not be of any use to you; and design-time features like code completion may not be completely accurate. Generally, the IDE only pays attention to project.xml and the various properties files when it is deciding how to let you work with a project in the GUI.

Running Targets

Update for NetBeans 6.0: see design document linked from ProjectConfigurationProvider Javadoc

When you select an action in the IDE’s GUI (such as Build) this just runs a target of a predefined name (e.g. build) in your build.xml. From there, the regular Ant integration in NetBeans takes over. The target is run using the globally configured Ant installation—by default, the copy of Ant 1.6.2 bundled with the IDE, but you can choose another installation (at least version 1.6.0 is required for most project types). The IDE loads and runs Ant in the IDE’s JVM.

Ant prints various messages about its progress, which are sent to the Output Window. If any messages are printed starting with an absolute file name followed by a colon, they are automatically hyperlinked; line numbers, column numbers, line/column ranges, and trailing messages are supported here. (Java stack traces are also hyperlinked if the corresponding source can be found.) Modules are permitted to customize hyperlinking and other things using AntLogger. You can thus navigate typical build errors using F12 without leaving the source editor window. If you need to halt the build prematurely, you may do so using Build | Stop Building. Otherwise the build runs to completion, or failure. You can run multiple builds at once. You can also adjust the verbosity level of Ant to suppress noncritical information, or on the other hand to include debugging information such as which commands are being run with what arguments at each stage of the build.

The NetBeans IDE bundles some special Ant tasks which can only be run in the same VM as the IDE, as they do something to the IDE itself. A build-impl.xml may use these tasks in some targets—where appropriate, targets not using them are also available in case you are running the script outside the IDE. <nbbrowse> opens the IDE’s configured web browser on an HTML file and may be used to e.g. display Javadoc after it is generated. <nbjpdastart> prepares the IDE’s JPDA debugger to work with a debugged program. Since these are run like any other Ant task, you can use them in build.xml if you wish.

Targets in build-impl.xml normally reflect appropriate dependencies: for example, a project is automatically built before it is run. Ant tasks normally do internal dependency analysis, so e.g. Java source files will not be recompiled gratuitously. A project will typically have a Clean action that deletes all build products, and a Rebuild action that first cleans and then does a regular build—plain Build is an incremental build.

Inter-project Dependencies

You can create multiple projects, and have multiple projects open in the GUI at once. Currently you are free to open as many projects as you like at once regardless of whether they are related to one another. Remember that little depends on which projects are open; this is primarily a list of what projects you would like to look at in the GUI.

A project may have some required projects, or informally subprojects. (Two projects may both depend on the same subproject, but you cannot have cycles.) Typically building a project first builds its subprojects (recursively), and cleaning a project also cleans its subprojects—you can disable this behavior in the GUI, however. Currently the GUI prompts you to open subprojects when you open the parent project by default.

You do not need to do anything special to set dependency relations between projects. Just use a build product of one project in another project, using the GUI. For example, if in a Java project you configure the classpath to include a JAR built by another project, you will have set up a dependency between the two projects. If you remove that JAR from the classpath, you remove the dependency. There is no special GUI for configuring build order—subprojects are simply built before projects that depend on them.

The parent project keeps a reference to the location of the subproject on disk. Using a VCS system, this will normally be a relative file path. See [#vcs below] for more information.

If you edit build.xml by hand, you can of course arrange to build other projects (or any Ant scripts) as part of your build, using the <ant> or <subant> tasks. Note that a build-impl.xml, when building a foreign project, calls its build.xml (rather than skipping to its build-impl.xml), so you can freely mix a hand-customized project with IDE-customized projects.

It is permitted for one project A to depend on projects B and C, both of which in turn depend on a project D—i.e. for the dependency graph to be a DAG (directed acyclic graph). This may happen if a generic library is used by several components of an application. Currently the Ant build scripts will run D’s build script twice if you ask to build A. This is probably not a serious problem, as the incremental nature of builds means that the second run will just complete quickly and quietly once Ant sees that everything in D is already up to date, but it does add some overhead to the build (especially in a deeply tangled DAG). This may be resolved in a future release.

J2SE Project Type

The J2SE project type supports general Java applications, typically client-side. It can contain one tree of Java sources in the src/ folder, some JUnit tests in the test/ folder, and produces a single JAR file from compiled classes and resources. Important targets include:

Build
Compile Java sources and create the JAR. Clean and Rebuild are as you would expect.
Run
Run the program using a main(String[]) class.
Javadoc
Create Javadoc for classes in the project.
Test
Run JUnit tests. You can also run a single test case by itself.
Debug
Debug the program, i.e. run it using a main(String[]) class but attach the debugger to it. You can also debug a single test case if it is failing but you do not know why.

Since vanilla J2SE development does not imply a particular way of packaging or deploying an application, a J2SE project has no special distributable beyond the JAR file. (Of course you can edit build.xml to do something more.) A project type for a more structured platform such as Java WebStart or J2EE could know exactly how to package the complete application, including references to libraries produced by subprojects.

Headless Builds

To build a project from the command line, or from another IDE, just run its Ant script.

The project type may use Ant optional tasks which require specific libraries to be in Ant’s classpath. For example, the plain J2SE project type supports JUnit-based testing. Ant ships with the <junit> task, but requires that junit.jar be present in Ant’s own classpath. (Ant does not currently permit you to select the location of junit.jar from the build script.) Therefore in order to run J2SE projects from the command line, you need to include junit.jar in $ANT_HOME/lib/, or your $CLASSPATH, or in a directory specified by -lib, or in a path appended in ~/.antrc, &c. The copy of junit.jar included in the IDE distribution will work, but the user could use a different copy instead.

If you build a project from outside the IDE that needs some resources from build.properties in the user directory, by default the build will fail since only Ant running inside the IDE knows where to find this file. You can either pass the location of build.properties to Ant using the user.properties.file property, or point to a separately maintained properties file (useful for SCM—see below). Or you can simply define whatever properties you need in any other way supported by Ant.

In some cases the IDE may ship with libraries or other resources that are needed by some project type, in which build.properties will automatically include references to the library for use inside the IDE. When running a build from outside the IDE, you can choose to refer to the IDE’s copy of the library; or you may prefer to refer to your own copy stored elsewhere. (Library paths should be individually overridable.) Strict software configuration management (SCM) principles dictate that everything needed to build a project should be kept in a version control system, ready for use by an unattended headless build server with a tightly controlled environment. In such a case, you would copy libraries shipped with the IDE into your VCS (or use your own preferred copies) and adjust properties to refer to your preferred versions. The GUI does not currently assist you in making this conversion to a strict SCM style, though a future release might.

If a project’s build-impl.xml requires tasks that are shipped with the IDE and not present in a standard Ant distribution—this is not the case for a plain J2SE project, but other project types (e.g. for J2EE) need to do this—the tasks will need to be defined in the build script according to an overridable classpath, very much like any library definition. This should be compatible with an SCM development style.

Code Architecture

The primary code involved in the build system is divided into three portions: a generic project system; an Ant-based implementation of that project system; and some Java-specific facilities.

This section will just describe the concepts beyond the code architecture, not every class and method. Reasonably complete Javadoc is available if you want to know the details.

Generic Project System

The projects/queries module defines some generic queries. The term “query” is used to refer to a design pattern by which two modules can communicate about a very concrete issue without needing to have a hard dependency—or indeed without needing to know anything about the model or design which decides the question. This design pattern is very important for the build system since it permits gratuitous inter-module dependencies to be avoided, and would permit very different architectures to be substituted for various subsystems.

For example, the editor module should not have to comprehend a complete model of how Java projects manage their sources, just in order to provide appropriate code completion for a Java source. All it really needs to know is the intended compile-time classpath for a given source file—all other details are no interest in this context. Therefore, it calls a query which maps a file to a classpath, and uses the result. The query is implemented in this case by a Java-based project type which knows what the classpath should be for a given source root. If the architecture of the project type is radically changed, there will be no effect on the editor—in fact a module could provide code completion information for certain sources not in any project, if that were useful. The communication between the querier and the provider is managed in a simple way by Lookup.

The queries in this module pertain to information about files and do not imply the presence of any projects, though they are of especial use in conjunction with projects. Other modules can and do define other query interfaces, in addition to implementing them; generally a query interface should live in the most general API-providing module that it can, so as to minimize gratuitous module dependencies.

The projects/projectapi module defines the generic infrastructure for projects. This infrastructure has no dependencies on Ant or Java; it could in principle be used for other build systems (e.g. Maven), or other languages (e.g. C++).

The Project interface represents one project, as loaded into memory from disk. ProjectManager can be used to load and save projects. “Loading” a project does not imply that it is in any way displayed in the GUI (though projects displayed in the GUI will always be loaded); a project is loaded quietly when someone first requests it. A project is always kept in memory while it is modified; if it is not modified, it may be released by the garbage collector. A ProjectFactory is responsible for recognizing projects of a certain type on disk, loading them, and saving them when they are modified. (For Ant-based projects, you do not need to create your own ProjectFactory; see below.)

A project has an associated Lookup giving optional capabilities. SubprojectProvider is a predefined capability of projects which might have some “subprojects” (the meaning is intentionally left vague, but see [#deps above] for the typical interpretation). ActionProvider is a capability of projects which can run some user actions. ExtensibleMetadataProvider permits arbitrary modules to store some information in a project without explicit support from the project type—this may serve as a useful alternative to .nbattrs files for many purposes.

FileOwnerQuery lets you find the project that owns a file. The default implementation just searches for the nearest containing project directory.

The projects/projectui module defines how projects may be displayed in the GUI, and provides the standard generic UI elements. LogicalViewProvider lets a project define a logical view for use in the Projects tab, instead of just showing the raw folder as it appears in the Files tab. This view might, for example, display Java sources arranged in a flat package list, or EJBs—the possibilities are endless. (But some voluntary UI consistency between project types is necessary to avoid a hodgepodge appearance.) CustomizerProvider lets you provide a rich GUI for customizing the project’s configuration; there are some support SPIs for this.

Ant-Based Project System

The ant/project module contains mostly support classes which make it straightforward to define an Ant-based project with the conventional semantics described above. An AntBasedProjectType implementation is registered to define one Ant-based project type, for example plain J2SE projects. The project implementation receives an AntProjectHelper instance which supplies a great deal of standardized behavior and which would normally be used to implement Project methods and various optional capabilities.

Design note: generally a project type should live in its own module and nothing else should depend on it—it should not expose any public packages etc. All of its functionality that is not directly encapsulated in the project type itself should be exposed only indirectly through query implementations. This design constraint ensures that it is possible to produce additional project types with some of the same functionality, without hacking other modules.

AntProjectHelper principally deals with reading and writing project.xml (and private.xml), and project.properties (and private.properties). It tracks the modification status of the project automatically and handles persistence to disk of these files. It also helps regenerate build-impl.xml from project.xml with an XSLT stylesheet; implement GUI actions by running Ant targets; interpret build properties as Ant itself would interpret them, to make design-time features in the IDE match the build behavior as closely as possible; and more.

You may also create a ReferenceHelper object which manages subproject references according to the conventional semantics. Associated APIs such as AntArtifact let an Ant-based project export a list of build products it expects other projects might wish to use, or conversely let an Ant-based project find appropriate build products from potential subprojects.

Java Facilities

The java/api module defines some general APIs particular to the Java language (or VM). Most significantly, this includes the Classpath API which lets you both find and declare the classpath that will be used to compile a Java source; the classpath that will be used to run it; etc.

java/project defines some SPIs helpful for implementors of Java-oriented projects and some miscellaneous APIs.

The java/platform module defines an API for finding or registering Java platforms—i.e. installations of the JRE or SDK. It supports multiple editions such as J2SE, J2ME, and J2EE, as well as profiles such as are used heavily by J2ME. java/j2seplatform provides an implementation of the API for J2SE, and includes a wizard to autodetect J2SDK installations. Portions of the platform information relevant to Ant are made available in build.properties in the user directory.

The projects/libraries module lets you register and search libraries, such as JARs that might be shared by multiple projects. This will be used by project types to let you configure the location of everything of interest about a library (e.g. binary, source, and Javadoc) in one place in the GUI, and then quickly add a defined library to a project. (Modules are also be able to declare information about libraries they bundle.) Portions of the library information relevant to Ant are made available in build.properties in the user directory.

java/j2seproject is the implementation of the J2SE project type. Most of the code is currently concerned with providing an appropriate GUI, especially for the project customizer. It also implements a number of queries, and of course supplies the Ant-based project type itself.

VCS Implications

While the build system does not directly implement any VCS-specific UI, it is designed to be “VCS friendly”, i.e. to store metadata in such a way as to make typical VCS operations work smoothly.

Inter-Project Dependencies and VCS

One aspect of VCS friendliness pertains to subproject references. Often the subproject will reside in the same VCS repository as the parent project. If the IDE determines that both projects are collocated in this way, it will use a relative file path to the subproject in the parent project’s configuration, so that the set of projects can be checked out from VCS as a unit and be ready to run. Such relative paths are kept in the shared project.properties. If the projects are not collocated in a recognized way, an absolute file path will be used, in private.properties.

The same consideration applies when deciding whether to use a relative or absolute path for a plain file reference from a project, e.g. a JAR (not included in a “library”) added to a Java project’s classpath.

The generic query CollocationQuery permits a VCS filesystem implementation to tell other modules in the IDE whether files are collocated. There is one implementation of this query in VCS module.

Clear Separation of Versionable and Nonversionable Files

As noted [#project-layout above], projects should clearly separate subdirectories which should and should not be kept in VCS. Typically everything is versionable except for build output directories and nbproject/private/. The generic query SharabilityQuery permits a VCS to find out whether a given file or directory ought to be considered versionable, without knowing any details of the project system. VCS filesystem implementations in NetBeans can check this query and use it to decide whether or not to add a file or directory to the repository by default. Note that SharabilityQuery supports VCS implementations, such as ClearCase, which need to know whether a directory will be shared before it is created on disk.

Stability of Metadata

A critical aspect of VCS friendliness is that all metadata files which are written to by some GUI element in the IDE should be stored in a way that will minimize the size of standard textual diffs, in order to reduce the chance of merge conflicts, make code reviews more pleasant, etc. For typical Ant-based projects this boils down to project.xml and project.properties.

project.xml is formatted as a namespaced XML document. It is always written to disk using conventional formatting (i.e. line breaks between elements and four-space indentation). It consists of a short general section defined by AntProjectHelper; a section controlled by the project type provider, otherwise freeform; and zero or more sections supplied by other modules using the ExtensibleMetadataProvider implementation provided by AntProjectHelper, including a subproject references section defined by ReferenceHelper. The stable sort order is guaranteed for elements in the general section; the root elements of the project type provider’s section and each of the extended sections; and the subelements defined by any code in the ant/project and java/j2seproject modules. Other project types should also take care to ensure that their subelements are stably sorted and otherwise produce minimal diffs when modified. (Also it is expected that changes to project.xml are not requested unless some actual change to the XML DOM tree was made, though this is not currently enforced.) Since project.xml is not expected to be edited as text by typical users, preservation of formatting (comments, whitespace, etc.) is not a high priority.

Similarly, project.properties is formatted as a Java properties file. The EditableProperties class is designed for reading/writting of properties. It is similar to java.util.Properties, but it is designed to retain additional information needed for safe hand-editing. Added functionality provided by this class compared to java.util.Properties:

  • minimizes textual diffs of changes, by preserving comments and formatting in lines not affected by a change
  • order of entries preserved during modifications whenever possible
  • can associate comments with particular entries
  • inserts new properties in lexicographic order
  • can split long composite property values (e.g. classpaths) into multiple lines that can be more easily diffed and merged

Unlike project.xml, project.properties is intended to be editable as text by semiadvanced users, so preservation of formatting is important.

If necessary to reduce the likelihood of merge conflicts in project.xml, a couple of lines of whitespace or comments could be introduced between sections. Typical merge tools have a “horizon” of around three lines of text that they consider to indicate conflicting changes, so this trick would prevent conflicts in case two different sections were both modified. A similar trick could be used for project.properties, though the syntactic simplicity of properties files makes merge conflicts less onerous (and the relatively large number of independent properties makes them less likely).

Build Scripts

A project’s build-impl.xml file is kept under version control, yet is regenerated from project.xml. In principle a merge conflict could arise if two developers both modified a shared project.xml and build-impl.xml was correspondingly modified for each. In such a case, the developer responsible for resolving the merge conflict could simply resolve any conflicts in project.xml, delete build-impl.xml, and open the project in the IDE to force the script to be regenerated.

build.xml is technically similar to build-impl.xml, but is intended to be hand-editable, in which case there are no special requirements for versioning—users who wish to avoid extraneous diffs can do so themselves. If the user never makes any edits except through the GUI, then it is automatically and quietly regenerated just like build-impl.xml, in which case the same merge conflict situation described above could be resolved in the same way.

Versioned Filesystem Mounts

There is no special code in the build system for handling VCS working directories. MasterFS (openide/masterfs) is used which makes all files on disk available to the Filesystems API and guarantees a one-to-one mapping between files and file objects.

VCS-Related GUI

The Versioning Manager independently handles configuration of VCS information. Currently the build system provides no special GUI elements for interacting with VCS modules. If a file in a project is in a VCS, the regular VCS context menu should be available. Project type providers are also free to supply VCS-specific annotations or actions in their logical views. Beyond that, no special support is provided. However tighter integration in the UI between projects and the VCS is planned.

VCS Commands in Build Scripts

The user is free to write top-level “starter” build scripts that check out all sources for a project and its subprojects from a VCS before building them, using standard Ant tasks. No special GUI support for this is planned.

Validation of Project Metadata

Currently there are no special provisions for ensuring that project metadata on disk are valid before loading the project (e.g. in case a user made mistakes when resolving a VCS merge conflict), or for ensuring that it is valid before saving it (as insurance against buggy modules). W3C XML schemas do exist for all the namespaces used in the infrastructure and standard project types, which can be used to manually validate XML-formatted metadata (Alt-Shift-F9), but they are not currently used at runtime. Developers of new project types are urged to develop schemas to describe the syntax of their XML fragments.

(W3C XML Schema is much superior to DTDs for this purpose, not only because of support for XML namespaces, but also because of support for associating multiple schemas with a single document and validating fragments piecemeal. It is well supported by Xerces.)

Regarding properties files, it is important to note that when you check out a project from a VCS, private.properties will not yet exist. In this case, if any properties are missing the opening of the project will warn the user and ask him or her to resolve these problems using a provided GUI. This UI can involve setting the paths to foreign projects or foreign files, creating missing platforms or libraries, etc. private.xml does not pose any issues if it is missing—Ant does not look for it, and AntProjectHelper automatically creates it if something needs to be added to it.

Freeform Projects

Users with existing Ant build systems of substantial complexity, or those who simply need advanced control over their build, are not likely to find the IDE-generated build-impl.xmls adequate; and they may have special requirements for projects that are not satisfied by stock project types, such as multiple compilation units in a single project. The build system provides a technique for such users to gain significant benefits of IDE integration while maintaining full control over their build scripts.

It is helpful to think of project types in the following order, with later entries providing more power but requiring more setup:

  1. Project types supplied by the IDE with no editing of the build script. The user might not know how to use Ant (or want to know); or they might be an experienced Ant user but simply have no special requirements for this project. They can create a new project out of the box (or import a simple source tree, depending on the particular project type), customize a couple of things in GUI dialogs, and begin development. The project type should support commonly needed configuration parameters, but no more—the GUI should be kept simple and easy to learn, and the purpose of the project (e.g. “build a web application, deploy and test it”) should be predetermined and focussed.
  2. The same as #1, but with customizations made to the build script. The user has some knowledge of Ant and wants to either make some modifications to the build process not supported by the GUI (e.g. excluding some files from the created JAR); and/or wants to add extra targets for functionality not covered by the GUI (e.g. running Checkstyle).
  3. The freeform project type. This user is comfortable with Ant and does not need to rely on the IDE to help him or her write a build script. The user may have special structural requirements such as multiple source roots with interdependencies, in addition to the kinds of customizations possible in #2. There are many ways to configure the project type, though within limits—e.g. all source roots must be explicitly declared.It is important to note that labels like “J2SE application” and “J2EE web application” apply to levels #1 and #2. A particular freeform project could be used as a J2SE application or a J2EE web application or a VoiceXML extension library or anything else; the IDE does not know or care, though it may care about certain aspects of e.g. J2EE technology, so e.g. JSP code completion can work correctly.
  4. A custom-built project type. This is a power user able to write NetBeans module extensions who wants to provide tight integration for a specialized class of project, which might be used only in-house (for a large suite of projects), or might be more generally applicable. At this level, almost anything is possible. While the author of the project type module needs to be rather sophisticated, the user of a project managed by it need not be.

The solution to #3 is a special freeform project type which defines a project.xml syntax but which does not create a build-impl.xml and assumes that the user is wholly responsible for editing some build.xml (or even multiple scripts); in fact the user should be able to use an existing build script unmodified if it already provides all the functionality he or she needs. The project supplies a GUI customizer which is just an editor for the major functionality exposed by project.xml; the user can also manually edit project.xml and various properties files as text. The user creates a project.xml giving sufficient information about the structure of the project to permit the freeform project type to implement important “queries” and delegate to Ant when running actions, but no more.

The user can use a GUI wizard to create project.xml and there is also a project properties dialog permitting further GUI customizations. Not all features of the freeform project type are exposed in the GUI, only the most commonly needed functionality; additional customizations can be made by editing project.xml manually as text. (In which case syntactic or semantic validation errors are automatically displayed in the Output Window when the file is saved.)

What does the freeform project.xml contain in practice?

  • A display name.
  • A sequence of fixed property definitions (e.g. config.dir=etc) and property file load declarations (e.g. ${config.dir}/build.properties) that are used by the build script(s) and might be relevant to the IDE for configuring other entries in project.xml. The intention is to mirror the most commonly used syntax variants of Ant’s <property> task so that the user can make various configuration changes in property files and have them affect both the Ant build and the IDE configuration at the same time. The IDE tracks changes to these property files and updates its display and behavior accordingly.
  • A mapping from IDE internal action names (e.g. compile.single) to Ant targets to implement them (e.g. ${basedir}/build.xml#compile-one-file). The user can set the target names and optionally the build script to use in each case. If a user does not configure an Ant target for a given action, it is simply disabled for that project.In the case of context-sensitive actions (like Compile File), the user can also configure an Ant property name to use to pass the selection (e.g. javac.includes) as well as a predefined format (e.g. “comma-separated list of package-root-relative slash-separated file names”). The IDE would then run the Ant target with the correct definition (e.g. -Djavac.includes=org/foo/Foo.java,org/foo/Bar.java).A list of Ant targets that should get menu items in the context menu of the project’s logical view, for commonly needed actions.Some actions like Debug intrinsically require NetBeans-specific Ant tasks to be used in the build script, as they act on the live IDE; clearly such targets would not be useful except inside the NetBeans IDE. Other than that they behave no differently from other targets.In the case of IDE-oriented actions (including both context-sensitive and IDE-only actions), the IDE can in some cases automatically generate a plausible initial implementation of the Ant target (and associated mapping) for you automatically, if you try to run the action without a binding. This is currently done for some actions; the rest are expected to be handled in a future release.
  • A list of top-level source directories, defaulting to just be the project directory. Each should have a display name. This satisfies the SourceGroup contract for Sources.TYPE_GENERIC and also means that Find Files, Scan To-Do Items, etc. would work correctly even if there are external source roots.Also a list of file locations with display names to show in the Projects tab: each could be a single file, a plain folder, or a Java package root (shown as a package list). In any event, the Files tab always shows all top-level source directories as plain folders.
  • A list of Java compilation units (one or perhaps multiple package roots apiece). Each package root should give a display name for use in the IDE (e.g. New File wizard) and a location inside or outside of the project (perhaps expressed using Ant properties). This satisfies the SourceGroup contract for Sources.TYPE_JAVA.Each compilation unit should have a compile-time classpath (usually expressed using Ant properties), and maybe also a bootstrap classpath and runtime classpath if necessary. This satisfies the ClassPathProvider contract. It should have a Java source level (e.g. 1.4); this satisfies the SourceLevelQueryImplementation contract. Together these things permit code completion and refactoring operations to work accurately, as well as many other less obvious Java IDE features.A compilation unit may also have an associated list of binary targets (e.g. build/classes/ or dist/app.jar) which correspond to it. This satisifies the SourceForBinaryQueryImplementation contract, and improves code completion and refactoring as well as making Ant stack trace hyperlinking more reliable.
  • Other information about specialized source folders relevant to J2EE web application development, e.g. location of any document roots, or tag libraries used by JSPs in a docroot.
  • Other miscellaneous information which is useful to the IDE but not strictly required for everyday usage may be added. These things would be optional for the user to specify as well. For example:
    • A marker that a given source root contains unit tests, so that Open Unit Test can work.
    • A list of build products (e.g. JARs) that can be used by stock IDE project types such as in the J2SE project type’s classpath customizer (Add Project… button).
    • A list of subprojects to open with the main project.

Just setting up a few Ant targets and a Java compilation unit or two is suffice to have a project that you could build, run, and use refactoring and code completion on just as well as in an IDE-supplied project type. And the result is of course versionable so your colleagues can check out the project including project.xml and begin work right away. If you configure enough stuff in project.xml, you can simulate nearly all the functionality of the J2SE project type and perhaps other project types except for the customizer dialogs.

Some experimental work has also been done which indicates that it may be feasible to automatically scan an existing Ant build script and detect such information automatically; perhaps even updating this information every time the build script is changed. Effectively it would be a freeform project but would write or update much of your project.xml automatically. These possibilities provide rich opportunities for future releases to improve the workflow of developers working on large and complex applications.

NetBeans Module Projects

There is also a special project type that supports NetBeans module development. This project type is tightly tuned to the special requirements of NetBeans module development.

Release-to-Release Upgrades

What happens to a user project when some part of the IDE is upgraded? Some infrastructure is currently in place to handle such scenarios, though it may need to be expanded. Policies are still in development.

  • If a new version of a project type provider includes an updated XSLT stylesheet for build-impl.xml, user projects are currently quietly upgraded when the project is opened. This may be wrong.
  • If a new version of a project type provider makes use of a different set of standard Ant property names, a user project’s project.properties can be quietly upgraded to include the new property names and remove the obsoleted ones, when the project is opened.
  • Changes in the supported syntax of project.xml are possible between releases; if the user makes configuration changes in the properties dialog which would require the new format, he or she is warned about the format change and the project is written in the new format.
  • If the set of supported build targets for a project type changes with a new release, there is not much to do other than warn the user if they have any build.xml customizations.

Though these problems are somewhat similar to questions about what to do with old settings in the user directory, typically project configuration is much more important to a user than unversioned, global per-user IDE settings, and project metadata should be treated as precious data.

GeneratedFilesHelper permits you to determine whether either of build.xml and build-impl.xml have been modified manually by the user; whether they were generated from an older project.xml; and whether they were generated from an older (or just different) XSLT stylesheet. The J2SE project type regenerates both build.xml and build-impl.xml when it is safe to do so—i.e. when there have been no user modifications to the file. This is done both when saving changes to project.xml, and when the project is opened in the GUI with out-of-date scripts. You are warned if build-impl.xml needs to be regenerated but has been edited.

Project type providers are currently left to their own devices as regards upgrading domain-specific configuration data in project.xml. The support SPIs permit various kinds of upgrade policies to be implemented.

The “Natures” Question

Currently the project system expects that the project type is atomic, i.e. a single project has a predefined set of behaviors. A future release might expand this system to permit additional natures or sets of behaviors to be plugged into the project, either when it is created or perhaps after the fact. For example, a plain J2SE project could then be extended to also support obfuscation of the JAR. Currently, this would be possible with full GUI support only if obfuscation were directly supported by the IDE module supplying the standard J2SE project type, though a user could still manually configure the project to do anything Ant can do.

This feature is not currently planned due to its potential API complexity. Instead, a reasonable set of capabilities are hardcoded into the supplied project types, and users needing to do other things can edit build.xml to support that. As the J2SE project type demonstrates, supporting a variety of build steps in a project type need not be difficult, especially if Ant tasks to support the feature already exist.

If a third-party module really needs to extend a project with a new capability, as a workaround it could insert added targets into build.xml itself, perhaps using a wizard. Or it could simply define non-Ant-based actions in the context menu of object types it defines—for example, the XML modules do this for XML validation and XSLT processing, without project system participation.

For a future release the Ant-based project type infrastructure could provide an option for the project type to declare its extensibility by natures, for which an SPI would be created. A nature might need to be able to influence the initial build.xml; the generated build-impl.xml; the initially generated properties files; the customizer dialog; the logical view; the project’s Lookup; etc. Hopefully such an API extension can be done compatibly or at least with only minor incompatibilities. However no concrete proposal has been put on the table to date.

The “Portfolios” Question

Update for NetBeans 6.0: see ProjectGroups

Since a number of people have asked why the build system does not define the concept of a “portfolio”, it is worth looking at the question in detail. The main answer is that there are many possible interpretations of the question, and under some interpretations, the requirement is already satisfied. Let us enumerate some possible meanings of the term first, with some more precise synonyms:

master project
A project which has build-time dependencies on some subprojects, perhaps recursively. In order to build the master project, you must necessarily build the subprojects. Often the build products of the subprojects are not just used during the master project’s build, but are incorporated into the final distributable.There are numerous examples of this pattern. For example, a J2EE EAR can be considered a master project with subprojects which are WARs, EJB-JARs, or provisionable client JARs. A Java WebStart application can contain various libraries which are referenced from the JNLP file. A J2ME MIDlet can inline libraries into its JAR as it is being packaged and verified. For plain J2SE projects, the build-time dependency is clear, though there is no standard way of packaging up the subprojects.Master projects are directly supported by the build system’s infrastructure and implemented for J2SE projects. They are currently loosely supported in the GUI: opening a master project by default opens its subprojects too.The project system GUI currently lets you configure the master project open at any given time. (The GUI label is Main Project.) Some project actions, such as Run, apply to it (when invoked from the global menu or toolbar).
aggregate project
A pseudo-project which just aggregates some number of subprojects together in a group. They need not have any particular relationship to one another, with respect to the build process or otherwise; but it may be desirable to set an explicit relative build order of the subprojects in the aggregate project. The aggregate project might package up each of the subprojects into one big distributable (somehow). It might define one or perhaps multiple “entry points” (subprojects) for running, deployment, etc. It might have “batch” targets to build documentation for each subproject in turn, run all unit tests, etc.Currently aggregate projects are not directly supported by the build system, but it would be reasonably straightforward to create an aggregate project type using the existing APIs—perhaps a generic aggregate project type applicable to many application domains, or perhaps a more specific type. Aggregate projects would work much the same as master projects as far as the UI is concerned; the differences mainly lie in build semantics.Of course any user comfortable with Ant can create a wrapper build script with this kind of behavior; the integration in the IDE’s UI will not be tight, however.
open project list
A simple list of projects that should be opened (or closed) as a unit. They may have no further relationship to one another; the user simply wishes to see them all at the same time.The current build system UI does not support this concept, though it could easily be extended to do so. You need to open and close the desired projects individually. However you are permitted to have any projects open at once that you like.
VCS project group
A group of projects all collocated in a VCS—perhaps as sibling directories, perhaps as one top directory with subdirectories. No particular build-time semantics or interdependencies are implied by this term, so it can be used to qualify other terms like “master project” or “open project list”. The UI would ideally make it easy to check out the group as a whole and open it all automatically—probably by making an open project list as a text file shared in the VCS and using relative file paths.The build system does not directly support this concept, though it would be straightforward to add to the UI infrastructure.
scope for refactoring (…)
The set of projects open in the GUI is the scope for large operations such as Java refactoring. I.e. only sources contained in the scope are considered for updates when renaming a method, etc.A note: since the scope for refactoring is limited in this way, the refactoring operation may break the build of some project which was accidentally omitted from the list. Currently the IDE makes no attempt to warn the user about this condition, though it conceivably could (e.g. using a reverse dependency index).Currently the build system provides no special support for this concept beyond the list of projects open in the GUI.Some other IDE functionality need a similar scope: for example, the Java Fast Open feature needs to locate a source file by fully-qualified name. There might be multiple such sources in projects on disk, especially if the user has been keeping checkouts of multiple VCS branches of an application. Therefore the action searches only projects open in the GUI for a matching source.(Beware of overgeneralizing the “scope” concept to actions to which there is already a more clearly-defined scope. For example, the apparently similar Go To Class (or method, etc.) context menu item and keyboard shortcut in the Java source editor should actually behave differently. In this case, the scope in which the IDE needs to find the named class is clearly defined by the logic of the situation: it should be the sourcepath of the selected Java source file. I.e. the IDE should always jump to the source against which this file would actually be compiled, regardless of what is open in the GUI. The distinction may sound subtle, but it becomes very important when you are rapidly switching back and forth between several VCS branches: if A.java compiles against B.java, and you happen to have a copy of A.java checked out from the release44 branch open in the editor and the caret is over a method defined in B.java and you press Alt-G, you certainly wish to go to the release44 version of B.java. You could become quite confused if you were sent to a checkout of B.java from a different branch, which might not even contain that method!)

Clearly there are many potential interpretations of the term “portfolio”, often partially overlapping. Currently a more sophisticated portfolio system is not planned simply because it will require a lot of time to do the usability studies, surveys, and analyses necessary to judge which of these concepts are valuable, intuitive, and expected.

Various sorts of portfolios can be added for a future release if there is a broad consensus on what functionality should be subsumed. In the meantime, the master project concept is likely to be adequate for the common application types: plain J2SE application; J2ME MIDlet; J2EE WAR; J2EE EAR.

Most existing Ant-based applications with nested components use either the master or aggregate project patterns, so this style should come as no surprise to a developer accustomed to working on large projects with a scripted build system. Typically subprojects are physically contained within the master project’s directory, which effectively makes them VCS project groups as well.

(An interesting side note is provided by the open-source Maven project comprehension system, for which there is also a project type provider. While Maven-based projects typically follow the master project pattern, the subprojects need not form a VCS project group as such: instead you can download a particular binary release of a foreign project as part of your build. Systems such as Apache Gump make it easy to ensure that open-source prerequisites are stable and available.)

Sometimes a development team will write a wrapper script (see [#vcs-wrapper above]) that checks out a configured list of VCS modules, sometimes from different branches; this can be considered a combination of a VCS project group and an open project list.

Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo