Package Includes & Excludes

Feature for NetBeans 6.0 permitting include and exclude lists to be configured for project source roots (and perhaps binary classpath entries too).

Infrastructure

Tracker issue (now closed): #49026 For new requests or problems, please file separate issues and list them as blocking #49026.

CVS information (now obsolete): branch excludes_49026 (using base tag excludes_49026_base_6) exists in ant/freeform, ant/project, and java; all else from trunk. Final diff of merge, and commit log, attached to #49026.

Goals, Use Cases, Priorities

• (P1) User downloads open-source JDK. If the Java sources are in a single tree (mainly j2se/src/share/classes/) it should be possible to open just one subsystem (say, Swing) as a project. Only Swing-related packages should appear in Projects. Scanning should if possible be quicker than for the whole JDK.
• (P2) User downloads JDK sources and opens several subsystems as independent projects. The IDE understands which project each class belongs to.
• (P3) User downloads JDK sources and opens several subprojects. The IDE additionally offers code completion appropriate to the component. For example, code completion in the Swing project should not offer RMI classes. (See Retouche Integration below for more.)
• (non-goal) User downloads just the Swing portion of the JDK and tries to open it. This is unlikely to work even disregarding IDE issues, since the build script will need to compile against other JRE classes, and compiling against an arbitrary older JRE may not work. So we can assume that the complete source tree in fact exists on disk.
• (P1) User is employed by a large company working on a very large project, all kept in one tree. User is interested only in a few packages. User wishes to look at, compile, and otherwise deal with only those packages. Excluded classes should not be scanned.
• (P1) User has a project with some sources which are currently uncompilable (and unused). User wishes to temporarily exclude them from the build.

Status

Includes and excludes are used in the new NB projectizations of JDK subsystems! Get complete JDK 7 sources (not just src.zip) starting with b11 and open some of the projects beneath j2se/make/netbeans to try it out.

What works so far

• ClassPath and SourceGroup can both express includes and excludes, with support for Ant patternset syntax in SourcesHelper and a new PathMatcher utility
• PackageView displays included and excluded packages and files appropriately, both in package list and tree mode
• The New File wizard for Java-type templates shows only included packages.
• j2seproject can be configured with includes and excludes in the new from existing sources wizard or at any time in the Sources panel
• j2seproject build script honors includes and excludes
• Freeform projects can also be configured with includes and excludes at the time of creation or later.
• If some classes (or packages) in a Java source root are excluded, code completion does not see excluded classes, Go to Type does not see excluded classes, and references to excluded classes (from included classes) are marked as errors in the editor.
• If some classes are excluded but the binary output directory is on the compile classpath (see below), code completion displays the excluded classes and references to them are not marked as errors, but Go to Type still omits them. Rolled back for performance reasons on 20070307. May not be needed for JDK anyway.

What is notably missing so far

• any ability to specify includes and excludes of binary classpath items as opposed to source paths (under consideration as a separate feature for NBM and freeform and Maven projects - should enhance ProjectClassPathSupport.createPropertyBasedClassPathImplementation)
• some Java language features might not behave sensibly when excludes are in effect (TBD; needs testing and UI review)
• New File wizard for non-Java files (e.g. property files) honors includes if they are specified for the generic project source group, which usually they are not
• VCS actions probably ignore excludes. Ought to e.g. update only included files.
• TBD whether Find and Find in Projects do or should honor excludes.

API changes

• ClassPath.PROP_INCLUDES, ClassPath.Entry.includes overloads
• FilteringPathResourceImplementation
• PathMatcher
• SourcesHelper.addPrincipalSourceRoot and .addTypedSourceRoot overloads with includes and excludes parameters
• IncludeExcludeVisualizer
• new properties defined for j2seproject: includes and excludes
• freeform.spi.ProjectNature.IncludeExcludeAware
• freeform.spi.support.Util.get/putPrimaryConfigurationData and Util.NAMESPACE
• freeform-project-general-2.xsd (java/freeform automatically picks up containership info for ClassPath.SOURCE from a matching <source-folder> of type java)

Quick summary of API (see source diffs for details):

// ClassPath
public static final String PROP_INCLUDES;

// ClassPath.Entry
public boolean includes(String resource);
public boolean includes(URL file);
public boolean includes(FileObject file);

public interface org.netbeans.spi.java.classpath.FilteringPathResourceImplementation extends PathResourceImplementation {
  String PROP_INCLUDES;
  boolean includes(URL root, String resource);
}

public final class org.netbeans.spi.project.support.ant.PathMatcher {
  public PathMatcher(String includes, String excludes, File base);
  public boolean matches(String path, boolean useKnownIncludes);
  public Set<File> findIncludedRoots() throws IllegalArgumentException;
}

// SourcesHelper
public void addPrincipalSourceRoot(String location, String includes, String excludes, String displayName, Icon icon, Icon openedIcon) throws IllegalStateException;
public void addTypedSourceRoot(String location, String includes, String excludes, String type, String displayName, Icon icon, Icon openedIcon) throws IllegalStateException;

public class org.netbeans.spi.java.project.support.ui.IncludeExcludeVisualizer {
  public IncludeExcludeVisualizer() {}
  public void setRoots(File[] roots) throws IllegalArgumentException;
  public String getIncludePattern();
  public void setIncludePattern(String pattern);
  public String getExcludePattern();
  public void setExcludePattern(String pattern);
  public void addChangeListener(ChangeListener l);
  public void removeChangeListener(ChangeListener l);
  public JComponent getVisualizerPanel();
}

// J2SEProjectProperties
public static final String INCLUDES = "includes";
public static final String EXCLUDES = "excludes";

// freeform.spi.ProjectNature
// added params:
Node createSourceFolderView(Project project, FileObject folder, String includes, String excludes, String style, String name, String displayName) throws IllegalArgumentException;

// freeform.spi.support.Util
public static final String NAMESPACE = "http://www.netbeans.org/ns/freeform-project/2";
public static Element getPrimaryConfigurationData(final AntProjectHelper helper);
public static void putPrimaryConfigurationData(final AntProjectHelper helper, final Element data);

// freeform-project-general-2.xsd
<!-- ... -->
<xsd:complexType name="maybe-typed-source-root">
  <xsd:sequence>
    <xsd:element name="label" type="xsd:token"/>
    <xsd:element name="type" type="xsd:NMTOKEN" minOccurs="0"/>
    <xsd:element name="location" type="substitutable-text"/>
    <xsd:element name="includes" type="substitutable-text" minOccurs="0"/>
    <xsd:element name="excludes" type="substitutable-text" minOccurs="0"/>
  </xsd:sequence>
</xsd:complexType>
<!-- ... -->
<xsd:complexType name="view-items">
  <!-- ... -->
  <xsd:element name="source-folder">
    <xsd:complexType>
      <xsd:sequence>
        <xsd:element name="label" type="xsd:token" minOccurs="0"/>
        <xsd:element name="location" type="substitutable-text"/>
        <xsd:element name="includes" type="substitutable-text" minOccurs="0"/>
        <xsd:element name="excludes" type="substitutable-text" minOccurs="0"/>
      </xsd:sequence>
      <xsd:attribute name="style" type="source-folder-style" use="required"/>
    </xsd:complexType>
  </xsd:element>
  <!-- ... -->
</xsd:complexType>
<!-- ... -->

Review: issue #95983

UI

Screenshot of j2seproject showing provisional UI:

Proposed official UI: Excludes49026_UI

Open UI questions

• Exactly which other aspects of IDE behavior should be affected by includes & excludes?
• Should the include/exclude visualizer remain the way it is, with two text fields to type in patterns plus a preview, which is quick and powerful but possibly intimidating? Should it instead offer a package list (or tree?) with checkboxes, which is intuitive but clumsy on large source trees? (This is much less expressive so there needs to be some fallback behavior in case the actual current incl/excl patterns cannot be expressed this way.) Should both modes be offered, e.g. in a tabbed pane?
• Where should the include/exclude chooser be offered in a New Project wizard? Currently the j2seproject wizard offers it in a separate panel; the freeform wizard offers it as a button in the usual Sources panel, just like it will be shown in the Properties dialog (this is a bit easier to implement). One style should be chosen and used consistently.

"Retouche" (Java infrastructure) integration

Only included files are scanned when a project is opened. Go to Type and friends only offer included top-level types.

Multiple projects per root

A source tree may be split among multiple projects so long as all of them share the same classpath settings and source level. Only files included in the union of open projects should be scanned. For caveats and a demo, see: #97168

There are no current plans to support several projects using the same source root but with different classpath settings. (The IDE will not currently detect and prevent this anomaly.)

Other features

The exact set of IDE features which should, or should not, honor includes and excludes is an open question. Probably we need to consider each one when it comes up, with a realistic scenario in mind. For example, if you rename an included class which an excluded class refers to, should the IDE offer to update the excluded class? It is not obvious what the right thing to do in such a case is.

Output dir on classpath

There is special treatment for the case that the output dir is on the compile classpath. (I.e. ClassPath.COMPILE on a source tree reports an entry for which SourceForBinaryQuery reports the same source root.) In this case everything in the source tree will be scanned, and the main effect of the excludes on Java infrastructure (besides the Source Packages view) is to affect Go to Type. This mode might be used for displaying a view over JDK sources. (Which all need to compile together since they are not really modularized.) There will be no GUI mode support for this mode, however; it must be set up by hand in project metadata. Also switching in and out of this mode requires an IDE restart.

This is a special case of the more general case where part of the source tree has a classpath which refers to part of the binary output tree, i.e. the source tree is logically modularized. The IDE does not support the more general case and there are no current plans to do so, but if there are, the same API could be used (just more usages of it would be honored).

May not be used for JDK sources; might use binary bootcp instead.

FAQ

Q: What does an exclude really mean? Is it as if the file does not exist at all?

A: Not exactly. Excluded files are still displayed in the Files tab. (Currently they are also displayed in the Projects tab if you use the tree display mode rather than package list display mode, but this should probably be fixed.) They can still be opened, etc. But wherever possible they are treated as if they do not belong to the project. This is a similar situation to files owned by physically nested projects.

Q: If you specify includes and/or excludes, will the project no longer be compilable if some of the included classes depend on an excluded one?

A: It will not be compilable; the Ant script enforces it. The user is expected to not exclude classes which are in fact required. Violations are shown as errors in the editor. An additional enhancement would be to change IncludeExcludeVisualizer to run a zero-output test compilation as the user adjusts include and exclude patterns, displaying any illegal references.

Q: So you can change includes/excludes on the fly. That means changing it in the project properties dialog? Is there any API for listening to the changes?

A: Yes (see screenshot); and yes (see SourceGroup.PROP_CONTAINERSHIP and ClassPath.PROP_INCLUDES).

Q: What is the exact way of finding out what files belong to a project and what project a file belongs to? What is the SourceGroup.contains method for?

A: Use FileOwnerQuery as always to determine the owner of a file. Note that even excluded files beneath a project dir or otherwise included files will be treated as owned by a project unless there is some more specific owner registered. This is similar to ownership of files by nested projects.

Given a project, you can find the Java sources it contains using Sources and SourceGroup as usual, and yes SourceGroup.contains should be used to verify that a particular file (or package) is included. You may also get a ClassPath (of type Source) and use various ClassPath methods documented to honor includes and excludes.

Q: Did you pay enough attention to performance? The project system is already a performance bottleneck. The features described look like they could slow it down even further.

A: Impacts remain to be seen and will of course depend on implementation choices made in Java scanning code etc. However note that ClassPath methods checking for inclusion which take a String (or URL) rather than a FileObject do not necessarily require FileObject creation; in fact the j2seproject implementation of these methods will do no more than a match against a (precompiled) regular expression. Interactive testing of IncludeExcludeVisualizer shows it to be pretty fast even on large source trees even while rapidly changing the include and exclude lists.

Q: What happens when the includes/excludes are defined in such a way that one file is included in more than one project?

A: Then just one project will "claim" it as far as the IDE is concerned. Such a situation could already occur in NetBeans 4.0, using external source roots. There as yet no mechanism to detect and prevent such situations generally. See #51810 and #53369.

Q: What's up with "floating includes" or known include roots?

A: Say you have Ant sources and you include just org/apache/tools/ant/ (i.e. skip GZip utility classes etc.). Both ClassPath and SourceGroup require that if a file or folder is included then its parent folder is too. Therefore, the folders org/apache/tools, org/apache, and org, as well as the root folder, must be treated as included. This is configurable in PathMatcher; SourceGroup and FilteringPathResourceImplementation implementations should set useKnownIncludes.

If the source directory is externally located, just org/apache/tools/ant will be registered as owned by the project.

In this example, PathMatcher can know quickly what the included roots are. If you set includes to **/tools/ant/, it will still find out. But it will need to do a quick scan over the folder to look for actual matches. This will add a little bit of overhead. The overhead only applies to users of wildcards in the include list.

Competition

What do other Java IDEs do to support excludes?

IDEA:

GoTo Class:      Excluded class not visible
CodeCompletion:  Excluded class not offered
CC in excl file: In excluded files does not work at all.
Compile ex. file:Can't be compiled
Errors:          Like if the file would not exists at all.
Compilation:     Project not compilable
Explorer:        Excluded packages not visible
Other:           Different icon in editor tab for excluded files
                 Nice customizer for including/excluding packages

Other project on the same source:
1 module project: possible to create but notice that IDEA does not permit
opening more than one such project. (It starts the second frame which I
suspect being other copy of IDEA)
N modules project: not possible to have two modules sharing the same source root.

ECLIPSE:


GoTo Class:      Excluded class not visible
CodeCompletion:     Excluded class not offered
CC in excl file: Works Included class offered
Compile ex. file:?
Errors:          Like if the file would not exists at all.
Explorer:        Files visible under separate node (files)
Compilation:     ?

Other project on the same source: Not possible to create 

Demo Script

Tested on: NB 6.0 M9 (20070419), JDK 6.0u1, Ubuntu.

1. Make a new Java application project using main class app.Main.
2. Make a new class stable.Model.
3. Make a new class unstable.Model.
4. Add to Main.main: System.out.println(new stable.Model()); System.out.println(new unstable.Model());
5. Run.
6. Go to unstable.Model and introduce a typo: public Model() { x } and save the class. Error badges appear on Model.java, unstable, Source Packages, and the project node.
7. Try to run again; you get a compilation error.
8. Right-click the project and choose Properties.
9. Under Sources, click Includes/Excludes...
10. In Excludes, type: unstable/ and watch as unstable/Model.java is pushed to Excluded Files. Click OK twice.
11. The unstable package node disappears.
12. Go back to Main.java. You will see an error: package unstable does not exist
13. Try to run. You will get the same message - the package does not exist.
14. Comment out the usage of unstable.Model.
15. Run again and the program succeeds. Note: some error badges will remain at this point. This is a bug. Workaround is to disable error badges in: Tools > Options > Java Code > Tasklist > Enable Error Badges on Files
16. Build > Generate Javadoc. Note that only app and stable packages are documented. Note: classes appear twice in the class index. This is a bug.
17. Go back to the includes/excludes config dialog. Enter app/ stable/ for Includes and make Excludes blank. You will see the same set of files resulting. Accept this change. You can also mix includes and excludes however you like.
18. Reset everything by setting Includes to **. unstable should reappear.
19. Move unstable.Model into the stable package under the name UnstableModel using rename and move refactorings.
20. Configure Excludes as follows: **/Unstable* Accept changes. UnstableModel should disappear again.
21. Look in the Files tab. You can always see the excluded classes there.

The appearance of the includes/excludes customizer dialog is likely to change for 6.0: Excludes49026_UI

Attachments

49026-j2seproject-screenshot.png		160602 bytes
project.xml		4562 bytes