---

How to store external libraries in the NetBeans CVS repository - updated

In the spirit of building on the shoulders of giants, NetBeans takes advantage of external libraries which are not developed on netbeans.org. Those libraries are either open-source software, or binary-only software but with liberal licenses. A few examples: Apache Tomcat; JUnit; JavaHelp (runtime); javac compiler; JSR-88 interface classes.

For convenience, these libraries are stored in the same CVS repository as the source code under CDDL. They are placed in well-known places in the source tree. The license text is associated with the binary file to make it clear which terms and conditions the users/developers must agree to besides being compliant with the CDDL itself. Only source code covered by the CDDL (or BSD, in the case of samples) can be hosted in the netbeans.org CVS repository. As the NetBeans CVS tree is growing, we need to initiate stricter rules and check that all external binary files have a correct associated license. There are also several recommendations on avoiding unnecessary additions of binary files into CVS.

The build system will automatically check if all binary files under <nbmodule>/external are stored correctly with appropriate license and all required information. <nbmodule> means NetBeans project module, e.g. external is on same level as nbproject. Failing to do so will result in a broken build!

Questions:

• I need to store some binaries in my own CVS. Should I follow same rules as well? No, you do not have to. You can store your binaries under release/modules/ext/, more details are described in harness/README
• My binary is not a library and I need to store it somewhere else. It has been also created under CDDL. Then you should update nbbuild/antsrc/org/netbeans/nbbuild/extlibs/ignored-binaries
• How can I check all is all right? Run ant -f nbbuild/build.xml verify-libs-and-licenses

Here are the rules NetBeans committers must follow when placing external libraries into NetBeans CVS:

• Legal due diligence must be observed before using a new external library, to make sure that the library license is suitable for use in NetBeans.
• All external binaries should be stored under a subdirectory named <nbmodule>/external, and nowhere else. (XXX I guess we still make an exception for contrib/*/external. -jglick. You are right, contrib/* is a very special case which needs a waiver from the process -petr)
• External binaries are put in CVS as is - don't "scramble" them.
• Each external binary should have a corresponding license file stored in the same directory as the binary itself.
• The name of the binary must follow the convention somelib-x.y.z.jar or somelib-x.y.z.zip where x.y.z is the version number. The corresponding license file must be named somelib-x.y.z-license.txt.
• All license files should be in UTF-8 encoding with appropriate line and paragraph breaks. The license file must end with a newline. Lines should not exceed 80 characters.
• The license file should follow a specific format. Details below.

License file format

License files should be in the following format:

Name: SomeLib
Version: 1.2.3
Description: Library for management of some blah blah blah.
License: Apache_V20 [see note regarding normalized names]
OSR: 1234 [OSR number, refer to LFI previously]
Origin: http://www.xyz.org [where file(s) were downloaded from]
Files: xyz.jar, xyz-doc.zip, xyz-src.zip [optional; see below for explanation]
Source: URL to source [mandatory for LGPL, otherwise optional]
Comment: needed until NB runs on JDK 6+ [optional: why is this library here?]

Use of SomeLib version 1.2.3 is governed by the terms of the license below:

[TEXT OF THE LICENSE]

If the Files header is not present, then a license name-x.y.z-license.txt must correspond to a binary name-x.y.z.jar or name-x.y.z.zip. If present, it should list the names of all binaries to which it corresponds.

The header fields are read during the build process and removed. Therefore this information will not appear in the final build or NBMs.

Template-based licenses

If there is template-based license (like BSD one http://www.opensource.org/licenses/bsd-license.php), e.g. the license file has several ad hoc places to be updated accordingly. The template itself should have the license file stored under nbbuild/licenses with well-defined tags __XXX__, these tags will be replaced during the build. Template-based licenses stored along with the binary in CVS must have be in original form as they came with binary:

Example BSD License, as it is stored in nbbuild/licenses:

Copyright (c) __YEAR__, __OWNER__

All rights reserved.

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

    * Redistributions of source code must retain the above copyright notice,
      this list of conditions and the following disclaimer.
    * Redistributions in binary form must reproduce the above copyright notice,
      this list of conditions and the following disclaimer in the documentation
      and/or other materials provided with the distribution.
    * Neither the name of __ORGANIZATION__ nor the names of its contributors
      may be used to endorse or promote products derived from this software
      without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Example BSD License, as it is stored in CVS along with binary:

Copyright (c) 2007, NetBeans

All rights reserved.

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

    * Redistributions of source code must retain the above copyright notice,
      this list of conditions and the following disclaimer.
    * Redistributions in binary form must reproduce the above copyright notice,
      this list of conditions and the following disclaimer in the documentation
      and/or other materials provided with the distribution.
    * Neither the name of NetBeans nor the names of its contributors
      may be used to endorse or promote products derived from this software
      without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

NBM build, managing correct license for NBMs

Required licenses should be listed in project.properties. (There still must be a license along with the binary in CVS.) The new entry will be called extra.license.files, where the license files will be relative to project basedir, e.g.

extra.license.files=external/x-1.0-license.txt,external/y-2.0-license.txt

This will create an NBM with two extra licenses besides the usual CDDL. This also maintains compatibility with the current build system.

As a convenient shortcut for the common case that you simply want to copy some files to the target cluster (but cannot use the release directory since third-party binaries are involved), you may use the newly introduced release.* Ant properties which should be specified in project.properties. Each key names a file in the source project; the value is a path in the target cluster. Any such pair will automatically:

• Copy the source file to the cluster. (No need to override the release Ant target.)
• Cause the target file to be included in the NBM file list. (No need to add to extra.module.files.)
• In the case of release.external/* properties, cause the associated binary to be included in the NBM license. (No need to override the nbm Ant target or add to extra.license.files.)

Example (from the form module; details):

release.external/beansbinding-0.6.1.jar=modules/ext/beansbinding-0.6.1.jar
release.external/beansbinding-0.6.1-doc.zip=docs/beansbinding-0.6.1-doc.zip

(Note: if you wish for the binary to be in the classpath of the module as a library, you will still need a <class-path-extension> in your project.xml.) You can also use a ZIP entry on the left side and it will be extracted from the ZIP to your cluster:

release.external/stuff-1.0.zip!/stuff.jar=modules/ext/stuff-1.0.jar

Normalized names

There will be a license repository under nbbuild/licenses where all licenses in use should be available. Each license type will be given a unique name: Apache_V11, Apache_V20, etc. This name must be referred to in the License field. This allows us to count licenses and file names and build a 3rd-party README as well as NBMs. Make sure that the license for a new binary is correctly included under nbbuild/licenses. If there is no existing license of the same type, it must be reviewed prior to committing.

NetBeans Samples

If a sample is created for NetBeans itself, it can be packaged into ZIP file and should not be in the external/ folder. To ensure tests correctly skip over it, the owner must add an entry for the binary into nbbuild/antsrc/org/netbeans/nbbuild/extlibs/ignored-binaries and include a brief explanatory comment.

Alternately, it may be preferable to keep the sample files unpacked directly in CVS, and create the ZIP during the module's build process (either directly into the cluster, or into build/classes for inclusion inside the module). This not only prevents tests from warning about it, but can make it easier to update minor parts of a sample and may make version control operations more pleasant.

The sample itself must be covered by the BSD license; the license must be included in every file (excepting binaries such as icons).

Copyright (c) <YEAR>, Sun Microsystems, Inc.

All rights reserved.

Redistribution and use in source and binary forms, with or
without modification, are permitted provided that the following
conditions are met:

* Redistributions of source code must retain the above
  copyright notice, this list of conditions and the following 
  disclaimer.
* Redistributions in binary form must reproduce the above
  copyright notice, this list of conditions and the following
  disclaimer in the documentation and/or other materials
  provided with the distribution.
* Neither the name of Sun Microsystems, Inc. nor the names of
  its contributors may be used to endorse or promote products
  derived from this software without specific prior written
  permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE. 

If sample is not created solely for NetBeans, e.g. bundled in a third-party product and covered by a separate license, it must follow the same rules as for any other binary library.

---

Common mistakes

A binary file has no associated license. (E.g. xyz.jar is missing xyz-license.txt.)

A binary file has an associated license, but does not maintain the naming convention, or has typos. (E.g. xyz.jar with xy-license.txt.)

Binary files from the original distribution are unzipped and included in the CVS tree with one associated license. There are several problems in the current repository that the binary was correctly stored as a ZIP file, but when the ZIP was uncompressed by the build, these binaries from the original appeared in the CVS tree. This is because there is not a correct .cvsignore file, so the CVS support thinks that these files should be added to CVS (and this can be easily overlooked). This can result in the same binaries being included several times in the CVS tree and this significantly increases overall checkout size!

Similar to the above, but these files are generated and added to CVS later (test data, etc.).

Licenses are not pure text. (E.g. they contain HTML.)

A binary file is duplicated in several places. Before adding a new library, please make sure that library is not already available in the CVS tree. If it is, check if the version there is suitable for you; if so, communicate with the owner regarding possible upgrades and/or available packages if they are not available. You might need to move the library to a parent cluster as well. If you do depend on such a third cluster, make sure your module is marked as eager, otherwise it will get disabled.

The names of the binary and its license file will change when the binary is upgraded to a newer version. Update project.properties (or, less commonly, build.xml) to reflect this change.

Before moving from my own repository to NetBeans CVS, I used release/modules/ext/ for storing my binary libraries. They need to be moved into external/ unless the library itself is covered by CDDL, build script, licenses etc., must be updated accordingly!

How do I know if some other modules is relying on the source location of my external binaries? Answer: it's not hard to find out. For example, if you want to know who uses web/external, try this (Unix / Bash syntax):

cd nb_all
for f in */{,*/,*/*/}{build.xml,nbproject/*.{properties,xml},test/{,*/}*.xml}; \
  do fgrep -H web/external $f; done

Implementation work

Continuous build

Interesting files from build:

1. Current license summary
2. VerifyLibsAndLicenses test
3. CreateLicenseSummary test
4. Unreferenced or overreferenced files

CVS information

In trunk.

Initial cleanup

• Create nbbuild/licenses.
• Put initial set of licenses into nbbuild/licenses, the licenses can be grabbed from java.net.
• Issues:
  1. http://www.netbeans.org/nonav/issues/show_bug.cgi?id=111437
  2. http://www.netbeans.org/nonav/issues/show_bug.cgi?id=111438
  3. http://www.netbeans.org/nonav/issues/show_bug.cgi?id=113713

Static verification of CVS

Run at least once a day; could be a separate Hudson project or part of regular build. Only pays attention to CVS-controlled files in the checkout, so can run on a built source tree without becoming confused. Writes results in JUnit format for easy browsing from Hudson.

• Look for *.jar not in */external/ dirs (with some exceptions).
• Every license file has at least mandatory headers.
• Every license file has lines at most 80 characters long, etc.
• For LGPL, must have Source header.
• Check that every external *.jar or *.zip has a matching license. (Or it can be mentioned in Files header of some license.)
• Every binary has a version number in name.
• No binary occurs more than once, under any name (so check by CRC-32 or SHA-1 etc.). (Look inside ZIP files for nested JARs.)
• Every license file's License field refers to something in nbbuild/licenses.
• The file in nbbuild/licenses exactly matches the body of the license file. Whitespace-only changes are permitted, e.g. rewrapping lines to make them fit. For licenses with templates (e.g. BSD License) any tokens between two underscores can match whatever character sequence.

All implemented.

Changes in IDE build

Generate a third-party JAR & license summary. Find every binary in the IDE build which is either present directly in some */external dir or present inside a ZIP in some */external dir. For every such binary, retrieve the license from nbbuild/licenses. Make a single document listing all of the binaries and licenses.

Verify that no such binary is present in more than one place.

Implemented. See THIRDPARTYLICENSE-generated.txt in the build.

Changes in NBM build

nbbuild/templates/projectized.xml (netbeans.org CVS modules only) will look up extra.license.files and use them in Info.xml.

release.* properties honored (see above).

Implemented.

Golden files

nbbuild/build/generated/external-libraries.txt should probably be generated directly from external dirs rather than scanning the IDE build for JARs as it does today.

Implemented. But does not yet take account extra.license.files correctly. Also may not be a complete list of libraries.