ExtractingStandaloneCluster

Contents

Introduction

Some module clusters in the NetBeans main repository are no longer actively developed or supported by the NetBeans team. (See NetBeansArchivedFeatures for an overview.)

The main repo is intended only for stable, actively developed modules. To make it feasible for volunteers to take over maintenance of the "downsized" modules, the modules need to be moved into their own source repositories, converted to be buildable in a standalone fashion against NetBeans platform & core IDE binaries (modules in main and contrib use a special legacy build convention), and otherwise cleaned up.

This page lists the steps I performed while running this process for the NetBeans Ruby support.

Moving Source Code

Identifying modules to be moved

First it is necessary to decide exactly which modules (~ top-level directories) need to be moved. Open nbbuild/cluster.properties and look for the cluster definition; these are all the modules normally built in the cluster.

glassfish.jruby
jellytools.ruby
libs.jrubyparser
libs.yydebug
o.jruby
o.jruby.distro
o.kxml2
o.rubyforge.debugcommons
ruby
ruby.codecoverage
ruby.debugger
ruby.extrahints
ruby.help
ruby.hints
ruby.javaint
ruby.kit
ruby.platform
ruby.project
ruby.railsprojects
ruby.rakeproject
ruby.refactoring
ruby.rhtml
ruby.samples.depot
ruby.testrunner
spellchecker.bindings.ruby

Also look around for other modules in the source tree which appear to be related, though they may not currently be built in that cluster, perhaps because they are poorly maintained. They should still be moved; the new developers can decide to fix them up and include them in builds, delete them, leave them in sources but unbuilt, etc. For example, searching by name:

main$ ls -1d *ruby*
glassfish.jruby
jellytools.ruby
libs.jrubyparser
o.jruby
o.jruby.distro
o.rubyforge.debugcommons
ruby
ruby.codecoverage
ruby.debugger
ruby.extrahints
ruby.help
ruby.hints
ruby.javaint
ruby.kit
ruby.merbproject
ruby.platform
ruby.project
ruby.railsprojects
ruby.rakeproject
ruby.refactoring
ruby.rhtml
ruby.rspec
ruby.samples.depot
ruby.testrunner
ruby.themes
spellchecker.bindings.ruby

turns up three new entries not in the cluster list:

ruby.merbproject
ruby.rspec
ruby.themes

http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastSuccessfulBuild/artifact/nbbuild/build/generated/deps.txt lists all module-to-module dependencies among modules in registered clusters (including the contrib repo), which is useful for verifying that other modules will not be broken by removing all Ruby-related modules. http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastSuccessfulBuild/artifact/nbbuild/build/generated/cluster-deps.txt is also useful for verifying that all the Ruby-related modules can be built against the basic IDE (platform, harness, and ide clusters) and that other clusters do not refer to the ruby cluster.

In this case you might have noticed the libs.bytelist and libs.jvyaml modules. But these are used by the languages.yaml module, which despite its Ruby associations is in the ide cluster (a dep of editor.kit): YAML text file support is for now at least a standard part of the base IDE distribution, since it is occasionally used in non-Ruby-based app frameworks.

Can also check for historical modules since deleted:

main$ (cd .hg/store/data; ls -1d *ruby*)

which turns up just one additional item, _ruby_application1, which was bogus and need not be converted.

Should also check for related modules in contrib:

contrib$ ls -1d *ruby*
ls: cannot access *ruby*: No such file or directory

but there appear to be none. If there were, these would likely need to be converted as well; removed from contrib; and the conversion result forcibly merged into the conversion result from main (so as to provide both sets of modules). As with modules in main not in the official cluster, these additional modules might or not not be used, but it would be best to make them available just in case.

Can also check for related files in misc. In this case you may need to use the rename directive to convert, since file layout here is arbitrary.

Extracting module history into a separate repo

HgHowTos#Transfer_a_module.27s_history_to_another_repository is the basic technique; we will see how to apply that by example here, where the details will differ a bit.

First, make sure your version of the main repo (core-main in my case) is as up-to-date as possible:

$ hg fetch
$ hg fetch http://hg.netbeans.org/main-silver/
$ hg out
$ hg push

Now for the actual conversion. Currently Mercurial releases are incapable of doing this in a reasonable amount of time for a repo as big as main, due to http://mercurial.selenic.com/bts/issue991, so we need to use a patched version.

crew$ hg up 1.7.5
...
crew$ hg qpu --move convert-faster-991.diff
...

In my setup, .../testhg (in $PATH) runs Mercurial from its source tree:

#!/bin/sh
dir=.../crew
make -C $dir all 1>/dev/null 2>&1
export PYTHONPATH=$dir/build/lib.linux-i686-2.6
exec $dir/build/scripts-2.6/hg --traceback "$@"

so I can use this command where necessary:

$ testhg version
Mercurial Distributed SCM (version 1.7.5+1-05783ac9edbf)

For simplicity, we will assume that the converted modules should live in the same source structure as they do in main, so we can use the include directive but no rename directive in the convert command.

In order to capture historical branch information - i.e. changes made to IDE release stabilization branches to Ruby modules, usually backports of fixes from "trunk" - I will actually switch to another local repo which contains everything in http://hg.netbeans.org/core-main/ and http://hg.netbeans.org/releases/ too. This is optional but might be useful. Say a Ruby developer receives a bug report with a stack trace from NetBeans 7.0 beta 1 - they might like to check out the tip of the release70_beta branch to make sure line numbers in their sources exactly match the stack trace.

releases$ hg pull https://hg.netbeans.org/releases/
releases$ hg pull ../main
releases$ hg up default

(Running conversion from a separate local repo also makes it possible for me to do unrelated changes in my normal repo while the conversion repo is running.)

So let's start:

releases$ (for m in glassfish.jruby jellytools.ruby libs.jrubyparser \
               libs.yydebug o.jruby o.jruby.distro o.kxml2 \
               o.rubyforge.debugcommons ruby ruby.codecoverage ruby.debugger \
               ruby.extrahints ruby.help ruby.hints ruby.javaint ruby.kit \
               ruby.merbproject ruby.platform ruby.project ruby.railsprojects \
               ruby.rakeproject ruby.refactoring ruby.rhtml ruby.rspec \
               ruby.samples.depot ruby.testrunner ruby.themes \
               spellchecker.bindings.ruby; do echo include $m; done) | \
          time testhg convert --filemap /dev/stdin . ../ruby

This will do a few preparatory calculations, then start counting down changesets in historical order as it searches for relevant file modifications and converts them. Note that scanning gets slower per changeset as you get into newer history. You can watch progress from another shell window:

$ruby hg log -l1 -Mv

Time for a leisurely lunch!

8568.77user 740.19system 2:41:44elapsed 95%CPU (0avgtext+0avgdata 2158592maxresident)k

Using --config convert.hg.saverev=1 can also be helpful for matching converted changesets back to the originals, though currently Hg exposes no GUI for accessing this hidden field.

Removing modules from main

main$ hg rm glassfish.jruby jellytools.ruby libs.jrubyparser libs.yydebug \
o.jruby o.jruby.distro o.kxml2 o.rubyforge.debugcommons ruby ruby.codecoverage \
ruby.debugger ruby.extrahints ruby.help ruby.hints ruby.javaint ruby.kit \
ruby.merbproject ruby.platform ruby.project ruby.railsprojects ruby.rakeproject \
ruby.refactoring ruby.rhtml ruby.rspec ruby.samples.depot ruby.testrunner \
ruby.themes spellchecker.bindings.ruby
main$ hg ci

Removed also these entries from nbbuild/cluster.properties:

nb.cluster.ruby.dir
nb.cluster.ruby.depends
nb.cluster.ruby
validation.nb.cluster.ruby
clusters.config.ruby.list

Also removing nb.cluster.ruby from clusters.config.bloated.list, and removing ruby.themes from experimental cluster. Can also find some other mentions of the Ruby support that need to be cleaned up this way:

main$ hg locate -r . -0 nbbuild\*\* \\.\* | xargs -0 grep -l -- ruby

23c261cc6b0b is the resulting commit; 99934ed62e8e is the merge also reflecting the removal of the module sources.

(The .hgignore entries need to be copied to the new repo.)

Check for any remaining deps from main modules:

main$ ant init
Cannot find build prerequisite org.netbeans.modules.ruby.platform of .../contrib/portalpack.commons

Turns out PortalPack has some dependencies on the Ruby support. Details can be seen:

contrib$ hg locate -r . -0 portalpack\*\* | xargs -0 grep -l -- \[Rr\]uby

The dependency could very likely be made soft. However, the PortalPack modules are not currently built even in the experimental cluster config, so we will punt on this for now.

Publishing separate repo

Pushed to https://bitbucket.org/jglick/nbruby temporarily. Once http://hg.netbeans.org/community-ruby/ created, pushed there.

Preparing Standalone Build

Converting to Ant-based module suite

Using IDE, created a new module suite in a temp dir with the name 'ruby', against a platform named 'nb69'. Closed project. From command line, created nbproject dir in new repo, and moved three files into it from template project:

nbproject/project.xml
nbproject/project.properties
nbproject/platform.properties

Opened the root of the new repo as a project, which is now a suite; this generates some more files:

build.xml
nbproject/build-impl.xml
genfiles.properties

plus some nonversionable stuff in nbproject/private. Right-click project, Rename, type in 'NB Ruby' (do not move!); this just sets app.title in project.properties for display purposes. (Want to keep <name>ruby</name> in project.xml for use as cluster name.) Also in platform.properties, unselect all clusters from platform except

platform
harness
ide
nb

Can soon Add Existing... to add the modules beneath it, but this is not available for converting former netbeans.org modules until they are prepared a bit better; in the current state they are not even loadable as projects. First need to insert <standalone/> in each project.xml so they are treated as self-contained module projects:

ruby$ perl -pi -e 's!            <module-dependencies!            \
      <standalone/>\n            <module-dependencies!' */nbproject/project.xml

Now Add Existing... works. Check that we did all of them:

ruby$ fgrep standalone */nbproject/project.xml

Need to update our ignore file so we can see what we really need to add. We append to the imported entries from main to exclude build and nbproject/private subdirs of every project (suite & modules):

^o.jruby.distro/unpatched_source$
^o.jruby/patched_source$
(^|/)build$
(^|/)nbproject/private$

Now we can track all the new files:

ruby$ hg add

There is still a problem that every module's build.xml tries to refer to ../nbbuild/templates/projectized.xml, which is for netbeans.org modules only. We could have solved this simply by deleting build.xml before "Add Existing", causing them to regenerated with the correct import, but a lot of these build scripts have customizations we need to keep. So instead:

ruby$ perl -pi -e 's!\.\./nbbuild/templates/projectized\.xml!\
                     nbproject/build-impl.xml!' */build.xml

There's a lot more cleanup to do, but let's commit what we have so far: c8c08c1453fc

Handling external binaries

We will also handle */external/binaries-list files as ExternalBinaries describes: e54ce82201a2

That just made sure the binaries are downloaded, which is required for in-IDE code completion and so on. The original projects also used release.* properties to copy JARs to $cluster/modules/ext as DevFaqExternalLibraries#NBM_build.2C_managing_correct_license_for_NBMs describes; this not available for external projects in the 6.9 harness, so we need do some more work: e69851c445db

Fixing up suite build

Time to try our first suite build. We have to make some corrections for different Ant target names in overrides, and make sure that external binaries are downloaded at the start of the build: 57a65875b2c5

Trying again, we run into a real problem:

ruby.build-init:
Cannot compile against a module: .../nb69/ide/modules/org-netbeans-modules-csl-api.jar
because of dependency: org.netbeans.modules.csl.api/2 > 2.11

Seems that there was some sort of incompatible change in the CSL API in 7.0, whereas for now we are building against 6.9. Inspection of the changeset bd130b29325c as it applied to the ruby cluster shows that no code in the Ruby support actually changed, meaning the change was probably bogus (original author should have merely pushed a fixed version of editor lib without touching downstream modules!) and we are safe just reverting the spec version in the dep: d08f4934b8b6

The next problem:

ruby.build-init:
Cannot compile against a module: .../nb69/ide/modules/org-netbeans-modules-editor.jar
because of dependency: org.netbeans.modules.editor/3 > 1.53

Again inspection of the changeset 10f8e107bf98 shows an increment in module deps for no clear reason; we will just back it out: deace5ee3cf6

Now we have:

ruby.build-init:
Cannot compile against a module: .../nb69/ide/modules/org-netbeans-modules-editor-lib.jar
because of dependency: org.netbeans.modules.editor.lib/3 > 3.1

Inspection of b6f862b9772e shows that the author made no code changes in the ruby cluster yet requested a new major version. This is simply wrong; he should have used a release version range instead. Again we will fix this: 79e7ae432654

Next is an error in RubyCodeCompleter; da32d187895f consumed an incompatible API change without proper versioning or a compatibility layer. Looks like we can work around this: a42810a50c00

Now it seems ruby.merbproject is uncompilable - not surprising given that it was not being built continuously. We will just remove the three "experimental" modules from the suite for now: 87b8450b7476

Next:

ruby.testrunner.build-init:
Cannot compile against a module: .../nb69/ide/modules/org-netbeans-modules-gsf-testrunner.jar
because of dependency: org.netbeans.modules.gsf.testrunner > 1.16

This is more serious because main #f3f2a4970c8d was clearly incompatible (changed signature of an interface). We can make it compilable against 6.9 and time will tell if the result links against 7.0: 30de9b8f9784

Finally the modules all compile!

Building NBMs fails due to

spellchecker.bindings.ruby.nbm:
.../nb69/harness/suite.xml:222: The following error occurred while executing this line:
.../nb69/harness/common.xml:426: Exception reading blurb from .../ruby/standard-nbm-license.txt

which can be fixed by removing a bogus definition of license.file: 7a8ad2a1e64f

Fixing tests

Unit tests are still uncompilable since CslTestBase cannot be found (Ant-based NB modules cannot publicly export test utilities); we need to copy that into ruby.platform test sources from 6.9 sources, along with various other test utility dependencies: d12ce02008b8

Now all error badges are gone, but some tests can still not be built, much less run:

ruby.platform.test-unit-dep-build:
.../nb69/harness/common.xml:524: The following error occurred while executing this line:
.../nb69/harness/common.xml:519: Cannot open .../nb69/platform/lib/nbproject/project.xml

which seems to be a reporting bug in the harness; the problem is <test/> on platform modules remaining in ruby.platform/nbproject/project.xml, which needs to be removed.

Inspection of project.properties also shows that it is still using old-style test deps:

test.unit.run.cp.extra=${core.startup.dir}/core/core.jar:${o.n.bootstrap.dir}/lib/boot.jar

These ought to have been migrated after FitnessTestsWithoutX but apparently no one got to them, so we will fix them now. Also it uses

test-unit-sys-prop.xtest.jruby.home=${netbeans.dest.dir}/ruby/jruby-1.5.1

(there are a lot of other instances of this basic definition) which won't work for an external suite since there is no Ruby cluster in the target platform. We can find offenders this way:

ruby$ egrep '\.dir|\$\{netbeans\.dest\.dir\}/ruby' */nbproject/project.properties

f846f288ccf0 has these fixes.

Locating platform binaries

To make it possible for all developers to build against a controlled copy of NetBeans, we will download it right away in the suite build. We put this in the .hg subdir so that hg clean (including from Hudson) will not remove it: a292d486d861

http://source.apidesign.org/hg/netbinox/file/1075e21a049d/platform.xml demonstrates another possible approach which is more flexible in that it lets you pick up just the clusters you need. The download is potentially slower in this case since each module has to be retrieved separately.

Alternate: converting to Maven build

Not (yet) done in this case so cannot give exact steps. The basics would be:

  1. Remove all nbproject dirs.
  2. Create Maven POMs. One pom-packaging project at root with <modules>. One nbm-packaging project per module. And one nbm-application-packaging project to represent the "suite", with dependencies on the modules.
  3. Remove all build.xml. Replace any custom build functionality not handled by Maven by default with the appropriate Maven plugins.
  4. Move sources and test sources into the expected locations.
  5. Move functional (GUI) tests into the nbm-application-packaging project.
  6. Find original artifacts for external binaries and declare dependencies.

Setting up a Hudson job

Job config history can be used to see diffs from the old job (2010-10-20_17-59-53) to the current state.

Other Possible Steps

Issue tracking and mailing lists could be moved if necessary. (Requires site admin permissions.) Not done in this case.

Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo