ExtractingStandaloneCluster

Revision as of 17:48, 19 February 2011 by Jglick (Talk | contribs)

Contents

Introduction

Some module clusters in the NetBeans main repository are no longer actively developed or supported by the NetBeans team; the main repo is intended only for stable, actively developed modules. To make it feasible for volunteers to take over maintenance of the "downsized" modules, the modules need to be moved into their own source repositories, converted to be buildable in a standalone fashion against NetBeans platform & core IDE binaries (modules in main and contrib use a special legacy build convention), and otherwise cleaned up.

This page lists the steps I performed while running this process for the NetBeans Ruby support.

Moving Source Code

Identifying modules to be moved

First it is necessary to decide exactly which modules (~ top-level directories) need to be moved. Open nbbuild/cluster.properties and look for the cluster definition; these are all the modules normally built in the cluster.

glassfish.jruby
jellytools.ruby
libs.jrubyparser
libs.yydebug
o.jruby
o.jruby.distro
o.kxml2
o.rubyforge.debugcommons
ruby
ruby.codecoverage
ruby.debugger
ruby.extrahints
ruby.help
ruby.hints
ruby.javaint
ruby.kit
ruby.platform
ruby.project
ruby.railsprojects
ruby.rakeproject
ruby.refactoring
ruby.rhtml
ruby.samples.depot
ruby.testrunner
spellchecker.bindings.ruby

Also look around for other modules in the source tree which appear to be related, though they may not currently be built in that cluster, perhaps because they are poorly maintained. They should still be moved; the new developers can decide to fix them up and include them in builds, delete them, leave them in sources but unbuilt, etc. For example, searching by name:

main$ ls -1d *ruby*
glassfish.jruby
jellytools.ruby
libs.jrubyparser
o.jruby
o.jruby.distro
o.rubyforge.debugcommons
ruby
ruby.codecoverage
ruby.debugger
ruby.extrahints
ruby.help
ruby.hints
ruby.javaint
ruby.kit
ruby.merbproject
ruby.platform
ruby.project
ruby.railsprojects
ruby.rakeproject
ruby.refactoring
ruby.rhtml
ruby.rspec
ruby.samples.depot
ruby.testrunner
ruby.themes
spellchecker.bindings.ruby

turns up three new entries not in the cluster list:

ruby.merbproject
ruby.rspec
ruby.themes

http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastSuccessfulBuild/artifact/nbbuild/build/generated/deps.txt lists all module-to-module dependencies among modules in registered clusters (including the contrib repo), which is useful for verifying that other modules will not be broken by removing all Ruby-related modules. http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastSuccessfulBuild/artifact/nbbuild/build/generated/cluster-deps.txt is also useful for verifying that all the Ruby-related modules can be built against the basic IDE (platform, harness, and ide clusters) and that other clusters do not refer to the ruby cluster.

In this case you might have noticed the libs.bytelist and libs.jvyaml modules. But these are used by the languages.yaml module, which despite its Ruby associations is in the ide cluster (a dep of editor.kit): YAML text file support is for now at least a standard part of the base IDE distribution, since it is occasionally used in non-Ruby-based app frameworks.

Should also check for related modules in contrib:

contrib$ ls -1d *ruby*
ls: cannot access *ruby*: No such file or directory

but there appear to be none. If there were, these would likely need to be converted as well; removed from contrib; and the conversion result forcibly merged into the conversion result from main (so as to provide both sets of modules). As with modules in main not in the official cluster, these additional modules might or not not be used, but it would be best to make them available just in case.

Extracting module history into a separate repo

HgHowTos#Transfer_a_module.27s_history_to_another_repository is the basic technique; we will see how to apply that by example here, where the details will differ a bit.

First, make sure your version of the main repo (core-main in my case) is as up-to-date as possible:

$ hg fetch
$ hg fetch http://hg.netbeans.org/main-silver/
$ hg out
$ hg push

Now for the actual conversion. Currently Mercurial releases are incapable of doing this in a reasonable amount of time for a repo as big as main, due to http://mercurial.selenic.com/bts/issue991, so we need to use a patched version.

crew$ hg up 1.7.5
...
crew$ hg qpu --move convert-faster-991.diff
...

In my setup, .../testhg (in $PATH) runs Mercurial from its source tree:

#!/bin/sh
dir=.../crew
make -C $dir all 1>/dev/null 2>&1
export PYTHONPATH=$dir/build/lib.linux-i686-2.6
exec $dir/build/scripts-2.6/hg --traceback "$@"

so I can use this command where necessary:

$ testhg version
Mercurial Distributed SCM (version 1.7.5+1-05783ac9edbf)

For simplicity, we will assume that the converted modules should live in the same source structure as they do in main, so we can use the include directive but no rename directive in the convert command.

In order to capture historical branch information - i.e. changes made to IDE release stabilization branches to Ruby modules, usually backports of fixes from "trunk" - I will actually switch to another local repo which contains everything in http://hg.netbeans.org/core-main/ and http://hg.netbeans.org/releases/ too. This is optional but might be useful. Say a Ruby developer receives a bug report with a stack trace from NetBeans 7.0 beta 1 - they might like to check out the tip of the release70_beta branch to make sure line numbers in their sources exactly match the stack trace.

releases$ hg pull https://hg.netbeans.org/releases/
releases$ hg pull ../main
releases$ hg up default

(Running conversion from a separate local repo also makes it possible for me to do unrelated changes in my normal repo while the conversion repo is running.)

So let's start:

releases$ (for m in glassfish.jruby jellytools.ruby libs.jrubyparser \
               libs.yydebug o.jruby o.jruby.distro o.kxml2 \
               o.rubyforge.debugcommons ruby ruby.codecoverage ruby.debugger \
               ruby.extrahints ruby.help ruby.hints ruby.javaint ruby.kit \
               ruby.merbproject ruby.platform ruby.project ruby.railsprojects \
               ruby.rakeproject ruby.refactoring ruby.rhtml ruby.rspec \
               ruby.samples.depot ruby.testrunner ruby.themes \
               spellchecker.bindings.ruby; do echo include $m; done) | \
          time testhg convert --filemap /dev/stdin . ../ruby

This will do a few preparatory calculations, then start counting down changesets in historical order as it searches for relevant file modifications and converts them. Note that scanning gets slower per changeset as you get into newer history. You can watch progress from another shell window:

$ruby hg log -l1 -Mv

Time for a leisurely lunch!

8568.77user 740.19system 2:41:44elapsed 95%CPU (0avgtext+0avgdata 2158592maxresident)k

Removing modules from main

main$ hg rm glassfish.jruby jellytools.ruby libs.jrubyparser libs.yydebug \
o.jruby o.jruby.distro o.kxml2 o.rubyforge.debugcommons ruby ruby.codecoverage \
ruby.debugger ruby.extrahints ruby.help ruby.hints ruby.javaint ruby.kit \
ruby.merbproject ruby.platform ruby.project ruby.railsprojects ruby.rakeproject \
ruby.refactoring ruby.rhtml ruby.rspec ruby.samples.depot ruby.testrunner \
ruby.themes spellchecker.bindings.ruby
main$ hg ci

Removed also these entries from nbbuild/cluster.properties:

nb.cluster.ruby.dir
nb.cluster.ruby.depends
nb.cluster.ruby
validation.nb.cluster.ruby
clusters.config.ruby.list

Also removing nb.cluster.ruby from clusters.config.bloated.list, and removing ruby.themes from experimental cluster. Can also find some other mentions of the Ruby support that need to be cleaned up this way:

main$ hg locate -r . -0 nbbuild\*\* \\.hg\* | xargs -0 grep -l -- ruby

23c261cc6b0b is the resulting commit; 99934ed62e8e is the merge also reflecting the removal of the module sources.

(The .hgignore entries need to be copied to the new repo.)

Check for any remaining deps from main modules:

main$ ant init
Cannot find build prerequisite org.netbeans.modules.ruby.platform of .../contrib/portalpack.commons

Turns out PortalPack has some dependencies on the Ruby support. Details can be seen:

contrib$ hg locate -r . -0 portalpack\*\* | xargs -0 grep -l -- \[Rr\]uby

The dependency could very likely be made soft. However, the PortalPack modules are not currently built even in the experimental cluster config, so we will punt on this for now.

Publishing separate repo

Pushed to https://bitbucket.org/jglick/nbruby until a permanent home can be found, probably on hg.netbeans.org.

Preparing Standalone Build

Converting to Ant-based module suite

Using IDE, created a new module suite in a temp dir with the name 'ruby', against a platform named 'nb69'. Closed project. From command line, created nbproject dir in new repo, and moved three files into it from template project:

nbproject/project.xml
nbproject/project.properties
nbproject/platform.properties

Opened the root of the new repo as a project, which is now a suite; this generates some more files:

build.xml
nbproject/build-impl.xml
genfiles.properties

plus some nonversionable stuff in nbproject/private. Right-click project, Rename, type in 'NB Ruby' (do not move!); this just sets app.title in project.properties for display purposes. (Want to keep <name>ruby</name> in project.xml for use as cluster name.) Also in project properties, unselect all clusters from platform except

platform
harness
ide
nb

Can soon Add Existing... to add the modules beneath it, but this is not available for converting former netbeans.org modules until they are prepared a bit better; in the current state they are not even loadable as projects. First need to insert <standalone/> in each project.xml so they are treated as self-contained module projects:

ruby$ perl -pi -e 's!            <module-dependencies!            \
      <standalone/>\n            <module-dependencies!' */nbproject/project.xml

Now Add Existing... works. Check that we did all of them:

ruby$ fgrep standalone */nbproject/project.xml

Need to update our ignore file so we can see what we really need to add. We append to the imported entries from main to exclude build and nbproject/private subdirs of every project (suite & modules):

^o.jruby.distro/unpatched_source$
^o.jruby/patched_source$
(^|/)build$
(^|/)nbproject/private$

Now we can track all the new files:

ruby$ hg add

There is still a problem that every module's build.xml tries to refer to ../nbbuild/templates/projectized.xml, which is for netbeans.org modules only. We could have solved this simply by deleting build.xml before "Add Existing", causing them to regenerated with the correct import, but a lot of these build scripts have customizations we need to keep. So instead:

ruby$ perl -pi -e 's!\.\./nbbuild/templates/projectized\.xml!\
                     nbproject/build-impl.xml!' */build.xml

There's a lot more cleanup to do, but let's commit what we have so far: c8c08c1453fc

We will also handle */external/binaries-list files as ExternalBinaries describes: e54ce82201a2

Fixing up suite build

Time to try our first suite build. We have to make some corrections for different Ant target names in overrides, and make sure that external binaries are downloaded at the start of the build: 57a65875b2c5

Trying again, we run into a real problem:

ruby.build-init:
Cannot compile against a module: .../nb69/ide/modules/org-netbeans-modules-csl-api.jar
because of dependency: org.netbeans.modules.csl.api/2 > 2.11

Seems that there was some sort of incompatible change in the CSL API in 7.0, whereas for now we are building against 6.9. Inspection of the changeset bd130b29325c as it applied to the ruby cluster shows that no code in the Ruby support actually changed, meaning the change was probably bogus (original author should have merely pushed a fixed version of editor lib without touching downstream modules!) and we are safe just reverting the spec version in the dep: d08f4934b8b6

The next problem:

ruby.build-init:
Cannot compile against a module: .../nb69/ide/modules/org-netbeans-modules-editor.jar
because of dependency: org.netbeans.modules.editor/3 > 1.53

Again inspection of the changeset 10f8e107bf98 shows an increment in module deps for no clear reason; we will just back it out: deace5ee3cf6

Now we have:

ruby.build-init:
Cannot compile against a module: .../nb69/ide/modules/org-netbeans-modules-editor-lib.jar
because of dependency: org.netbeans.modules.editor.lib/3 > 3.1

Inspection of b6f862b9772e shows that the author made no code changes in the ruby cluster yet requested a new major version. This is simply wrong; he should have used a release version range instead. Again we will fix this: 79e7ae432654

Alternate: converting to Maven reactor tree

XXX

Locating platform binaries

XXX

http://hg.netbeans.org/community-uml/file/13ec0d76abee/build.xml http://source.apidesign.org/hg/netbinox/file/1075e21a049d/platform.xml

Setting up a Hudson job

XXX

Other Possible Steps

Issue tracking and mailing lists could be moved if necessary. (Requires site admin permissions.) Not done in this case.

Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo