ExtractingStandaloneCluster

(Difference between revisions)
(Converting to Ant-based module suite)
(Removing modules from main)
Line 170: Line 170:
(The .hgignore entries need to be copied to the new repo.)
(The .hgignore entries need to be copied to the new repo.)
 +
 +
Check for any remaining deps from main modules:
 +
 +
main$ ant init
 +
Cannot find build prerequisite org.netbeans.modules.ruby.platform of .../contrib/portalpack.commons
 +
 +
Turns out PortalPack has some dependencies on the Ruby support. Details can be seen:
 +
 +
contrib$ hg locate -r . -0 portalpack\*\* | xargs -0 grep -l -- \[Rr\]uby
 +
 +
The dependency could very likely be made soft. However, the PortalPack modules are not currently built even in the experimental cluster config, so we will punt on this for now.
==Publishing separate repo==
==Publishing separate repo==

Revision as of 16:48, 19 February 2011

Contents

Introduction

Some module clusters in the NetBeans main repository are no longer actively developed or supported by the NetBeans team; the main repo is intended only for stable, actively developed modules. To make it feasible for volunteers to take over maintenance of the "downsized" modules, the modules need to be moved into their own source repositories, converted to be buildable in a standalone fashion against NetBeans platform & core IDE binaries (modules in main and contrib use a special legacy build convention), and otherwise cleaned up.

This page lists the steps I performed while running this process for the NetBeans Ruby support.

Moving Source Code

Identifying modules to be moved

First it is necessary to decide exactly which modules (~ top-level directories) need to be moved. Open nbbuild/cluster.properties and look for the cluster definition; these are all the modules normally built in the cluster.

glassfish.jruby
jellytools.ruby
libs.jrubyparser
libs.yydebug
o.jruby
o.jruby.distro
o.kxml2
o.rubyforge.debugcommons
ruby
ruby.codecoverage
ruby.debugger
ruby.extrahints
ruby.help
ruby.hints
ruby.javaint
ruby.kit
ruby.platform
ruby.project
ruby.railsprojects
ruby.rakeproject
ruby.refactoring
ruby.rhtml
ruby.samples.depot
ruby.testrunner
spellchecker.bindings.ruby

Also look around for other modules in the source tree which appear to be related, though they may not currently be built in that cluster, perhaps because they are poorly maintained. They should still be moved; the new developers can decide to fix them up and include them in builds, delete them, leave them in sources but unbuilt, etc. For example, searching by name:

main$ ls -1d *ruby*
glassfish.jruby
jellytools.ruby
libs.jrubyparser
o.jruby
o.jruby.distro
o.rubyforge.debugcommons
ruby
ruby.codecoverage
ruby.debugger
ruby.extrahints
ruby.help
ruby.hints
ruby.javaint
ruby.kit
ruby.merbproject
ruby.platform
ruby.project
ruby.railsprojects
ruby.rakeproject
ruby.refactoring
ruby.rhtml
ruby.rspec
ruby.samples.depot
ruby.testrunner
ruby.themes
spellchecker.bindings.ruby

turns up three new entries not in the cluster list:

ruby.merbproject
ruby.rspec
ruby.themes

http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastSuccessfulBuild/artifact/nbbuild/build/generated/deps.txt lists all module-to-module dependencies among modules in registered clusters (including the contrib repo), which is useful for verifying that other modules will not be broken by removing all Ruby-related modules. http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastSuccessfulBuild/artifact/nbbuild/build/generated/cluster-deps.txt is also useful for verifying that all the Ruby-related modules can be built against the basic IDE (platform, harness, and ide clusters) and that other clusters do not refer to the ruby cluster.

In this case you might have noticed the libs.bytelist and libs.jvyaml modules. But these are used by the languages.yaml module, which despite its Ruby associations is in the ide cluster (a dep of editor.kit): YAML text file support is for now at least a standard part of the base IDE distribution, since it is occasionally used in non-Ruby-based app frameworks.

Should also check for related modules in contrib:

contrib$ ls -1d *ruby*
ls: cannot access *ruby*: No such file or directory

but there appear to be none. If there were, these would likely need to be converted as well; removed from contrib; and the conversion result forcibly merged into the conversion result from main (so as to provide both sets of modules). As with modules in main not in the official cluster, these additional modules might or not not be used, but it would be best to make them available just in case.

Extracting module history into a separate repo

HgHowTos#Transfer_a_module.27s_history_to_another_repository is the basic technique; we will see how to apply that by example here, where the details will differ a bit.

First, make sure your version of the main repo (core-main in my case) is as up-to-date as possible:

$ hg fetch
$ hg fetch http://hg.netbeans.org/main-silver/
$ hg out
$ hg push

Now for the actual conversion. Currently Mercurial releases are incapable of doing this in a reasonable amount of time for a repo as big as main, due to http://mercurial.selenic.com/bts/issue991, so we need to use a patched version.

crew$ hg up 1.7.5
...
crew$ hg qpu --move convert-faster-991.diff
...

In my setup, .../testhg (in $PATH) runs Mercurial from its source tree:

#!/bin/sh
dir=.../crew
make -C $dir all 1>/dev/null 2>&1
export PYTHONPATH=$dir/build/lib.linux-i686-2.6
exec $dir/build/scripts-2.6/hg --traceback "$@"

so I can use this command where necessary:

$ testhg version
Mercurial Distributed SCM (version 1.7.5+1-05783ac9edbf)

For simplicity, we will assume that the converted modules should live in the same source structure as they do in main, so we can use the include directive but no rename directive in the convert command.

In order to capture historical branch information - i.e. changes made to IDE release stabilization branches to Ruby modules, usually backports of fixes from "trunk" - I will actually switch to another local repo which contains everything in http://hg.netbeans.org/core-main/ and http://hg.netbeans.org/releases/ too. This is optional but might be useful. Say a Ruby developer receives a bug report with a stack trace from NetBeans 7.0 beta 1 - they might like to check out the tip of the release70_beta branch to make sure line numbers in their sources exactly match the stack trace.

releases$ hg pull https://hg.netbeans.org/releases/
releases$ hg pull ../main
releases$ hg up default

(Running conversion from a separate local repo also makes it possible for me to do unrelated changes in my normal repo while the conversion repo is running.)

So let's start:

releases$ (for m in glassfish.jruby jellytools.ruby libs.jrubyparser \
               libs.yydebug o.jruby o.jruby.distro o.kxml2 \
               o.rubyforge.debugcommons ruby ruby.codecoverage ruby.debugger \
               ruby.extrahints ruby.help ruby.hints ruby.javaint ruby.kit \
               ruby.merbproject ruby.platform ruby.project ruby.railsprojects \
               ruby.rakeproject ruby.refactoring ruby.rhtml ruby.rspec \
               ruby.samples.depot ruby.testrunner ruby.themes \
               spellchecker.bindings.ruby; do echo include $m; done) | \
          time testhg convert --filemap /dev/stdin . ../ruby

This will do a few preparatory calculations, then start counting down changesets in historical order as it searches for relevant file modifications and converts them. Note that scanning gets slower per changeset as you get into newer history. You can watch progress from another shell window:

$ruby hg log -l1 -Mv

Time for a leisurely lunch!

8568.77user 740.19system 2:41:44elapsed 95%CPU (0avgtext+0avgdata 2158592maxresident)k

Removing modules from main

main$ hg rm glassfish.jruby jellytools.ruby libs.jrubyparser libs.yydebug \
o.jruby o.jruby.distro o.kxml2 o.rubyforge.debugcommons ruby ruby.codecoverage \
ruby.debugger ruby.extrahints ruby.help ruby.hints ruby.javaint ruby.kit \
ruby.merbproject ruby.platform ruby.project ruby.railsprojects ruby.rakeproject \
ruby.refactoring ruby.rhtml ruby.rspec ruby.samples.depot ruby.testrunner \
ruby.themes spellchecker.bindings.ruby
main$ hg ci

Removed also these entries from nbbuild/cluster.properties:

nb.cluster.ruby.dir
nb.cluster.ruby.depends
nb.cluster.ruby
validation.nb.cluster.ruby
clusters.config.ruby.list

Also removing nb.cluster.ruby from clusters.config.bloated.list, and removing ruby.themes from experimental cluster. Can also find some other mentions of the Ruby support that need to be cleaned up this way:

main$ hg locate -r . -0 nbbuild\*\* \\.hg\* | xargs -0 grep -l -- ruby

23c261cc6b0b is the resulting commit; 99934ed62e8e is the merge also reflecting the removal of the module sources.

(The .hgignore entries need to be copied to the new repo.)

Check for any remaining deps from main modules:

main$ ant init
Cannot find build prerequisite org.netbeans.modules.ruby.platform of .../contrib/portalpack.commons

Turns out PortalPack has some dependencies on the Ruby support. Details can be seen:

contrib$ hg locate -r . -0 portalpack\*\* | xargs -0 grep -l -- \[Rr\]uby

The dependency could very likely be made soft. However, the PortalPack modules are not currently built even in the experimental cluster config, so we will punt on this for now.

Publishing separate repo

Pushed to https://bitbucket.org/jglick/nbruby until a permanent home can be found, probably on hg.netbeans.org.

Preparing Standalone Build

Converting to Ant-based module suite

Using IDE, created a new module suite in a temp dir with the name 'ruby', against a platform named 'nb69'. Closed project. From command line, created nbproject dir in new repo, and moved three files into it from template project:

nbproject/project.xml
nbproject/project.properties
nbproject/platform.properties

Opened the root of the new repo as a project, which is now a suite; this generates some more files:

build.xml
nbproject/build-impl.xml
genfiles.properties

plus some nonversionable stuff in nbproject/private. Right-click project, Rename, type in 'NB Ruby' (do not move!); this just sets app.title in project.properties for display purposes. (Want to keep <name>ruby</name> in project.xml for use as cluster name.) Also in project properties, unselect all clusters from platform except

platform
harness
ide
nb

Can soon Add Existing... to add the modules beneath it, but this is not available for converting former netbeans.org modules until they are prepared a bit better; in the current state they are not even loadable as projects. First need to insert <standalone/> in each project.xml so they are treated as self-contained module projects:

ruby$ perl -pi -e 's!            <module-dependencies!            \
      <standalone/>\n            <module-dependencies!' */nbproject/project.xml

Now Add Existing... works. Check that we did all of them:

ruby$ fgrep standalone */nbproject/project.xml

Alternate: converting to Maven reactor tree

XXX

Locating platform binaries

XXX

http://hg.netbeans.org/community-uml/file/13ec0d76abee/build.xml http://source.apidesign.org/hg/netbinox/file/1075e21a049d/platform.xml

Setting up a Hudson job

XXX

Other Possible Steps

Issue tracking and mailing lists could be moved if necessary. (Requires site admin permissions.) Not done in this case.

Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo