ExtractingStandaloneCluster

(Difference between revisions)
(Publishing separate repo)
(Identifying modules to be moved)
Line 76: Line 76:
In this case you might have noticed the libs.bytelist and libs.jvyaml modules. But these are used by the languages.yaml module, which despite its Ruby associations is in the ide cluster (a dep of editor.kit): YAML text file support is for now at least a standard part of the base IDE distribution, since it is occasionally used in non-Ruby-based app frameworks.
In this case you might have noticed the libs.bytelist and libs.jvyaml modules. But these are used by the languages.yaml module, which despite its Ruby associations is in the ide cluster (a dep of editor.kit): YAML text file support is for now at least a standard part of the base IDE distribution, since it is occasionally used in non-Ruby-based app frameworks.
 +
 +
Should also check for related modules in contrib:
 +
 +
contrib$ ls -1d *ruby*
 +
ls: cannot access *ruby*: No such file or directory
 +
 +
but there appear to be none. If there were, these would likely need to be converted as well; removed from contrib; and the conversion result forcibly merged into the conversion result from main (so as to provide both sets of modules). As with modules in main not in the official cluster, these additional modules might or not not be used, but it would be best to make them available just in case.
==Extracting module history into a separate repo==
==Extracting module history into a separate repo==

Revision as of 15:25, 19 February 2011

Contents

Introduction

Some module clusters in the NetBeans main repository are no longer actively developed or supported by the NetBeans team; the main repo is intended only for stable, actively developed modules. To make it feasible for volunteers to take over maintenance of the "downsized" modules, the modules need to be moved into their own source repositories, converted to be buildable in a standalone fashion against NetBeans platform & core IDE binaries (modules in main and contrib use a special legacy build convention), and otherwise cleaned up.

This page lists the steps I performed while running this process for the NetBeans Ruby support.

Moving Source Code

Identifying modules to be moved

First it is necessary to decide exactly which modules (~ top-level directories) need to be moved. Open nbbuild/cluster.properties and look for the cluster definition; these are all the modules normally built in the cluster.

glassfish.jruby
jellytools.ruby
libs.jrubyparser
libs.yydebug
o.jruby
o.jruby.distro
o.kxml2
o.rubyforge.debugcommons
ruby
ruby.codecoverage
ruby.debugger
ruby.extrahints
ruby.help
ruby.hints
ruby.javaint
ruby.kit
ruby.platform
ruby.project
ruby.railsprojects
ruby.rakeproject
ruby.refactoring
ruby.rhtml
ruby.samples.depot
ruby.testrunner
spellchecker.bindings.ruby

Also look around for other modules in the source tree which appear to be related, though they may not currently be built in that cluster, perhaps because they are poorly maintained. They should still be moved; the new developers can decide to fix them up and include them in builds, delete them, leave them in sources but unbuilt, etc. For example, searching by name:

main$ ls -1d *ruby*
glassfish.jruby
jellytools.ruby
libs.jrubyparser
o.jruby
o.jruby.distro
o.rubyforge.debugcommons
ruby
ruby.codecoverage
ruby.debugger
ruby.extrahints
ruby.help
ruby.hints
ruby.javaint
ruby.kit
ruby.merbproject
ruby.platform
ruby.project
ruby.railsprojects
ruby.rakeproject
ruby.refactoring
ruby.rhtml
ruby.rspec
ruby.samples.depot
ruby.testrunner
ruby.themes
spellchecker.bindings.ruby

turns up three new entries not in the cluster list:

ruby.merbproject
ruby.rspec
ruby.themes

http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastSuccessfulBuild/artifact/nbbuild/build/generated/deps.txt lists all module-to-module dependencies among modules in registered clusters (including the contrib repo), which is useful for verifying that other modules will not be broken by removing all Ruby-related modules. http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastSuccessfulBuild/artifact/nbbuild/build/generated/cluster-deps.txt is also useful for verifying that all the Ruby-related modules can be built against the basic IDE (platform, harness, and ide clusters) and that other clusters do not refer to the ruby cluster.

In this case you might have noticed the libs.bytelist and libs.jvyaml modules. But these are used by the languages.yaml module, which despite its Ruby associations is in the ide cluster (a dep of editor.kit): YAML text file support is for now at least a standard part of the base IDE distribution, since it is occasionally used in non-Ruby-based app frameworks.

Should also check for related modules in contrib:

contrib$ ls -1d *ruby*
ls: cannot access *ruby*: No such file or directory

but there appear to be none. If there were, these would likely need to be converted as well; removed from contrib; and the conversion result forcibly merged into the conversion result from main (so as to provide both sets of modules). As with modules in main not in the official cluster, these additional modules might or not not be used, but it would be best to make them available just in case.

Extracting module history into a separate repo

HgHowTos#Transfer_a_module.27s_history_to_another_repository is the basic technique; we will see how to apply that by example here, where the details will differ a bit.

First, make sure your version of the main repo (core-main in my case) is as up-to-date as possible:

$ hg fetch
$ hg fetch http://hg.netbeans.org/main-silver/
$ hg out
$ hg push

Now for the actual conversion. Currently Mercurial releases are incapable of doing this in a reasonable amount of time for a repo as big as main, due to http://mercurial.selenic.com/bts/issue991, so we need to use a patched version.

crew$ hg up 1.7.5
...
crew$ hg qpu --move convert-faster-991.diff
...

In my setup, .../testhg (in $PATH) runs Mercurial from its source tree:

#!/bin/sh
dir=.../crew
make -C $dir all 1>/dev/null 2>&1
export PYTHONPATH=$dir/build/lib.linux-i686-2.6
exec $dir/build/scripts-2.6/hg --traceback "$@"

so I can use this command where necessary:

$ testhg version
Mercurial Distributed SCM (version 1.7.5+1-05783ac9edbf)

For simplicity, we will assume that the converted modules should live in the same source structure as they do in main, so we can use the include directive but no rename directive in the convert command.

In order to capture historical branch information - i.e. changes made to IDE release stabilization branches to Ruby modules, usually backports of fixes from "trunk" - I will actually switch to another local repo which contains everything in http://hg.netbeans.org/core-main/ and http://hg.netbeans.org/releases/ too. This is optional but might be useful. Say a Ruby developer receives a bug report with a stack trace from NetBeans 7.0 beta 1 - they might like to check out the tip of the release70_beta branch to make sure line numbers in their sources exactly match the stack trace.

releases$ hg pull https://hg.netbeans.org/releases/
releases$ hg pull ../main
releases$ hg up default

(Running conversion from a separate local repo also makes it possible for me to do unrelated changes in my normal repo while the conversion repo is running.)

So let's start:

releases$ (for m in glassfish.jruby jellytools.ruby libs.jrubyparser \
               libs.yydebug o.jruby o.jruby.distro o.kxml2 \
               o.rubyforge.debugcommons ruby ruby.codecoverage ruby.debugger \
               ruby.extrahints ruby.help ruby.hints ruby.javaint ruby.kit \
               ruby.merbproject ruby.platform ruby.project ruby.railsprojects \
               ruby.rakeproject ruby.refactoring ruby.rhtml ruby.rspec \
               ruby.samples.depot ruby.testrunner ruby.themes \
               spellchecker.bindings.ruby; do echo include $m; done) | \
          time testhg convert --filemap /dev/stdin . ../ruby

This will do a few preparatory calculations, then start counting down changesets in historical order as it searches for relevant file modifications and converts them. Note that scanning gets slower per changeset as you get into newer history. You can watch progress from another shell window:

$ruby hg log -l1 -Mv

Time for a leisurely lunch!

8568.77user 740.19system 2:41:44elapsed 95%CPU (0avgtext+0avgdata 2158592maxresident)k

Removing modules from main

main$ hg rm glassfish.jruby jellytools.ruby libs.jrubyparser libs.yydebug \
o.jruby o.jruby.distro o.kxml2 o.rubyforge.debugcommons ruby ruby.codecoverage \
ruby.debugger ruby.extrahints ruby.help ruby.hints ruby.javaint ruby.kit \
ruby.merbproject ruby.platform ruby.project ruby.railsprojects ruby.rakeproject \
ruby.refactoring ruby.rhtml ruby.rspec ruby.samples.depot ruby.testrunner \
ruby.themes spellchecker.bindings.ruby
main$ hg ci

Removed also these entries from nbbuild/cluster.properties:

nb.cluster.ruby.dir
nb.cluster.ruby.depends
nb.cluster.ruby
validation.nb.cluster.ruby
clusters.config.ruby.list

Also removing nb.cluster.ruby from clusters.config.bloated.list, and removing ruby.themes from experimental cluster. Can also find some other mentions of the Ruby support that need to be cleaned up this way:

main$ hg locate -r . -0 nbbuild\*\* \\.hg\* | xargs -0 grep -l -- ruby

23c261cc6b0b is the resulting commit; 99934ed62e8e is the merge also reflecting the removal of the module sources.

(The .hgignore entries need to be copied to the new repo.)

Publishing separate repo

Pushed to https://bitbucket.org/jglick/nbruby until a permanent home can be found, probably on hg.netbeans.org.

Preparing Standalone Build

Converting to Ant-based module suite

XXX

Alternate: converting to Maven reactor tree

XXX

Locating platform binaries

XXX

http://hg.netbeans.org/community-uml/file/13ec0d76abee/build.xml http://source.apidesign.org/hg/netbinox/file/1075e21a049d/platform.xml

Setting up a Hudson job

XXX

Other Possible Steps

Issue tracking and mailing lists could be moved if necessary. (Requires site admin permissions.) Not done in this case.

Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo