Developers' NetBeans Mercurial How-To

General workflow

Installing Mercurial

Version 1.0 of Mercurial has been released. Use this version if available for your platform, it contains several fixes especially one regarding merges.

Please use Python version at least 2.4.4, version 2.5.1 is recommended

Binary packages should be the right choice for most users. See Mercurial installation tutorial for more information if the binary packages are not enough for you.

(Mac users: be sure to select the correct installer, depending on whether you use Mac OS X 10.4 or 10.5.)

(Solaris packages are at Opensolaris.org site

Another option for Solaris users is to use a Blastwave Mercurial package.

However, on Solaris, the binary packages don't always work, so if they don't, follow these steps (SPARC or x86):

• Make sure you have openssl installed. Check for the existence of /usr/local/ssl or /usr/sfw/include/openssl. If you do not, get it from http://www.sunfreeware.com and follow the build and installation instructions in the INSTALL file.
• Download source for python 2.4.4 from http://www.python.org and hg 0.9.5 from http://www.selenic.com/mercurial.

• Follow the instructions in the README in the python dir to build and install to /usr/local, but add this step between configure and (g)make:
  • edit Modules/Setup and uncomment the lines for SSL support (around line 200). If your ssl is not in usr/local, change the "SSL=" definition to point at the actual location.

• After building, test ssl support by running the SSL socket test

• Follow the instructions in the README in the hg dir to build and install to /usr/local. Make sure you use gmake instead of make and set the PYTHON= arg to point to your python location
• You will have an error for asciidoc during the last step in the build, but you can still proceed, you just may not have the man pages set up. There is an hg(1) page at http://www.selenic.com/mercurial/hg.1.html which you can use.
• If you have errors during debuginstall, follow http://www.selenic.com/mercurial/bts/issue632 : gunzip the

More Fedora RPMs

If you are using Windows, you probably want to convert newlines to CRLF in your checkout. Mercurial does not do this by default, but you can edit Mercurial.ini in your installation and make sure it includes

[extensions]
win32text =
[decode]
{}** = cleverdecode:
[encode]
{}** = cleverencode:

Even if you are not working on Windows, you may sometimes have files which may have CRLF in them. If you commit those files and then try to push them, your push command will fail due to the forbidcrlf hook on our servers. If you had made more commits after the one that commited the file with CRLF then it is harder to get rid of that problematic commit. So the best way is to prevent that situation in the first place. If you have Mercurial 1.0 or later this is easy. Just add the following hook to your Mercurial.ini on Windows, ~/.hgrc on Unix, or .hg/hgrc inside a repo.

[hooks]
pretxncommit.crlf = python:hgext.win32text.forbidcrlf

In fact just like the Ant target that installs the extension related to binaries i.e.

[extensions]
external = /home/sc32560/work/sun/netbeans/hgworkspacess/trunk/jsdebugger/nbbuild/antsrc/org/netbeans/nbbuild/extlibs/external.py

could also install the pretxncommit.crlf hook for the user.

Be sure to configure your "user name" in the configuration file. (Mercurial.ini on Windows, ~/.hgrc on Unix, or .hg/hgrc inside a repo.)

It should be your netbeans.org email address (based on your login ID). Example:

[ui]
username = jhacker@netbeans.org

Mercurial uses default HTTP ports (80 and 443), so if you use firewall, make sure that firewall permits connection to http://hg.netbeans.org/main for hg on these ports. If you have to use a proxy for HTTPS, try adding to your global config file e.g.

[http_proxy]
host = webcache.uk.sun.com:8080

Mercurial takes heed of the environment variable "http_proxy". Make sure to set it properly.

If one uses KDiff3 as a merge program, it is useful to use hgmerge.py script in order to avoid possible incorrect merges. More information about merge programs for mercurial can be found here. Example:

[ui]
merge = /path/to/hgmerge.py

[hgmerge]
interactive = kdiff3
noninteractive = diff3

Understanding the NetBeans repositories

The NetBeans CVS repository was split into several repositories:

• main - modules in the standard distribution and stable AU, build infrastructure, tests and test infrastructure, installer
• main/contrib - additional projectized modules, and other former contents of contrib.netbeans.org
• main/misc - old unprojectized modules, assorted Java SE projects not used by module builds, other uncategorized files (feel free to move anything current and useful into one of the other repositories)

See HgMigration for background.

There will likely be additional repositories that are special clones of the above, especially main. These function somewhat like branches in CVS:

• release61 (for example): branched sources for the 6.1 release (the repository currently on hg.netbeans.org by this name is for infrastructure testing only, it is not a real branch!)
• main-golden: a copy of main synchronized only after a successful continuous build. Thus it is always "safe" to pull from. If a build is broken, it will contain only older changes.
• core-main (for example): an integration repository for use by the Core team.

Getting a working copy: cloning the NetBeans repository

To start working with the main NetBeans Mercurial repository you need to create a clone of it.

$ hg clone http://hg.netbeans.org/main/
requesting all changes
adding changesets
adding manifests
adding file changes
added 5 changesets with 5 changes to 10 files

(The actual numbers will be much bigger, say 60-70k.)

Note that the initial clone may sometimes take a significant amount of time. Some users reported it took up to two hours for them.

If you have write access and are intending to push changes back, add a push path to your .hg/hgrc:

[paths]
default = http://hg.netbeans.org/main/
# Add this:
default-push = https://jhacker:secretpass@hg.netbeans.org/main/

(Replace jhacker with your netbeans.org username. Replace secretpass with your password, which during the dry-run period will simply be hguser. After the switch to a real repository you should get a real password in the mail.)

Every Mercurial repository is complete, self-contained, and independent. It contains its own private copy of a project’s files and history. A cloned repository remembers the location of the repository it was cloned from, but it does not communicate with that repository, or any other, unless you tell it to.

Understanding the structure of the NetBeans Mercurial repository

hg clone http://hg.netbeans.org/main/ will create a main directory under your current directory. (If you pass another argument, it will be used as an alternate directory name to create.) The main directory then contains a full copy of the http://hg.netbeans.org/main/ repository with checked out sources. The main/.hg directory contains all the Mercurial control files.

Most other directories under main are NetBeans modules imported from the CVS repository using this simple template: java/j2seproject -> java.j2seproject, core/progress -> api.progress, openide/loaders -> openide.loaders, etc.; the new directory name is predictably derived from the module's code name base, just removing some common prefixes and introducing some simple abbreviations. You can see the complete mapping in the new file nbbuild/translations.

The basic build infrastructure remains in main/nbbuild.

Doing your first build

Building NetBeans is the same as it was under CVS. (There is also a top-level build.xml which delegates to nbbuild/build.xml for convenience.) You need JDK 5.0u14 and Ant 1.7.0.

$ cd main
$ ant

Note: you need to be connected to the network for the first build, because it will fetch external binaries (JARs and ZIPs) from a server. You will also need to be online when doing the first build after some update to external binaries.

Note2: do not forget to have proper ANT_OPTS set - ie ANT_OPTS=-Xmx384m

Committing changes

Mercurial's commit operation will change only your local copy of the NetBeans repository. Example:

$ cd main
$ # edit some files...
$ hg ci

and enter a log message when prompted. (Note: you should make the first line of the log a complete sentence as a summary. Any additional lines can supply details.)

If you only want to check in some files or directory, just specify them:

$ hg ci java.j2seproject

Adding new files

After creating new files you just need to run

$ cd main
$ hg add form/src/org/netbeans/modules/form/SpiffyBuilder.java form/test/data/

Like cvs add, this addition will only take effect when you next commit.

More info about operations on files could be found in the Mercurial book.

Making changes visible to others

First check what changes will be propagated to the server:

$ cd main
$ hg out

Then if you are ready to really push your changes, run:

$ cd main
$ hg push

(If you checked out just http://hg.netbeans.org/main/, and did not configure an HTTPS URL for the default-push path as suggested above, you will need to specify this full URL as a command argument.)

If someone else has pushed changes in the meantime (since you last pulled), you need to pull their changes first and merge; see the next section.

Note: the server checks some basic conditions about your push before accepting it. In particular, if you tried to commit carriage returns (CRLF) in a text file, this will be rejected. You need to enable CRLF translation in your Hg settings.

Tip: if you are using Windows and you get a weird error message about SSL, check if your Internet Explorer settings say to use a proxy.

Updating your sources from the server

To update your sources, you can use:

$ cd main
$ hg pull -u

This will pull other people's changes from the server into your repository. It will also update your working copy to the latest version.

If you have made your own changes in this repository (even in a completely unrelated directory), you will need to merge your changes with other people's changes. This requires running hg merge.

A convenient way to pull and merge at the same time is to use the Fetch extension. Enable it:

[extensions]
fetch =

and run

$ cd main
$ hg fetch

Be warned that there are two bugs associated with this extension:

1. It does not merge correctly if you are also using the inotify extension. Since inotify has some other bugs too, it is best to not use it for now.
2. If your pull path contains a password, Hg 0.9.5 will include that password in a merge commit message. This problem does not arise if your pull path is anonymous, as recommended above. (You can still keep authentication information in your push path, which fetch does not use.)

Getting all repositories

If you want to retrieve all the repositories at once, install and enable the Forest extension. Then run:

$ hg fclone http://hg.netbeans.org/main/ nb_all

You should get nb_all for main sources, nb_all/contrib for extra modules, and nb_all/misc for additional files.

You can open and build experimental modules normally, e.g.

ant -f contrib/quickfilechooser/build.xml nbm

Backing out bad changes

In most cases, if you make a mistake and want to fix it, you should just edit sources to fix it and commit a new revision. This preserves history of the mistake and its correction and is suitable for routine programming errors.

If a whole big commit was wrong (e.g. someone else broke the build), it is too much work to revert it manually. For this case you can use hg backout to revert its effects. (If the bad commit was not the last, you will need to merge since the subsequent commits should still be in effect.)

If you commit something that you really cannot tolerate being in history (e.g. legally forbidden files, passwords, ...) it is possible to discard the commit only if you have not yet pushed it. (Once it is on the server, it is permanent.)

If you have just committed the change and nothing else, just run hg rollback. Your working copy will still be modified; you can hg revert if you need to.

Sometimes hg rollback cannot be used, but it is still possible to kill off bad changesets. The simplest way of doing this is to use the hg strip command, added by the MQ extension:

[extensions]
mq =

If you strip out a changeset from your local repository, it will be removed along with all descendant changesets. Be careful. (The command ought to save a backup somewhere just in case.)

You can also make a version of the repository that does not include the latest changes. Use hg clone -r and pass in the "last known good" changeset ID.

Server hooks (abort: pretxnchangegroup.something hook failed)

hg.netbeans.org has a number of server hooks in place to prevent common accidents from being committed permanently into the repository. The hooks are checking your changesets one by one.

• pretxnchangegroup.crlf - text files with CRLF newlines
• pretxnchangegroup.user - commits not made by an @netbeans.org author
• pretxnchangegroup.subject - commits made by the Fetch extension which include your password
• pretxnchangegroup.binary - external binaries added in raw format (see HgExternalBinaries)
• pretxnchangegroup.forbid_long_path - files too long (max 206 chars for e.g. misc/.hg/store/data/some/..../_file.java.i)

1. Use hg export -g to record the changes you made.
2. Use hg strip to pop off all the commits (unless the earliest ones are all OK).
3. Use hg import to reapply the changes, perhaps after fixing problems.

• pretxnchangegroup.forbid_2heads - you need to merge before pushing. hg heads can print the heads for you local repository. If you don't have more than one head then you are pushing your changes with hg push -f command which could create new heads on server, you need to run hg fetch first.
• pretxnchangegroup.undead - there are some files which were supposed to be deleted which you accidentally resurrected during a merge. (Hg bug #988) The .hgundead file in the repository lists files which are subject to this problem. The last revision you push must not contain any listed files.
  • Immediate solution: hg remove those bogus files, commit, and try again.
  • Real fixes:
    1. If there are some files present which prevent a build from continuing, make sure you are working against an up-to-date source tree - they might have deleted already by someone else.
    2. Never delete some files without committing the deletion and then continue other work, especially merges.
    3. Use ant hg-clean to delete untracked files safely, e.g. to do a clean build.

Working with release clones

Typical workflow for release clones is to do all the fixes on main and merge after QE verification. This involves the question how the fixes/changes/changesets could be easily moved from one to repository to another.

The easiest way is to use the Transplant extension. First you need to enable it in your ${HOME}/.hgrc:

[extensions]
transplant=
[defaults]
transplant = --log

hg -R release-clone transplant -s main REV1:REV2

The second more CVS-like method is to create a patch first and then apply it to the release-clone. So the sequence of the commands would be following:

hg -R main export --git REV1:REV2 > myfix.patch
hg -R release-clone import myfix.patch

Don't forget to push your changes when done.

Importing old development branches from CVS

The CVS server will still be available at least for read access after the switch. Say you had an old branch over the xml and websvc modules, named new_gui, with a base tag of new_gui_base.

cvs rdiff -r new_gui_base -r new_gui xml websvc > /tmp/mybranch-cvs.diff
ruby nbbuild/translate-patch < /tmp/mybranch-cvs.diff > /tmp/mybranch-hg.diff
hg import -p0 -m 'imported new_gui branch' /tmp/mybranch-hg.diff

(The translate-patch script should fix up filenames to match the new source layout. If your patch added completely new modules, you will need to translate these by hand, e.g. xml/newspi -> xml.newspi.)

(Note: if you get bizarre errors from the CVS server, make sure your CVSROOT ends in /shared/data/ccvs/repository rather than simply /cvs and try again.)

Not ready to merge your branch to main yet? Just make a local clone for your branch, import changes there, and continue development. Use hg merge to pull in changes from main so you do not fall out of synch.

Transfer a module's history to another repository

Rather complex but seems to work. To move mod from repo1 to repo2 you need something like this:

1. echo include mod | hg --config extensions.convert= convert --filemap /dev/stdin repo1 repo1-mod (extract just history of mod)
2. hg -R repo1 rem mod
3. hg -R repo1 ci -m 'transferring mod to repo2'
4. hg -R repo2 --config extensions.parentrevspec= pull -rtip^ -f repo1-mod (skip final .hgtags commit, permit pull from "unrelated" repo)
5. hg -R repo2 merge
6. hg -R repo2 ci -m 'imported mod from repo1'

plus a lot of careful sanity checking to make sure you are doing what you think you are doing.

If you want the module to live under a different relative path in the new repo, you should use

(echo include mod; echo rename mod newmod) | ....

to move it during the conversion. Otherwise you could use regular hg -R repo2 ren mod newmod at the end, but this will consume additional disk space (Hg bug #883).

Applying patches from another Hg branch on Windows

The command 'hg export' generates a patch in Unix encoding (LF) independently on hgext.win32text setup. So before applying a patch run some unix2dos utility.

1. cd .\hg_main_repo
2. hg export 63857 > ..\63857.patch
3. cd ..
4. unix2dos 63857.patch
5. cd .\hg_another_repo
6. hg import ..\63857.patch

Develop API review patches using MQ

For routine development you will simply make some changes, test, and commit. However in special circumstances you may prefer to polish a patch for review by others. In particular, API changes normally go through a prereview process. This is different from normal development because:

1. Changes must be reviewed in advance. In fact, the change could be rejected. If you are doing several reviews in parallel, it is quite possible you will commit them in a different order than you started.
2. Reviewers generally want to see the entire change as one patch. It is confusing to see you make a mistake in one patch and then fix it later.
3. The quality of the patch is especially important. You want it as small and clear as possible. If you find that some change was unnecessary, you want to delete it. If a new method was implemented in a confusing way, you want to rewrite it. This all ensures that reviewers see the smallest, clearest possible change.

For this style of development, MQ (Mercurial Queues) is ideal. This is a powerful tool bundled with Mercurial for managing patches. To get started, enable the MQ extension:

[extensions]
mq =

You must be sure to not pull (or fetch) with patches applied. If you are a Unix user you can just

[hooks]
prechangegroup.mq-no-pull = ! hg qtop > /dev/null 2>&1

You will probably also want to use Git-format diffs. This enhanced diff format captures file renames, among other things. You can either turn this on for all Mercurial commands:

[diff]
git = 1

or just for MQ:

[defaults]
qdiff = --git
qnew = --git
qrefresh = --git

In your NetBeans source repository, set up a patch queue:

hg qinit

(Use -c if you also want to version it as a separate Hg repo.)

Now let's say you are working on fixing bug #123456 when you realize that it requires an API change. Just make the API change as a local modification and try to use it - verify that it helps. Do not commit anything. Feel free to add, delete, or rename files as well as editing them (using hg add, rm, ren as usual).

Now you can initialize a patch for this change. Since you already have local modifications, use -f; add -e to give an initial log message:

qnew -e -f 123456.diff

Note: be sure that your log message does not start with # as this character is special in patches (marks a header) or a number. So do not use

#123456: a problem.

but rather

Issue #123456: a problem.

or similar.

Now hg stat and hg di will show nothing - because you have no modifications beyond what is saved in the patch. hg tip will list a special temporary changeset for your working patch, because your patch is applied, as you can see with hg qapp.

When you have submitted the issue for review (keywords API API_REVIEW_FAST, assigned to apireviews), attach the patch directly from .hg/patches/123456.diff, or you can run hg qdi to produce a patch that also shows the base revision. Add a reminder to your calendar to commit the patch in a week if there are no objections.

hg qpop -a to go back to unpatched sources and go about your normal work. (Do not pull or push while you have patches applied!)

If reviewers request some changes in the following days, hg qgo 123456 to reapply patch; make whatever changes you want. hg di will show changes from the last saved patch. To update the saved patch, use hg qref, at which point you can sanity-check it using hg qdi. Then just attach the new version as before so reviewers can see the revised patch.

If you'd rather not attach every revision, you can also mail the patch easily: hg email qtip will send it out to interested parties. (Use -n to preview what would be mailed before you really do it!) You need a little configuration, for example:

[smtp]
host = mail-amer.sun.com
username = jh102201
tls = true
[email]
from = John Q. Hacker <jhacker@netbeans.org>
to = apireviews@netbeans.org
cc = ,

If some changes are made to the repository which causes your patch to not apply cleanly after you last worked on it, hg qpush or hg qgo will show some warnings and leave behind *.rej files containing "hunks" of the patch which did not apply. You need to look at these and figure out how to apply the equivalent to the new code. (It is possible but quite tricky to run a standard 3-way merge tool to resolve conflicts; a future version of MQ may make this more friendly.) hg qref when you have everything the way you want it again.

If your review is accepted, you can easily commit the patch as a permanent change (and remove it from the queue):

hg qpop -a
hg import .hg/patches/123456.diff
hg qdel 123456.diff

and you can then push as usual.

If you have multiple active patches, bear in mind that in MQ patches are applied in series. This means that if hg qser prints

123321.diff
123456.diff

and the two patches touch related files, then 123456.diff may depend on 123321.diff and build on top of it. If you want to ensure the patches are independent, just hg qpop -a, edit .hg/patches/series to reorder your patches, and run hg qgo 123456.diff to make sure it applies on its own.

Want to know more about MQ? Try chapters 12 and 13 of the Mercurial book.

$ hg init hgtest-how-to-merge
$ cd hgtest-how-to-merge
$ echo one > f
$ hg ci -A -m one
adding f
$ echo two > f
$ hg ci -m two
$ hg log
changeset:   1:de9c1703a8d3
tag:         tip
user:        ....
date:        ....
summary:     two

changeset:   0:58d574e6ba6e
user:        ....
date:        ....
summary:     one

$ hg clone -r 0 . dupe
requesting all changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd dupe
$ echo three > f
$ hg ci -m three
$ hg log
changeset:   1:04d4360ce505
tag:         tip
user:        ....
date:        ....
summary:     three

changeset:   0:58d574e6ba6e
user:        ....
date:        ....
summary:     one

$ hg fetch ..
pulling from ..
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files (+1 heads)
merging with new head 2:de9c1703a8d3
merging f
0 files updated, 1 files merged, 0 files removed, 0 files unresolved
new changeset 3:2ac5c85658f0 merges remote changes with local
$ cat f
two and a half

$ hg glog
@    changeset:   3:2ac5c85658f0
|\   tag:         tip
| |  parent:      1:04d4360ce505
| |  parent:      2:de9c1703a8d3
| |  user:        ....
| |  date:        ....
| |  summary:     Automated merge with file:/tmp/hgtest-how-to-merge
| |
| o  changeset:   2:de9c1703a8d3
| |  parent:      0:58d574e6ba6e
| |  user:        ....
| |  date:        ....
| |  summary:     two
| |
o |  changeset:   1:04d4360ce505
|/   user:        ....
|    date:        ....
|    summary:     three
|
o  changeset:   0:58d574e6ba6e
   user:        ....
   date:        ....
   summary:     one

$ cd ..
$ hg clone -r 0 . dupe2
requesting all changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd dupe2
$ echo three > f
$ hg ci -m three

$ HGMERGE=merge hg fetch ..
pulling from ..
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files (+1 heads)
merging with new head 2:de9c1703a8d3
merging f
merge: warning: conflicts during merge
merging f failed!
0 files updated, 0 files merged, 0 files removed, 1 files unresolved
There are unresolved merges, you can redo the full merge using:
  hg update -C 1
  hg merge 2
$ cat f
<<<<<<< /tmp/hgtest-how-to-merge/dupe2/f
three
=======
two
>>>>>>> /tmp/f~other.6OGVvm
$ echo 'two and a half' > f
$ hg di
diff --git a/f b/f
--- a/f
+++ b/f
@@ -1,1 +1,1 @@ three
-three
+two and a half
$ hg ci -m merged
$ hg glog
@    changeset:   3:081a22ed714b
|\   tag:         tip
| |  parent:      1:6b06aa95da85
| |  parent:      2:de9c1703a8d3
| |  user:        ....
| |  date:        ....
| |  summary:     merged
| |
| o  changeset:   2:de9c1703a8d3
| |  parent:      0:58d574e6ba6e
| |  user:        ....
| |  date:        ....
| |  summary:     two
| |
o |  changeset:   1:6b06aa95da85
|/   user:        ....
|    date:        ....
|    summary:     three
|
o  changeset:   0:58d574e6ba6e
   user:        ....
   date:        ....
   summary:     one

Hg equivalents of common CVS commands

CVS checkout

Mercurial has no direct equivalent to the CVS checkout command; you always have the entire repository with history on your computer.

$ hg clone http://hg.netbeans.org/main/ nb_all

The Mercurial command checkout is simply an alias for the update command. This does no network operations but might be used to switch your working directory to an older revision.

CVS update

hg pull -u (possibly followed by hg merge and hg ci), or hg fetch, is the closest equivalent to cvs up.

CVS commit

hg ci followed by hg push together act similarly to cvs ci.

CVS "what-if" update

You can use the -p (patch) option to either incoming or outgoing. This provides something similar to cvs -n up or cvs -n ci to preview what will be pulled or pushed. Example:

cd /path/to/repo1
hg in -p http://hg.netbeans.org/repo2/

.cvsignore

CVS has a .cvsignore file in each directory which can match files or subdirectories directly in that directory using a simple glob syntax.

Hg uses a single .hgignore file at top level which lists ignored files or subdirectories in the whole repository using a choice of syntax, by default regular expression.

The initially created NB repositories will have .hgignore files created by inspecting the old .cvsignore files (which are not copied over). Some general patterns cover a lot of cases, e.g. /nbproject/private$ ignores any such dir generically. If you need to add new special-case ignore patterns, just edit and commit this file.

External links to CVS=>Mercurial transition

1. CVS Concepts
2. CVS=>Mercurial commands table

More Tips and Tricks

http://www.selenic.com/mercurial/wiki/index.cgi/TipsAndTricks

Problems?

File a bug about the NetBeans Hg repositories and their usage.