Front Page

User FAQ
Developer FAQ
Community Docs
Popular Projects
Development
Quality
Bug DashBoard
Hudson
Sources
Nightly
Wiki Etiquette
Formatting Rules
Terms Of Use
Unlinked Pages
Undefined Pages
Categories
Page Index
Recent Changes
Find Page

[RSS]

Migration of NetBeans repository to Mercurial - the formal plan

Table of Contents

Goal

This document describes a plan of the migration of netbeans.org CVS repository to mercurial versioning system (Hg), lists the deliverables, tasks, schedule and points out issues and risks which were identified.

Constraints

The move has two constraints. It has to be finished as soon as possible and its impact on the work in progress on the 6.1 release of NetBeans IDE must be kept minimal. Most of the preparation must happen before the end of this calendar year. For the impact on all developers working on 6.1 in the main trunk to be minimal the actual switch must happen in January, after everything is prepared and tested and after all the developers are back from their holidays and a training has been delivered to them. See the Preparation, Training and Switch phases in the schedule below.

Deliverables

The list of deliverables of the migration includes the following items related to equipment and infrastructure, to the contents of the cvs.netbeans.org repository and to the ways engineers, content developers and automatic processes are using the data:

  • server - hg.netbeans.org as a replacement for cvs.netbeans.org - only for the data on main trunk and future development
  • all data included in the main trunk of cvs.netbeans.org in a changed directory structure - see details of the changed structure in content migration script details
  • cvs.netbeans.org server in a changed setup - (1) main trunk sources available only for read-only checkout using a named cvs tag, with the exception of */www folders which will be kept in the main trunk of cvs for web content developers; (2) certain groups will have access to the files on branches in read/write mode (e.g. sustaining)
  • user accounts for accessing the repository
  • automatic process of sending commit log notification messages to existing mailing lists which are managed at www.netbeans.org
  • ability to do concurrent development on separate branches in the main Hg repository (for individual development of features before they can be integrated into main trunk) and separate clones of the main Hg repository (for stabilization of releases - an equivalent of cvs branches, e.g. release60)
  • communication of changes and their impact for both internal and external users
  • training for all users of the repository

Non-deliverables

The following data will not be migrated:

  • history of binary files (only the latest trunk revision)
  • history of files deleted from the CVS trunk before the migration
  • history on CVS branches
  • most CVS tags (some important release-related tags will be retained)
  • web content (unless and when requested by the web team)

Team

The migration tiger team (nb-hg-migration@sun.com) consists of

  • Tonda Nebuzelsky (management, owner of the plan)
  • Pavel Kucera (infrastructure lab management)
  • Jesse Glick (development engineering)
  • Michal Zlamal (build engineering)
  • Jirka Skrivanek (quality engineering)

with some help of

  • Jack Catchpoole (issues related to collabnet and web)
  • Marian Mirilovic (quality engineering)

Tasks and Progress

Task Owner Deadline Dependencies Notes
Planning phase
Planning discussions the team Nov/16-22
Plan approved by management Tonda week of Nov/26-30
Draft of message for community prepared Tonda Nov/30
Message for community reviewed by evangelism Tonda Dec/6
Plan communicated to community Tonda Dec/6
Preparation phase
Server available Pavel done Server obtained from .sun organization. We have IP address and are able to login to the server.
Server backup setup Pavel done Backup will be running daily.
Server accessible as hg.netbeans.org Tonda done
Nov/29 		The DNS record is setup and working (#123051 ).
User account data available Jesse done
Dec/5 		We will try parsing out usernames from www.netbeans.org, issuezilla and previous commit logs in cvs.netbeans.org (e.g. users who commited to cvs.nb.org in the last year).
Apache HTTP authentication setup on hg server Pavel, Michal done
Dec/6 		user account data available? The authentication of users will be most probably done using Apache authentication.
Mailing lists mapping available Jesse done Mapping between commit log notification mailing lists and project folders in the repository is known.
Hg service setup on hg server Michal, Pavel done
Dec/6 		Apache authentication? Hg service will be setup on the server.
All IDE modules plus stable modules plus Dev AU modules buildable
All Javadoc and JNLP buildable
All content dumped in misc repo accounted for 		Jesse done
Dec/7 		see content migration script details
Consultancy contract signed with selenic.com to be able to ask for help and advise and get a timely response to bugs filed against Hg Tonda Dec/7
communication about issues is in progress 		Support from Selenic company may be highly desirable in case any problems are encountered after we switch to Hg as a production environment. We already know about some problems/bugs (e.g. see rename problem ).
Hg ambassadors team created Tonda done
Dec/14 		reps of all engineering teams which will be participating in the Wider Testing will become a team of Hg ambassadors who are able to help their teams with the start (see also HgMigrationChampionTeam)
Deliver first round of documentation and training materials to users Jesse done
delivered to team reps 		documentation prepared? We want to give the documentation to users as soon as possible so that the users can start learning about the change.
Build scripts Michal done
Dec/18 		Build scripts for production builds will be modified to reflect the migration. Production build will be tested internally.
Mail service setup on hg server Jesse, Pavel done
Dec/19 		mailing lists mapping available? There are two possible solutions: (1) mail server will be setup and a forwarding queue configured to reflect the commit log notification mailing list mapping, or (2) sending of the commit log notification mails will be done directly in Hg via a Python hook.
Tests in master xtest config buildable and seem to run Jirka, Jesse done Both automatic daily tests and tests invoked by developers have to continue working after the migration. Setup of test4u is ready and will be deployed right after the switch.
Binary repository Jesse, Michal, Pavel done
Jan/2 		Space on the server for the binary repository, upload script (CGI over HTTPS) accessible to authenticated users, ant script for using the binaries by users
Prepare documentation for users Jesse, Michal, Tonda done
Jan/6 		How-to document describing step by step procedures for beginners. Document describing the structure of the new repository in detail. General purpose documentation about Hg (for example the hgbook ). See also training details
Server populated with a fresh snapshot of data intended for the wider testing Michal done
Jan/2
Continuous build setup and started Michal done
Jan/4 		http://konik-z2.czech.sun.com:8080/hudson/job/NB-CB/
Migration team's sanity test of preparation phase completion the team done
week of Jan/2-4
Migration team's go/nogo the team done
week of Jan/2-4 		sanity test successful? GO
Testing guide ready Michal, Jesse done
Jan/6 		Guide for the wider testing task below.
BE production build setup and started Michal done The setup verified.
Evaluation of sufficiency of the Hg support in NetBeans IDE Jesse, others done
week of Jan/7-11 		Outcome of the evaluation will be included as recommendation in the training materials.
Evaluation of user scenarios and their performance Jesse, Michal done
week of Jan/7-11 		See also user scenarios evaluation and performance evaluation
Wider testing reps of all engineering teams (mandatory!) done
Jan/7-18 		Testing complete, received GOs from the team reps who participated in the testing.
Training phase
Training sessions Jesse, Michal, Alexey, Mikhail done
week of Jan/14-18 		Training sessions were delivered in BUR, MPK, PRG and SPB, and also with dial-in for those in US who could not directly attend.
Switch phase dependency on schedule of 6.1milestone1, this milestone has been branched out from main trunk so this is OK now HgMigrationChecklist
Server populated with final snapshot of data Jesse done
Jan/26,27
cvs.nb.org main trunk deleted Jesse done
Jan/26,27
Continuous build setup cleared for real start Michal done
Jan/26,27
BE production build setup cleared for real start Michal done
Jan/26,27
Temporary clones of the hg repository created on several RE servers Michal done
Jan/26,27 		The temporary clones are used for performance reasons. Users are encouraged to clone from these temporary clones instead of cloning directly from hg.netbeans.org.
Announcement sent to developers and community Tonda done
Jan/27 		all of above the announcement

Critical Issues

Issue Notes
2300 files have too long paths when migrated to Hg (see umbrella issue ) CVS paths were already approching the maximum as well. P1 bugs are being filed for top-level modules including these problematic paths and module owners have to fix it within a few upcoming days!

Risks

Risk Mitigation Plan
Setup of all the services may require more work than expected. Not enough time can be dedicated to prototyping the whole setup of hg.nb.org. Slip the Switch phase after Milestone 2 of 6.1 is stabilized on main trunk (to beginning of March), or even after 6.1 is branched out from main trunk and main trunk becomes a development vehicle for 7.0.
Support of hg.netbeans.org server is in a form of "best effort". Problem may be resolved within a few days. This means any issues related to the HW, network or the global zone can cause an outage of hg.nb.org server. A second virtual server will be soon obtained, this server will be setup as hg2.netbeans.org. In case of temporary failure of hg.netbeans.org, the users will be asked to switch to hg2.netbeans.org which will be an exact mirror of hg.netbeans.org.

TBD Stuff

Deadline TBD Notes
done CVS repository contains projects which are used only for testing - projects with only tests and no other sources, e.g. under performance/ The modules will be migrated either into main repo as other projects, or they will be migrated into misc repo
done Hook that rejects text files with CRLF Done, deployed on server
done Hook that enforces author name to be "My Name <something@netbeans.org>" or "something@netbeans.org"
done Changesets for any commits with a log message beginning with "Automated merge with " and including password should be rejected as these likely come from hg fetch.
done Add a server hook ensuring that people do not accidentally commit "raw" versions of external binaries.
done Forbid 2 heads pushes
done Forbid changesets which add files with longer path than 206 chars the storage path has to fix under 206 characters
done Ignore merge changesets in the mail notification process
done Make sure there are no remaining long paths in cvs A few files were not imported. See the comment in #122540.
done Unicode characters in changeset info should be handled correctly by the mailer.
done Create translatedfiles repository for l10n work and populate with content of translatedfiles/ from cvs. main/l10n repo ready and used
done Update the description for the repositories on the web of http://hg.netbeans.org
After switch Notification mails might be triggered for the whole changegroup, not for individual changesets To prevent flooding users by tons of emails when a big push happens.
After switch RSS feed setup with possible customization of the query
After switch Verisign Certificate for HTTPS access to hg.netbeans.org? Currently the site has a temporary certificate signed by us.

Sustaining