NB65NetCAT
NetCAT 6.5 Plan
NetBeans Community Acceptance Testing program
Goal of the program
NetCAT 6.5 will be run during stabilization of NetBeans 6.5 from Feature Freeze (not Milestone 2) to Release Candidate 1. Its primary purpose is to help test NetBeans 6.5, get early feedback on the main features and IDE quality. Secondary purpose is to serve as Customer Acceptance - ask NetBeans community for their Go/NoGo opinion about RC1 in a form of survey.
Organization
NetCAT 6.5 will be organized and lead by Jiri Kovalsky as the main engineer from Sun. Jiri will coordinate program and perform high level communication with community and internal engineering teams. He will closely work with QE engineers from each QE team responsible for main IDE areas. Areas and names of responsible QE engineers are available here. These engineers will watch feedback on the NetCAT mailing list to help users with issues and to better understand NetBeans 6.5 quality. They will also organize work inside their QE teams and prepare data for weekly NetCAT reports. They will also serve as primary contact for developers in case of hot threads. In addition Jiri will announce all the surveys, Betas, etc. to the NetCAT team. Test specification written for 6.5 will serve as a starting point for participants with lower knowledge in some IDE area.
Community participants selection process
NetBeans user community members who are willing to intensively help with testing NetBeans 6.5 will be asked to apply for participation some time before Feature Freeze. All volunteers must apply for program participation via a web form selecting their primary and secondary area of testing offer. We will select 60 members plus 30 substitutes based on their profile and recent activity in community. Only up to 1/3 of all people can be folks who participated in previous NetCAT programs. Inactive members (<20 CAT points) will be replaced in the mid of the program.
Infrastructure
This part talks about what tools will be made available mainly to plan on their timely development.
- NetCAT home page.
- Overall NetCAT quality dashboard will be available on some netbeans.org page and will include "NetCAT" column displaying bugs filled during NetCAT program.
- NetCAT perticipants use simplified IZ form for their issues. This form provides [[[65cat | [65cat]] prefix to each submitted issue subject.
- Mailing list netcat_at_netbeans_dot_org is created and all responsible QE engineers subscribe there to support users.
- Final CA survey will be available to ALL community.
Community Acceptance (CA) survey
- CA survey will be focused on critical-to-quality (CTQ) areas i.e. to get acceptance for NetBeans 6.5 release drivers.
- CA survey will not be accessible only to NetCAT participants but everybody from NetBeans community.
- If more than 20% of all survey respondents select particular IssueZilla bug as critical, it becomes RC showstopper by definition, it must be fixed and new RC build must be produced.
- If more than 30% of all survey respondents do not accept the RC build in some CTQ area, QE and Development departments will have to compile a detailed analysis evaluating all critical issues pointed out in the survey. Its purpose would be to either recommend the release or request more bugfixes and postpone it.
Apps to be developed:
Some of the applications will be running on qa.netbeans.org web.
- Participant subscription form
- Simplified IZ query/submission forms
- NetCAT dashboard (in case it's not part of the overall QE dashboard)
- CA survey form
Schedule
Schedule depends on overall NetBeans 6.5 schedule.
- 6/30 - Program announced on nbusers and nbdev lists. Volunteers submit their applications.
- 7/11 - Participants selected. Alternate members are selected as well.
- 7/14 - Feature Freeze. NetCAT team starts testing the latest daily builds.
- 7/21 - First weekly report sent.
- 8/11 - Milestone 2 release.
- 8/15 - Inactive participants replaced.
- 10/10 - Final CA survey questions defined.
- 10/22 - RC1 release. CA survey open.
- 11/3 - Last weekly report sent.
- 11/5 - Go/NoGo. CA survey closed, results processed.
- 11/7 - NetCAT game over
Rewards
NetCAT members will collect CAT points for their feedback:
| Activity type | CAT points |
|---|---|
| 1 | |
| RFE | 2 |
| Bug | 4 |
| CA survey | 10 |
| Other surveys | 5 |
| Docs review | 5 |
We will reward all active NetCAT participants (>40 CAT points) by a T-shirt and 3 best members (CAT points/days of service) by $50 vouchers at NetBeans shop.
