NB68PlanQE
NetBeans 6.8 Quality Engineering Plan
Contents |
QE schedule
This QE schedule defines main QE activities to be done according to overall NetBeans68 / Internal page plans.
| Task Activity | Dates(Release) | Description |
|---|---|---|
| Milestone 1 testing | Jul 24-30 (Aug 3) | NB68M1GoNoGo |
| Feature readiness check | Aug 17-18 (Aug 19) | NB68ReadyForFeatureFreeze |
| Feature Freeze | Sep 3-4 (Sep 6) | NB68ReadyForFeatureFreeze |
| Beta testing | Sep 14-18 (Sep 21) | NB68BetaGoNoGo |
| A11Y/I18N testing | Sep 22-25 | NB68A11YandI18Ntesting |
| Solaris & Vista testing | Sep 28-Oct 2 | NB68TestingProgress |
| MacOS testing | Oct 5-10 | NB68TestingProgress |
| Submit 508 form | Oct 12 | as a result of NB68A11YandI18Ntesting & following bugfixing |
| Ubuntu testing | Oct 12-17 | NB68TestingProgress |
| Samples & Help testing | Oct 19-20 | NB68SamplesAndHelpTesting |
| Win XP testing | Oct 21-29 | NB68TestingProgress |
| Code Freeze | (Oct 30) | |
| RC1 Testing:Go/No Go | Nov 2-5:6 (Nov 9) | NB68RC1GoNoGo |
| RC2 Testing:Go/No Go | Nov 9-12:13 (Nov 16) | NB68RC2GoNoGo |
| FCS Testing | Nov16-19: (Nov 23) | NB68GoNoGo, NetBeans68InstallerTesting, NetBeans68UpgradeTesting |
Responsibilities & Coverage for particular features
The whole coverage matrix and table of responsibilities is available at NB68TestingProgress.
Manual tests :
- Features Coverage during release cycle ( in details) :
- Full test - for Primary Features and Quality Modules- run all Test Specifications on all platforms and default JDK(on specific platform) once a release, if time permits cover by "second supported JDK" / "JDK7"
- Acceptance test - for some Modules in Maintenance Mode - run all Test Specifications just on one platform and default JDK(on specific platform) once a release
- Sanity test - for some Modules in Maintenance Mode - run Sanity Test Specification or subset of all Test Specifications on one or more platforms and default JDK(on specific platform) once release
- Features Coverage for Milestones :
- for all Milestones/Beta/Release Candidates - run NB68SanityTestSpecifications on one platform and default JDK
- for FCS - run All Test Specifications on one platform and default JDK
Automated tests :
- Daily build validation - we will run NB68StableBinaryTestsDistribution and NB68FullPerformanceTestSuite regularly during development cycle against each daily build, beside that we'll run at least once a week tests those are "ready to be used" nowadays
- Milestones validation - we will run UICommitValidation and NB68FullPerformanceTestSuite on all supported platforms and both supported JDKs on each Milestone
Other testing areas
- A11Y and I18N testing - we'll do test I18N/A11Y only in areas those changed UI a lot in NB 6.8 or hasn't been tested yet
- Installer testing - we'll do test installers on all supported platforms and JDKs, as part of installation test QE team performs also minimal testing of the installed components functionality to verify successful installation. We will also test NetBeans 6.8 native packaging for OpenSolaris, Fedora and Ubuntu operating systems. (do also upgrade all modules test!!!)
- Download new feature testing - QE will do download new feature testing after that do sanity test of downloaded feature as preparation for upgrade testing (e.g. NetBeans671UpgradeTesting)
- Update Center plugins testing - QE will do upgrade/installation testing of new plugins provided on <nop>NetBeans Stable Update Centers at time of RC1 and FCS releases
- Import Settings testing - QE will perform test of importing settings from previous versions (the list of versions ?)
- Help and samples testing - QE will perform test of IDE integrated help from within the IDE. Test will focus on verifying if correct help topics are assigned to UI elements. Samples provided for IDE will be tested to verify particular functionality. Main focus will be on samples bundled within IDE. Samples provided on web will be tested if time permits.
- Bundles testing - NetBeans 6.8 will be delivered with J2SE 6.0 ( JDKNBBundleInstallerUninstallerTests), Java EE SDK and MySQL+GlassFish bundles. The testing will be performed on all NetBeans supported platforms by QE team.
Customer Acceptance
NetBeans 6.8 will have its own NB68NetCAT program. Its purpose will be to get community feedback on the main new functional areas and also to have untested features covered. It will start with Feature Freeze and finish with Release Candidate 1. Jiri Kovalsky will coordinate the program. Besides, Community Acceptance survey focused on release drivers will be organized to gather data for Phase 5 Exit presentation in the end.
QE deliverables
QE will be responsible for NetBeans 6.8 Bugs dashboard containing main quality indicators for NetBeans 6.8, NetBeans 6.8 Comparative Dashboard as well as detailed Performance dashboard. QE will report major bugs weekly and provide the update for bug statistics.
QE Risks
Risks are covered in Phase 2 document presented to PAC on July/15 2009.
Testing participants & responsibilities
| Area | Owner |
|---|---|
| Platform & BTD | IS - Ivan Sidorkin |
| Maven, Emb. browser | JPo - Jaroslav Pospisil |
| PHP | FZ - Filip Zamboj |
| Ruby | TM - Tomas Musil |
| Debugger | VS - Vojta Sigler |
| C-dev | JU - Jara Uhrik |
| Editor/Java | JPr - Jirka Prox, PPi - Peter Pis |
| Projects | PD - Petr Dvorak |
| Form, Support Modules | PPr - Pavel Pribyl |
| DB | MH - Martin Havle |
| VCS | DK - Daniel Krezelok |
| Java EE | MS - Martin Schovanek / MM - Michal Mocnak |
| JavaScript | MZ - Martin Zmrhal |
| Web, JSP, HTML, CSS | JS - Jindrich Sedek |
| XML, Installer/AU | MN - Michael Nazarov |
| Mobility | ACh - Andrei Chistiakov |
| Java FX | AS - Alexandr Scherbatiy, LH - Lukas Hasik |
| CnD | AP - Alexander Pepin |
| Profiler | MD - Matus Dekanek |
| Performance | OK - Oleg Khokhlov |
| Server Plugins | DN - Davis Nguyen |
| Other responsibilities | |
| A11Y | JU - Jara Uhrik |
| I18N | MV - Michal Vanek |
Other Planned Tasks
| Task | Owner | Deadline | Status | Description |
|---|---|---|---|---|
| Jelly split | VS | Jun/15 | DONE | |
| Setup Findbugs job for teams running out of SWAN | IS | Jul/31 | DONE | http://qa-findbugs.netbeans.org/ |
| Post-mortem analyses of NB 6.7 HR fixes | All | Jul/31 | DONE | |
| Organize once a 4-6 weeks Performance Focus test to get tested | OK | Jul/31 | DONE | July/29 - first round |
| Setup server with all supported Databases | MH | Jul/31 | Prepare server with latest versions of Databases as defined in NB68TestedPlatformAndRuntimes | |
| Collect data about our HW | MM | Aug/1 | ||
| NB68SanityTestSpecifications - create list of Sanity Test Specifications as basement for Go/No Go testing of Milestone/Beta/RC builds | All | Aug/15 | ||
| NB & Eclipse DB support comparison | MH | Aug/27 | ||
| NB 6.7 HR fixes - cover not yet covered parts | All | Aug/31 | ||
| Collect at least real-projects (at least 1 project for 1 feature) | All | Aug/31 | ||
| Automated tests for refactoring | JPr | Aug/31 | ||
| Stabilize Automated tests for junit,javahelp,form,properties (preferable cover Sanity Test Specification) | PPr | Aug/31 | ||
| Address feedback from first round of Performance Testing | OK | Aug/31 | ||
| NB68BinaryTestDistributionTasks - Binary Tests Distribution enhancements collecting & implementation | IS | Sep/15 | ||
| VCS servers consolidation | DK | Aug/31 | Collect info from link and link | |
| Usages/Statistics consolidation | FZ | Sep/15 | ||
| Jemmy fixes | IS | Sep/15 | ||
| Collect NB68KenaiIntegrationUseCases and ?implement fixes? | PD | Sep/4 | - use cases collected, enhancements reported | |
| c-dev functional tests for share/Create/login/open BZ query in Kenai | JU | Sep/15 | ||
| Automatize NB Installation/Upgrade test specifications | ? | Sep/30 | as documented for NB 6.7.1 : NetBeans671UpgradeTesting and NetBeans671InstallerTesting | |
