UML Test Plan For Meteora Release -- DRAFT

Author: Peter Lam
Revision: 0.3
Last updated: April 01, 2008

1.0 Test Plan Identifier

http://wiki.netbeans.org/attach/UMLQE/meteora-testplan.html

2.0 Introduction

This test plan describes the testing strategy for the UML module (Meteora) in the NetBeans 6.x release giving an overview of the testing that will be done to ensure the product meets requirements.

2.1 Objectives

The objectives of this test plan include:


2.2 Scope

This test plan addresses system test coverage, activities, responsibilities, tools, environment, schedule, and dependencies for the Meteora release.

2.4 Goals

The goals of testing include:

2.5 References

The following references support this test plan.

3.0 Test Items

3.1 Product Level Modules

The following list describes the product level modules to be tested under this plan:

4.0 Features to be Tested

All UML features in the Meteora release are the same as in the NetBeans 6.0 UML (Hydra) release, except for the Tom Sawyer graphics package used in the diagram drawing areas that will be replaced with the in-house developed graphics package. With the replacement of the graphics package, there will be changes in the drawing area UI that affects the following features.
The following list describes the other existing features to be tested under this plan:
Migration from the following releases will be tested.
The following UML documentation will be tested under this plan.
The accessibility (a11y) support of the tool will be tested under this plan.

4.1 Assumptions

Some part of codes have been or will be unit tested by development engineers who create or make changes to existing codes, but QE is not responsible for unit testing.

5.0 Features not to be Tested

Any product not listed in Features to be Tested. Other products or features not to be tested include but not limited to the following list.

6.0 Approach

The UML QE team consists of 2 full-time QE engineers in MPK who are assigned to certify the UML module in the Meteora release. One extra QE engineer will work part-time for around one to 2 months to help migrate the existing automated tests to Meteora. The UML QE PL will coordinate the certification effort in the QE team.

To prepare for testing the UML module, the QE team will learn the new UML changes and functionalities by reading the following UML functional specification documents, attending iTeam meetings and talking to development engineers. New test specifications will be created to cover any new features or functionalities that are not currently covered by the existing test specifications. Existing test specifications will be updated to reflect any GUI and functional change(s) based on the following functional specification documents.

6.1 Test Strategy

The test strategy is broken down into the following areas that are described in detail in each subsections below.

6.1.1 Test Specifications and New Test Development

Due to some changes in the drawing area for all diagram types, some test specifications will be updated to reflect the changes in the Meteora release. New test specifications for new features will be created as functional specifications become available or enough information to understand the new features or changes to existing features are available. New test cases are to be included in the test specifications.

From the past release, code coverage data was collected and will be analyzed for test coverage improvement in the Meteora release. New test cases will be added to enhance test coverage.

The document format for test specifications includes HTML, XML, and original Excel format. With the format in XML, test specifications can be presented in different ways using a predefined style sheet on web browsers, and test cases and associated test steps can be referenced in bug description and test result summary.

6.1.2 Automated Test Migration

Since the graphics package used in the diagram drawing areas have changed, all automated tests will be broken and need to be migrated to use the new API in the test library. How much can be done at this stage is still unknown until information from the development  team has been communicated. There is a possiblity that the test library can be fixed by replacing the original API with the new API from the new graphics package. For the worse case, all tests will have to be recreated from scratch.

6.1.3 Test Automation

All UML spec functional tests will be automated using XTest as the test harness, Junit, Jemmy and Jelly. The goal is to automate all UML tests if time allows. Test automation will be done in parallel with development. Any problem detected in the UML module during test automation will be logged in Issuezilla.

Due to limited QE resource in the Meteora release, testing will have to be depending mostly on automated tests plus manual user scenario testing. Once existing automated tests have been migrated to support in the Meteora release, the plan is to automate new test cases in core areas that are not yet been covered.

This is the list of UML features that tests will not be automated due to the reason indicated for each feature.

6.1.4 Regression and Development Testing

Regression testing will be performed to ensure the existing UML features function as expected compare to previous release. Due to the under-staffed QE team, regression testing will be performed on only high use case feature areas as well as high use case scenarios. QE will perform new testing along side with development on feature changes to ensure no regression. Any regression and/or defect detected in the builds will be logged in Issuezilla.

6.1.5 Backward Compatibility Testing

The Meteora release will support models created in previous releases. The following lists those releases which will be supported. As of such, backward compatibility testing will be performed. Any incompatibility or problem detected will be logged in Issuezilla.

6.1.6 Test Execution

The UML QE team will execute tests on Windows XP and other supported platforms for the release. Full set of tests will be executed on Windows XP and only limited sanity tests will be executed on all other supported platforms. Any problem detected in the UML module will be logged in Issuezilla.

Due to limited QE resource, QE will not be able to execute the full set of tests in the test specifications in every major milestone.  QE will have to depend on automated tests to cover the core functionalities of each feature while manual testing will be dedicated to excercise user scenarios. A selected list of core automated tests will be identified to be exeucted on the daily builds. The full set of automated tests will be executed twice or three times per week.

6.1.7 Documentation Testing

The UML QE team will perform limited documentation testing to only verify that the documentation, such as online help, for the UML module can be accessed. Only the actual content changes and new additions in the online help content will be reviewed due to time constraints. Tutorials will be reviewed and verified against recent builds. Any problem detected during the documentation testing will be logged in Issuezilla.

6.1.8 A11Y Testing

Due to the major changes in the diagram drawing areas, full a11y testing in the diagram drawing areas will be performed as well as other UI areas affected by the new graphics package replacement. All other areas will be randomly tested to ensure the a11y support is not broken. Any a11y defect detected during the testing will be logged in Issuezilla.

6.1.9  Code Coverage Data Collecting

The UML QE team will execute both automated and manual tests using instrumented builds to measure the overall code coverage. This task will be done on only one platform at the last phase of the development lifecycle when all planned features are completed and the product has been stabilized.

6.1.10  Test Matrix

The following table shows the test matrix indicating which test type will be executed on which supported platforms. This table will be updated as more updated information becomes available for the Meteora release to support.

Table 0: Test Matrix

Platforms
 All Features listed in Section 4
 Quick Start Tutorials Comment
Windows XP Pro SP2
 full testing
 X
Windows Vista Business limited sanity testing X
Solaris 10 x86 *
 limited sanity testing X
Solaris 10 x64 *
 limited sanity testing
 X
Solaris 10 sparc *
 limited sanity testing X
Ubuntu Linux 7.x
 limited sanity testing
 X
Mac OS X PowerPC
 limited sanity testing X
Mac OS X Intel
 limited sanity testing X
Note: * indicates testing on only one Solaris platform will be performed.

7.0 Entrance and Exit Criteria

Criteria for determining Pass/Fail/Error are as follows:

7.1 Entrance Criteria

Testing can "begin" when the following criteria is met for each milestone.

7.3 Exit Criteria

Testing is "complete" when all the following criteria are met for each milestone:

  1. All required tests have been run successfully on at least one supported platforms.
  2. Results from all tests have been analyzed.
  3. All quality issues identified and reported as bugs in Issuezillal.
  4. No P1 bug, a few P2 bugs are acceptable.

8.0 Suspension Criteria and Resumption Requirements

8.1 Suspension Criteria

8.2 Resumption Requirements

Testing will continue once the cause of the suspension has been resolved. If a suspended testing activity cannot be completed before the scheduled end of the testing period, it might not be resumed. QE will decide whether or not to shorten the test's activity's duration and resume the testing activity for the testing period.

9.0 Test Deliverables

The following are the test deliverables:

9.1 Test Specifications

Please refer to the following link for detailed Test Specifications for the Meteora release.

http://uml.netbeans.org/qa/test-specs-nb60.html

9.2 Test Status

Test status will be posted on the following QE wiki page: http://jupiter.czech.sun.com/wiki/view/UMLQE/WebHome


10.0 Testing Tasks

The UML QE team is responsible for all UML testing tasks listed in the following table.  For a complete list of UML feature assignments to each UML QE engineer , please look at UML Feature Assignment List.

Table 1: Testing Task Responsibilities
 
Testing Task
Responsibility
Test Documentation Peter Lam
Test Specifications
 UML QE Team
Test Development and Automation
 UML QE Team
Testing Infrastructure Design and Development
 UML QE Team
Dashboard
 UML QE Team
Test Execution/Verification UML QE Team
Test Reports UML QE Team


11.0 Environmental Needs

Meteora supported platform is the same as for the NetBeans 6.1 release. See NetBeans 6.1 release supported platforms for complete detail.

11.1 Hardware

The hardware listed in the following table are required to complete testing.

Table 2: Hardware List

Priority
 Platform (In Order of Marketing Priority)
 CPU Operating System
1
 Intel
 x86
 Windows XP Professional SP2
2
 Intel
 x86
 Windows Vista Business
3 Linux x86 Ubuntu Linux 7.x
4
 Intel
 x86
 Solaris 10
Desktop Environment = Gnome
5
 AMD
 x64
 Solaris 10
Desktop Environment = Gnome
6
 Sparc
 Sparc
 Solaris 10
Desktop Environment = Gnome
7 Mac x86 Mac OS X
8 Mac ppc Mac OS X

Table 3: System Requirements, Minimum and Recommended Configurations

Architecture
 Minimum
 Recommended
x86
(Windows, Linux and Solaris)
 CPU: PIII 500 MHz
RAM: 512 Mb

 CPU: PIV 1.4 Gz or higher
RAM: 1 Gb

SPARC
 CPU: UltraSparc II 450 MHz
RAM: 512Mb

 CPU: UltraSparc III 750 MHz or higher
RAM: 1Gb


11.2 Software

The following software are required to complete testing:

11.3 Other Dependencies

12.0 Responsibilities

12.1 QE Engineering

The UML QE Team will be responsible for the tests specified in this plan. The responsibilities will include:

12.2 Release Engineering

12.3 Development Team

12.4 Documentation Team

12.5 NetBeans 6.x Release P-team

13.0 Staffing and Training Needs

The UML QE team described in section 6 above will be required to carry out all the testing efforts included in this plan. Training of the QE team is an ongoing process on the need-to-do basis.

14.0 Schedule

The following dates are general dates based on the Meteora Plan. As such they are subject to change. A more detail schedule will be provided here as more information will become available from the development team.

Milestones Development Complete Date Stabilization Date Features/Functional Areas
M1 Jan 28, 2008 Feb 04, 2008
  • Class Diagram (Trey)
    • Creation of Class, Interface, DataType, Package, Template Class, and Derivation Classifier
    • Creation of Generalization, Implemenation, Associations, and Comment Link
    • Support for Icons Views- Interface, Boundary, Controller, and Entity
  • General Diagram (Sheryl)
    •  rectangular selection of multiple widgets
    •  move multiple nodes on the diagram at the same time
    • common popup actions
      • calculate common actions when multiple nodes are selected
      • Select All ( integrate with NB "Select All")
      • Select All Similar ( done by Viktor )
      • Invert Selection ( done by Viktor )
      • Cut ( integrate with NB cut/copy/paste/delete actions )
      • Copy
      • Paste
        • paste in the same diagram
        • paste to a different diagram
      • Delete
      • Select in Projects
      • Properties
  • Save Actions (Thuy)
    • how Diagram after load
    • Able to Rename Diagram
    • Mark diagram as dirty
  • Persistence: Class diagram: (Jyothi)
    • Class, Interface, Template Class
    • Datatype
    • Derivation classifier
    • Comments
    • Boundary class, control class, entity class
    • Package (and containment)
    • Generalization, Implementation
    • Association connector
    •  Link to comment
    • Stereotypes, tagged values
    • Labels
    • Icons, properties
  • UI Spec (Craig)
  • Sequence Diagram (Sergey)
    • creation of lifelines,actor lifelines, combined fragments, comments, all elements should be below zero level
    • message to self can be created by creation of asynchronous of synchronous message to starting lifeline
    • all lifelines should be connected to car on trackbar
    • creation of messages between all elements except comments
    • messages can be nested
    • messages can be moved, messages bump other messages when move
    • navigate to classifier should work from trackbar
    • messages reconnection should work, all working actions from 6.0, not working may not work too or be disabled
    • combined fragment allow operator change, operands addition, edit constraints, operands deletion, operands move
    • asynchronous and synchronous messages allow operations addition for selection as operation associated with the message, with corresponding label
    • all messages allow to show message name
    • diagram support next actions: show message numbers, show all return messages, show interaction boundary
  • Graph Library (Kris)
    • Layout Testing Tool (not in UML code base)
    • Orthogonal Layout working with Meteora library (not in UML code base)
    • Print Utilities prototype complete, API review process initiated.
M2 Mar 03, 2008 Mar 10, 2008
  • Class Diagram (Trey)
    • Redefines Operations Compartment
    • Comment Widget - Multiline Text
    • Dependency Edges - currently only on the class
  • General Diagram (Sheryl)
    • palette item drop suggestion ( D&D, click & drop, shift click & drop )
    • toolbar button
      • select tool
      • pan
      • marquee zoom
      • interactive zoom
      • export to image
      • discover relationship ( class diagram only )
      • sync diagram
      • navigate link
      • move forward / move backward / move to front / move to back
    • global font and color preference
  • JavaIO to FileObjects conversion for 6.1 (Thuy) - 2 weeks
  • Activity Diagram (Thuy) - 3 weeks
    • activity palette
    • invocation node
    • activity group (moved from M3 to M2)
    • activity edge
    • initial node
    • activity final node
    • flow final node
    • horizontal fork
  • Persistence: Sequence Diagram: (Jyothi)
    • Lifeline
    • Actor lifeline
    • Combined fragment
    • Comment
    • Labels
    • Asynchronous message
    • Synchronous message
  • Hg Migration (Jyothi)
  • Sequence Diagram (Sergey)
    • combined fragments containment and proper layering
  • Graph Library (Kris)
    • Export Image (JPG, PNG)
      • simple scene to image functionality
      • export html image map data
      • will drive api review process
    • Default Print (dependent on NetBeans review process)
      • will drive the api review process
    • Layouts
      • Orthogonal Layout (further testing required)
      • Hierarchical
M3 April 7, 2008 April 14, 2008
  • Class Diagram (Trey)
    • Association Qualifiers
    • Nested Link
      • Not persisting Yet
    • Association Class -- Started, Still have persistence and delete to work on.
    • CDFS
  • Infrastructure (Trey) - 1 week
  • General Diagram (Sheryl)
    • drag and drop multiple elements from project tree to diagram ( relationship discovery is yet to be tested )
    • resize element to fit
    • common popup actions
      • Color / Font
  • State Diagram (Sheryl) - 3 weeks
    • palette item & widgets
      • simple state
      • state transition
      • initial state
      • final state
      • aborted final state
      • merge bars - vertical and horizontal
      • composite state
      • pseudo states
  • Activity Diagram (Thuy) - 4 weeks
    • vertical fork
    • parameter usage
    • data store
    • signal
    • decision node (moved from M2 to M3)
    • partition
    • color & font integration for M2 nodes
  • Persistence: Sequence Diagram: (Jyothi)
    • Message to self
    • Create Message
    • Destroy
    • Persistance for special cases
      • save/load hidden result messages
      • Interaction boundary
      • Messages to/from Interaction boundary
      • Messages to/from combined fragment
  • Persistence: General (Jyothi)
    • Fonts, Colors (waiting for Trey/Sheryl's API)
    • Persistence Howto (Jyothi)
  • Sequence Diagram (Sergey)
    • combined fragment shouldn't cover lifelines/comments and smaller cf shouldn't be covered by bigger one
    • updates based on persistence realization
    • updates based on general actions (colors, fonts) (most actions should be ready for testing, blocked: waiting for Trey/Sheryl's API for fonts/colors)
    • RE Operation
    • CDFS on existing interaction
  • JavaIO to FileObjects Conversion (Sergey)
    • merge changes from 6.1 trunk
  • Graph Library (Kris)
    • Export Image (JPG, PNG)
      • possible performance enhancements (Viktor's encode work)
      • batch export (used in uml web report)
M4 May 12, 2008 May 19, 2008
  • Infrastructure (Trey) - 3 week
  • General Diagram (Sheryl) - 1 week
  • State Diagram (Sheryl)
    • palette item and widgets
      • choice
      • shallow history
      • deep history
      • junction state
      • submachine state
      • relationships
  • Activity Diagram (Thuy) - 1 week
    • persistence
    • CDFS
  • A11Y (Thuy/Sergey)
  • Use Case Diagram (Jyothi)
    • persistence
    • CDFS
  • Use Case Diagram (Jyothi)
    • use case widget
    • actor widget
    • include relationship
    • extends relationship
  • Graph Library (Kris)
    • Satellite view performance
    • performance issues
M5 Jun 16, 2008 Jun 23, 2008
  • Component Diagram (Trey)
    • component node
    • interface
      • port
      • assembly connector
      • connected interface
    • delegate edge
    • artifact node
    • persistence
    • CDFS
  • State Diagram (Sheryl)
    • persistence
    • CDFS
  • Web Report (Sheryl) - 1 week
    • save images
      • depends on save as image from graph library
    • diagram/model navigation
  • Deployment Diagram (Thuy)
    • node
    • deployment specification
    • persistence
    • CDFS
  • Import TS Diagrams (Jyothi)
    • Recognize/display TS Diagram nodes in project tree
    • Open TS diagram as a new Meteora diagram (blank)
    • Populate new Meteora diagram with content from old TS diagram
      • Nodes
        • TS to Meteora coordinate conversion
      • Edges
      • Edge labels
      • Connector waypoints
    • Remove/Archive TS Diagram (upon opening/converting TS diagram
      • Remove old TS diagram node from project tree
      • Move TS diagram files to project backup dir
    • TS diagram Conversion UI Implementation
      • Badged icon for TS diagram nodes
      • Dialog, upon opening TS diagram, to alert user of necessity to convert diagram
      • Option to never show dialog in the future
  • Collboration Diagram (Sergey)
    • collaboration lifeline
    • collaboration actor lifeline
    • connector
      • create operation
      • arrow widget
    • show message numbers
    • persistence
    • RE operation
    • CDFS on existing interaction
Feature Freeze June 23, 2008
TBD
Beta TBD TBD
FCS TBD TBD

15.0 Risks and Contingencies

This section addresses risks and contingencies for this test plan.

15.1 Staffing

Risk: Staff attrition could negatively impact the test schedule.
Contingency: Adding additional staff may or may not be an option. If not an option extend schedules and/or reduce the number of concurrent testing tasks.

15.2 Lab resources

Risk: Network, servers and test machines not in place or fully functional.
Contingency:

15.3 Tools/technology and test development

Risk: May be impacted if it is necessary to cover additional features not identified in the test plan or a sudden change in release schedule
Contingency: Adding additional staff may or may not be an option. If not an option extend schedules.

15.4 Software licensing dependencies

Risk: Many applications and tools require licenses. Insufficient number of licenses, or the expiration of existing licenses can impact the development and execution of  test suites.
Contingency: Acquire additional licenses or renewal of existing software licenses.

15.5 Other/third party test suites/applications

Risk: Support from vendors suddenly not available
Contingency: Acquire new test suites/applications, support and training from other vendors.

15.6 Product availability for milestone testing

Risk: Delays in handing off milestone builds to QE could negatively impact the test schedule.       
Contingency: Extend/modify schedules, negotiate lesser level of testing with all parties involved.

Risk:  The period allocated for Beta testing is too short
Contingency:  Carefully evaluate the priorities, continue product testing after Beta goes live.

Risk:  The period allocated for FCS testing is too short
Contingency: Extend/modify schedules.

16.0 Revision History 

 Table 4: Document revision history 

Version Modifications Author Date
0.1
 Initial release of document Peter Lam
 12/04/2007
0.2 Updated with feature/functional list for each milestone Peter Lam 01/30/2008
0.3 Updated with new feature/functional list for each milestone, updated links, etc. Peter Lam 04/01/2008
















17.0 Reviews and Approvals

Table 5: Document Approval Sign Off
 
Reviewer Title
Name
Approval Date
Author Peter Lam
UML I-team
George Vasick
QE Manager Tammera Krone
P-team
 ???




18.0 Acronyms

Table 6: Acronym List

Acronym Name or Term
Coco
 Java Studio Enterprise Coco Release
Griffin
 UML Release for NetBeans 5.5
GUI
 Graphical User Interface
Hydra UML Release for NetBeans 6.0
JDK
 Java Development Kit
JES Java Enterprise System
JVM
 Java Virtual Machine
Meteora UML Release for NetBeans 6.x
NetBeans (NB) NetBeans Open Source IDE
RE
 Release Engineering
SJSE Sun Java Studio Enterprise
UML
 Unified Modeling Language