GUI Test Stability Guidelines

Author: Adam Sotona, Jiri Skrivanek
Last Update: May 17, 2011

Contents


Introduction

After all work dedicated to improving stability of UI tests in Jemmy and JellyTools, it is not possible to take care about all situations. Test developers has to pay an extra attention to writing their test cases. First condition to a good automated test case is a manual scenario where all steps have to be verifiable. For atomic actions like JTabbedPane tab selection or JTree node selection, there is already verification implemented inside Jemmy. For more complex actions, test developer has to do verification himself. In majority of cases verification is done by next step in scenario. For example, pushing Tools -> Options main menu item will result in opening of Options dialog. Verification that the dialog is opened is done by command "new OptionsOperator();" because constructor of an operator waits until appropriate component is found. In this case it will wait until dialog with title "Options" appears. Instance of OptionsOperator is then used in the rest of test case. So there is no need for an additional verification.

In cases where an extra verification is needed, you can use Jemmy or JellyTools which gives us possibilities how to solve even non standard cases. If you don't find any instructions how to solve your particular case in this document, please, send us an email to qa(at)qa.netbeans.org.

JellyTestCase usage

JellyTestCase helps you with default initialization.

It is recommended to use org.netbeans.jellytools.JellyTestCase as ancestor for all test classes. It initializes jemmy and helps to easier resolving of failed tests. Its capabilities are:

  • redirect output messages from jemmy to file jemmy.log in working directory
  • wait at least 1000 ms between test cases (waiting is done by new EventTool().waitNoEvent(1000);)
  • create screen shot (screen.png) if test fails
  • close all modal dialogs if test fails
  • dump xml hierarchy of all components (disabled by default)

Description of JellyTestCase can be found in other documentation

Use Jemmy waiter

Jemmy waiter enables to wait for anything for given period of time.

Use jemmy waiter, if you want to wait for an action. You can wait until dialog dismiss, button is enabled, status line changed and so on. In the following example we want to wait until a connection dialog dismiss. Remember that we have to locate the dialog first and then wait until it is no more found. Default time to wait is 60 second and it can be changed as stated in example:

       // wait connecting dialog appears
       String connectingTitle = "Connecting";
       new NbDialogOperator(connectingTitle);
       // wait 30 second until connecting dialog dismiss
       try {
           JemmyProperties.setCurrentTimeout("Waiter.WaitingTime", 30000);
           new Waiter(new Waitable() {
               public Object actionProduced(Object title) {
                   return JDialogOperator.findJDialog(title.toString(), true, true) == null ? Boolean.TRUE : null;
               }
               public String getDescription() {
                   return("Wait until Connecting dialog dismiss.");
               }
           }).waitAction(connectingTitle);
       } catch (InterruptedException e) {
           throw new JemmyException("Interrupted.", e);
       }

Inactivity waiting

Wait for application inactivity.

If there is no point to be synchronized to and previous operation could affect next operation, it is recommended to use waiting for inactivity:

	new EventTool().waitNoEvent(time);

It will wait until no event is registered for a given number of milliseconds. The smallest recommended number is 500.

Focus on test cases

Use GUI actions for test cases only and for environment preparation use IDE API calls as much as possible.

Before test execution there are usually a lot of things to prepare and during these preparations lot of tests fail. It is recommended to split current test cases into preparation parts and testing parts. Preparation part could be performed using IDE API calls to avoid failure before main testing part. Main testing part than can be performed normally using Jemmy or JellyTools.

Example could be JavaCVS validation test suite where some settings must be changed before testing. Accessing and changing some settings in Options could be very unstable set of actions, but using direct API call is simple and stable one line of code:

   ((JavaCvsSettings)SystemOption.findObject(JavaCvsSettings.class, true)).setUiMode(1);

Use default Jemmy Dispatching Model

Use default Jemmy Dispatching Model rather than Robot Model.

Do not change Jemmy Event Dispatching Model. Use Robot mode only for cases where you need to simulate real-like user interaction.

Switching can be performed by:

   JemmyProperties.setCurrentDispatchingModel(JemmyProperties.ROBOT_MODEL_MASK);

or:

   JemmyProperties.setCurrentDispatchingModel(JemmyProperties.getDefaultDispatchingModel());


and locally you can change dispatching model:

   <operator instance>.setDispatchingModel(...);

Use Debug Timeouts

Testing machines have different performance than your workstation and occasionally they can be a little jammed during tests.

It is not recommended to use debug timeouts for all test cases because it can slow down test execution time rapidly. But they can help to stabilize section of a test case where other synchronization is not possible.

Timeouts should be changed before first test case by:

   JemmyProperties.getCurrentTimeouts().loadDebugTimeouts();

They can be reverted back any time during test execution by command:

   JemmyProperties.setCurrentTimeouts(new Timeouts());

Delay start of automated tests

IDE is usually very busy just after first start with empty user settings, so let IDE do its job first.

You can add some Thread.sleep(time) or new EventTool().waitNoEvent(time) into suite() method to postpone test execution until no events are generated by IDE itself. If you use JellyTestCase as ancestor for your classes, it already waits 1000 ms before each test case is started.

Prolong timeouts in critical sections

Default and sometimes even debug timeouts are not enough in some cases.

Timeouts should be prolonged in cases like waiting for dialog with response from some remote server (database, versioning, update center) or waiting for some longer actions (compilation, execution). In case of possible very long response do not block test for this long time but split critical section into invocation and verification part and verification part treat as unstable (see section below).

Timeout could be changed globally by:

   JemmyProperties.setCurrentTimeout(String name, long newValue);

or locally:

   <operator instance>.getTimeouts().setTimeout(String name, long newValue);

Set of timeouts can be loaded from external file by:

  JemmyProperties.getCurrentTimeouts().load(filename or input stream);

For example here is content of debug.timeouts file used by loadDebugTimeouts():

AbstractButtonOperator.PushButtonTimeout=100
ComponentOperator.AfterDragTimeout=100
ComponentOperator.BeforeDragTimeout=100
ComponentOperator.MouseClickTimeout=100
ComponentOperator.PushKeyTimeout=100
DialogWaiter.AfterDialogTimeout=3000
EventDispatcher.RobotAutoDelay=100
FrameWaiter.AfterFrameTimeout=3000
JComboBoxOperator.BeforeSelectingTimeout=100
JComponentOperator.ShowToolTipTimeout=1000
JMenuItemOperator.PushMenuTimeout=100
JMenuOperator.WaitBeforePopupTimeout=100
JScrollBarOperator.OneScrollClickTimeout=10
JSplitPaneOperator.ScrollClickTimeout=10
JTextComponentOperator.BetweenKeysTimeout=100
JTextComponentOperator.PushKeyTimeout=100
JTextComponentOperator.TypeTextTimeout=60000
JTreeOperator.WaitAfterNodeExpandedTimeout=100
WindowWaiter.AfterWindowTimeout=3000

Repeat unstable actions

Repeat unstable actions in case of exception but do not create infinite loops.

In case that some action is really unstable, repeat action invocation with catching possible exceptions. Restrict cycle to some number of tries or timeout and do not forget to add short sleep before action is repeated.

Use supported environment

Tests well-tuned on extraordinary environment probably fail on test servers.

Use supported window managers and default environment configuration during tests development. Some critical settings are:

  • disable focus follows mouse
  • enable to focus application windows when they first appear
  • enable raise windows when focused
  • enable dialogs to inherit focus from parents
  • set focus on click
  • disable display tooltips
  • disable screen saver on test servers
Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo