Test Attribs in Use

We have Test distribution now, so the next step is to distinguish used test attributes and set rules to use them.

We would like to use following general test attribs :

• commit
• stable

"commit" test suite

• Purpose :
  • the most stable test suite to cover main features of the IDE, it has to be small and doesn't take long time to run
• Test run:
  • before commit
  • after new build
• Test suite :
  • current
    • module="core" testtypes="unit" attributes="commit"
    • module="core/windows" testtypes="unit" attributes="commit"
    • module="core/javahelp" testtypes="unit" attributes="commit"
    • module name="ide" testtypes="qa-functional" attributes="commit"
    • module name="editor" testtypes="qa-functional" attributes="commit"
  • proposed
    • testtype="unit" attributes="commit" from all modules
    • testtype="qa-functional" attributes="commit" from all modules
• Policy :
  • http://www.netbeans.org/community/guidelines/commit.html
• Owner :
  • module owners* Owner :
  • module owners
• If fails :
  • report a regression - P2, against appropriate module
  • remove test from the stable test suite
• If blocks machine :
  • report P1
  • remove test form the stable test suite
• Add new module with tests :
  • if test doesn't block all supported machines

• If fails :
  • report a regression - P1, against appropriate module or test

"stable" test suite

• Purpose :
  • stable test suite to cover main features of the each module
• Test run:
  • to verify quality of the new build
  • once a day on all supported machines (XP,Lin,Sol)
  • results have to be evaluated daily
• Test suite :
  • current
    • many modules with different attributes
  • proposed
    • testtype="unit" atributes="stable" from all modules
    • testtype="qa-functional" atributes="stable" from all modules
• Owner :
  • module owners
• If fails :
  • report a regression - P2, against appropriate module
  • remove test from the stable test suite
• If blocks machine :
  • report P1
  • remove test form the stable test suite
• Add new module with tests :
  • if test doesn't block all supported machines

"stable-j2ee" test suite

• Purpose:
  • some tests depends on installation of appserver.
• Test run:
  • TODO
• Owner :
  • module owners
• If fails :
  • report a regression - P2, against appropriate module
  • remove test from the stable test suite
• If blocks machine :
  • report P1
  • remove test form the stable test suite
• Add new module with tests :
  • if test doesn't block all supported machines