FitnessViaWhiteAndBlackList

This is a part of NetBeans Fitness project. The basic idea is to identify symptoms of various performance affecting tasks, write a test that will measure the number of these symptoms during certain operations and then work hard on lowering the number of such signs of misbehaviour. The expectation is that by preventing little regressions and by guaranteeing that our little fixes persist for ages, we can at the end make a significant progress. This is aligned with the vision declared in the clean the house manifest.

Indeed the ultimate goal of any improvements is to get end user satisfaction. However this is hard to measure quantity. As such we always substitute such satisfaction for something: number of bugs, time to startup, time to open a dialog, number of memory occupied after some operation, etc. These are more measurable, yet secondary signs of the ultimate goal, the satisfaction. The FitnessViaWhiteAndBlackList is a projection of this movement. It often offers measurements that are even more distinct from the ultimate goal. This is a kind of weakness, but it is balanced with the fact that the measurements are more exact, can catch the regressions or verify improvements more quickly. Moreover the symptoms are selected in a reasonable way, they are monotonic, e.g. an improvement in any of the symptoms does not result in disappointment of the end user. Thousands little clean ups can be miraculous!

Control Started Threads

NetBeans is multithreaded, however, as each new thread occupies 800KB of memory, it is unreasonable to just spawn new ones and let them waste such valuable resource. There is a whitelist to keep track of threads executed during common operations.

ThreadsTest in ide.kit module

Control I/O Reads and Open

During cold start, opening of each file is costly time consuming operation. For example the major speed up of cold start of during 6.1 release was caused by merging all the module JAR files into single cache file. The start got faster by 30%. As such we need to watch for all file reads and prevent them to improve start time. Every touch counts!

ReadAccessTest in ide.kit module

Control I/O Writes

While I/O reads and open are slow, yet sometimes unavoidable, writes during start are just plainly wrong. The declarative nature of NetBeans registrations and the performance team accent on doing things only when needed, shall prevent all writes during start. Such writes shall be replaced by more advanced declarative registration or delayed until requested functionality is really needed.

Location

The test is part of ide.kit tests.

How to reproduce test run

Open ide.kit project. Open Type "IDECommitValidationTest" and "IDEValidation" (search for method testWriteAccess and place breakpoint there), also see the "CountingSecurityManager". Debug the "IDECommitValidationTest", your breakpoint is hit at the moment when accumulated writes are about to be checked.

Control Use of Reflection

Reflective calls are costly. Especially first few calls to one method, when the system needs to verify the signatures and later generate an accessor proxy class. As such it makes sense to have them under control.

Location

The test is part of ide.kit tests.

How to reproduce test run

Test is written, but it is not enabled. Open ide.kit project. Open Type "IDECommitValidationTest" and "IDEValidation" (search for method testReflectionUsage and place breakpoint there), also see the "CountingSecurityManager". You need to enable the test by commenting out line in "IDECommitValidationTest" to start the "testReflectionUsage" test. Debug the "IDECommitValidationTest", your breakpoint is hit at the moment when accumulated writes are about to be checked.

Control what classes are loaded on NetBeans startup

BlacklistedClassesHandler tracks NetBeans classes loaded on startup. It performs checking according two lists: blacklist and whitelist. Blacklist is the list of classes which are not allowed to be loaded. Whitelist contains names of classes that are allowed to be loaded.

Location

BlacklistedClassesHandler is part of commit-validation suite (ide.kit\test\qa-functional)

Whilte list testing in not part of regular commit-validation suite run (it changes too often). Whitelist checking done by performance team (see Regular test runs section) and bugs are reported from time to time to various teams.

Blacklist file is ide.kit\test\qa-functional\data\blacklist.txt and whitelist files is ide.kit\test\qa-functional\data\whitelist_1.txt. These files can contain name of classes, comments (lines starting with #) or includes of other classes (line starts with #include filename.txt). The include directive saves the duplication as whitelist_X+1.txt includes whitelist_X.txt instead of repeating all its classes.

Configuration file is ide.kit/test/qa-functional/data/BlacklistedClassesHandlerConfig.xml

Regular test runs

Performance team runs whitelist checking each day on several platforms against Full IDE. By default warmup is disabled. Whitelist violators are collected during the week and then discussed on Performance i-team.

There are three whitelist tests now:

  1. 1st run with empty userdir
  2. 2nd run with empty userdir
  3. run with Tomcat6 project opened in previous NetBeans start

How to reproduce test run

The steps are as following:

  1. Invoke ant test in ide.kit/test/whitelist folder.
  2. Results are in ide.kit/build/test/qa-functional/work/o.n.t.i.W/testWhitelistN folders
  3. It is recommended to run ant unprepare after you finished with whitelist tests to remove quite big project from ide.kit/test/qa-functional/data dir. You could also remove temp dir if you're never going to run whitelist test again.

Do want to find who's loading your class? Either check the violation*.xml files generated by the test run, or use debugger's Class Breakpoint. The easiest way to launch debugger in mode fully similar to the test case is to run just N-th phase (N is 1, 2 or 3) with debug.port specified:

  • ant testN -Ddebug.port=XYZ and attach debugger to XYZ

Current state

List of currently unexpectedly loaded classes can be found at main performance builder page.

List of reported bugs contains in status field perf-whitelist

Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo