This is an overview of NetBeans performance work.
Defining Performance Criteria
TBD: startup time and other startup metrics (disk touches, loaded classes) in various conditions, UI responsiveness, scalability, raw performance of important operations, memory consumption, etc.
Performance testing is mostly focused on already analyzed or fixed problems, mainly for catching regressions (done repeatedly, keeping history data). Some of the tools used for testing can also help discover new problems, e.g. the Slowness Detector described below.
Comparative dashboard. In regular intervals (monthly), we run manual testing of selected features to compare our product with previous releases and competitive product. Internal link only!
Self-profiling actions. Combination of manual and automatic approach. Toolbar icon allows to start and stop CPU sampler, then opening the created snapshot in the IDE. Slowness-detector automatically starts and stops the sampler when it detects blocked AWT event queue for certain time. It also allows to send the snapshot via the exception reporter infrastructure. Here's an example of a report.
QE runs in regular intervals. Results available in performance dashboard.
Responsiveness Tests. Read Response Time Overview article. In NetBeans, we have adjusted these criteria:
- Menus (100 ms category) - main menus & contextual menus
- Dialogs (1000 ms category) - dialogs, wizards, windows
- Actions (100 ms, 1s and 10s category) - open/close file, create/open/close project, typing in editor, code completion invocation
Time measuring tests. Unit tests measure CPU-time consumption of selected actions (methods). Based almost on loggers. They can contain boundary or just measure the time and report them to performance dashboard. Examples: tests measuring scanning (e.g. test for the first project scan after opening) or various refactoring operations.
Note: In some specific cases, the tests are not reliable, because they are affected by other things: disk speed, network access etc.
Memory leak tests. There is an infrastructure available for testing memory leaks. See DevFaqMemoryLeaks.
Startup tests. Measuring cold startup of an empty IDE, cold startup of the IDE with a project opened, also measuring overall time after startup needed to finish up-to-date scan of the project; all that for different size of the IDE (basic, full).
Startup time is measured via a special logger that logs individual stages of the startup sequence and time taken. Comparing logs from different builds is useful for finding regressions. The logger can be turned on via -J-Dorg.netbeans.log.startup=print command line switch.
Counting tests. Used to detect redundant file reading/writing, especially during cold startup when the I/O performance is critical. The tests can be used in general for checking that some code should not touch disk.
Note: Sometimes they do not detect real problem. Accessing one file many times often means that first time call is very slow, other ones are very fast. Reducing the number of calls does not automatically mean that the real consumed time will be visibly reduced.
Whitelist and blacklist. Watching classes loaded during startup, see FitnessViaWhiteAndBlackList.
Evaluation and Profiling
Analyzing performance problems...
Performance Team Tools
VisualVM. All in-one troubleshooting GUI tool. Becoming very popular. Used for monitoring application, generates heap and thread dumps and much more. See homepage and its screencast for details.
NetBeans Profiler. Integrated to the IDE, very easy to use when developing in NetBeans. Full-featured tool, CPU and Memory profiling, heap walker.