NetBeans Fitness Program
Home page for NetBeans 6.1 performance.
Fixing NetBeans Performance After 6.0
This is a home page of the NetBeans 6.1 performance fixing effort (aka NetBeans Fitness Program). The goal is to fix various performance problems that accumulated from the 6.0 and earlier versions. Our measuring and feedback we receive indicate clearly that the performance is perceived as a serious problem. The IDE starts slowly, feels a bit too sluggish and over-weighted, consumes lots of memory. It is noticeably slower that the previous versions and the competition. See what people say. We need to perform some exercises...
What Makes a Good Performance
Several things together constitute a good performance. Not only the raw performance of selected tasks is important, but also perceived responsiveness of the IDE (how it can keep up with the user and how it indicates progress, etc), scalability (with size of projects, size of the IDE, system configuration), memory consumption, and startup time.
Obviously, the performance is critical in the most visible and most frequently used areas. We need to know main use cases and typical user work-flow (order of steps, repeated steps). Then we know what to primarily optimize for performance, how to define the performance criteria. In general we consider these areas most important:
- cold startup (with some projects opened),
- invoking menus and dialogs (anywhere),
- navigation in projects and files (opening, browsing, jumping to files, repetitive actions),
- creating new projects/files,
- editing (responsive code writing, language features, visual designing),
- execution/deploying (productive edit-compile-debug cycle).
There are some general short-term and long-term goals.
In short term the time is limited by the 6.1 release. Note that even though performance is a top priority for 6.1, we can't cover everything in such a short time. So the main goal is:
- Deliver visible improvements in critical areas. The effort will be user oriented, focusing on the main use cases/tasks and problems. We'd like all critical areas to have reasonable performance and no regressions from 6.0.
For more details see the plan below.
In longer term (post 6.1), there are several goals:
- Continue improving the performance in all areas, including complex and radical changes where needed (e.g. to solve various scalability issues).
- Improve measuring and testing infrastructure - to be more accurate, reliable, catching regressions sooner, etc.
- Find a working model (process) to maintain the performance in the future. Avoid falling back to a critical situation which is expensive to solve. Make performance work adopted by all teams (dev, QE) in all phases of their work.
Main Performance Problems
In order to choose the right things to work on, we need to better understand the current state - know what the main performance problems are. There is the wiki page for collecting main performance problems from the user perspective. We want to know what people think is most important (visible), what they mostly complain about. This page collects raw data, anybody can add to it a serious performance problem they know about. We've got data from direct feedback, mailing lists, talking to people, surveys, performance tests, filed bugs, etc.
Below is a summary of the identified problems.
There is clearly not just one big thing to fix and be done:
- Many problems relate to scalability, we can see scalability issues in all areas (startup, responsiveness, memory). Many problems are worse in the "big" IDE (full distribution) than in smaller configurations - e.g. startup, memory consumption, switching between editor tabs or navigation in the explorer (repetitive actions).
- The startup experience is especially bad for a cold start with large projects opened in the IDE.
- Excessive disk I/O seems to be a problem especially on Windows and for laptops.
- There are significant problems when some parts (IDE installation, JDK, user dir, projects) are on a network (slow filesystem).
Critical areas, main problems
- Cold startup, with projects opened, with scanning - the user waits too long... even longer with the big IDE. A waived issue from 6.0 (IZ 110011).
- expanding folders
- browsing - repetitive actions (e.g. pressed down arrow)
- enable incomplete "Go To Type" results throughout initial scanning/time-stamp check
- editor - code completion, typing, navigation (incl. switching tabs)
- JSP editor - reported as slow, code completion, editing
- mixed reports about Java editor - need analysis
- XML - code completion slow, problems with large schema files (editing, refactoring)
- building projects - e.g. time to see a small change applied (rebuilding/redeploying an application)
- indications of some occasional problems in debugger
- overall consumption - e.g. IZ 110015
- OOME reported often, IDE gets unusable after some time
- identify scenarios where to check memory conditions (either where known problems appear or where a proper memory handling is important) - then analyze the occupied memory, look for leaks, suggest include in tests, etc
- main issues mostly affected by scalability: TBD
Java EE & Web
- web projects
Visual Web Pack
- Generally slow on Windows.
- Hard-earned performance gains often get lost in time (regressions)...
Based on prioritization and analysis of the problems, we put together a list of suggested improvements/fixes. We choose fixes that have high user impact and are doable for 6.1 (i.e. in a relatively short time). The time left is really short for big changes from the ground. We try to combine the fixes in such a way they bring together a noticeable difference. In general, we prefer to fix less number of related issues than trying to fix a higher number of small issues even if they were easier to fix. Complex fixing with broad coverage is for a longer term.
Suggested Improvements - this is a summary of planned improvements/tasks with priorities (P1 - defining, P2 target, P3 nice to have). For more details about the final results see the 6.1 performance FAQs. See also the implementation plan for details about individual tasks.
Note: In all cases even if don't fix the actual problem (slowness) we should always make sure that at least the UI responsiveness is right. I.e. there must be immediate feedback of performing an action, no UI hangs (repainting problems), progress needs to be visualized properly (wait cursor, progress bar), very long operations cancelable, etc. The UI responsiveness rules are well known, violations are treated as routine bugs (except for special cases).
To get an overall picture of what the improvements look like for the users, read the Performance Press Release, for concrete results and more details see the 6.1 Performance FAQs. (If you wonder why we started 6.1 work with a press release, read about Working Backwards.)
Together with the performance fixes and improvements we are also going to extend the testing infrastructure.
- We need to be able to measure improvements properly and avoid future regressions. That implies there ideally should be a new test for every performance fix.
- We'd like to extend the types of tests (e.g. add memory leak tests), use more accurate measuring for reliable results, and do comparisons (for scalability - e.g. same test in different IDE sizes, different editors; comparing with competition).
- For developers it is also important to be able to quickly and precisely localize regressions and then be able to reproduce the tests - to find where a problem is and to verify it is fixed.
For more details see the testing requirements page. Concrete plans for 6.1 are yet TBD.
- See the NB 6.1 performance plan.
- Main performance problems - collected raw input
- Planned Improvements - list of goals & improvements planned for 6.1
- NB 6.1 implementation plan
- Press Release and 6.1 Performance FAQs (results)