---

See the whole UI Spec

Status

• Integrated into trunk builds. To be part of 6.7 M2.
• P2: Slight redesign of installer interaction
• P3: Static Analysis for debugger attach types
• P3: ProjectOpenedHook delegation
• Shall project metadata list required/suggested modules (like in IZ 155030)? Right now we there is XPath list of definitions which needs to be kept in sync. Creating standard API for general solution looks premature, as the requirements are in flux.
• P3: Static Analysis for project types
• Bug fixing of reported issues, plus more bugs

Goal

The goals is to decrease by 20% the overall consumption of system resources[1] for users downloading full edition and using[2] only limited[3] functionality.

The following table shows the latest (we measure periodically) measured startup times of the "Features On Demand IDE" compared to versions without any ergonomic adjustments:

Without any project	Platform	Full IDE	FoD IDE	Effectiveness
warm	4494	7806	5140	80 %
cold	10071	18185	14349	47 %
With J2SE project	Java SE	Full IDE	FoD IDE	Effectiveness
warm	6866	8770	7686	57 %
cold	17629	20017	18707	55 %
Web JSP Examples	Java All	Full IDE	FoD IDE	Effectiveness
warm	8399	9155	8681	63 %
cold	19388	20358	19849	52 %
PHP AirAlliance sample	PHP IDE	Full IDE	FoD IDE	Effectiveness
warm	5299	8479	6711	56 %
cold	14901	19446	17143	51 %
Ruby Depot sample	Ruby IDE	Full IDE	FoD IDE	Effectiveness
warm	5777	8683	6562	73 %
cold	15408	19637	17105	60 %
C++ Freeway Simulator sample	C++ IDE	Full IDE	FoD IDE	Effectiveness
warm	5632	8339	6417	71 %
cold	15446	19067	16806	62 %

Each time represents an average of 3 measurements. Cold is measured after computer reboot with existing userdir.
Effectiveness is 100-((FoD-bundleIDE)/(Full-bundleIDE))*100%

[#1] Permanent memory, consumed heap, CPU cycles during and after start, etc.

[#2] Edit, compile, run, debug, and so on.

[#3] Only J2SE, or only PHP, or only Ruby.

Schedule

• Nov/27 2008 - 1 QE will test the build
• Dec/4 2008 - Technical Council: API Inception Review
• Dec/1-5 - pre-integration testing happen cross the IDE
• Dec/20 - merge date

Before Merge Tasks

• P1 entry points implemented (as shown at FitnessForeverUISpec)
• (Dec 17): Fix all P1 and P2 from the list of Reported Issues.
• Preintegration testing on Dec/4-5 and Dec/16-18

Name	Area of responsibility	Go/No Go	Issues
Rashid Urusov	Profiler	2 new issues , Go from profiler	155559,155659
Andrey Komarov	AU, JH, utilities	Go for areas. They don't have any build specific issues. View/open action works properly with external pdf viewer.	Found suspicious behavior on activating Java App functionality: 155701
Ken Frank	I18N	GO -brief sanity in other locales, of create all project types both from create new project and activate from plugin mgr so its not asked for that project type; saw one possible problem related to server plugins and in touch with Marian about it
Jindra Sedek/Lukas Jungmann	J2EE, Groovy, Ruby
Alexander Pepin	CND	Go. Stoppers have been fixed. No new issues found
Andrei Chistiakov	Mobility	Go. No new issues found
Mikhail Matveev	PHP
Filip Zamboj	J2SE Debugger, Form	Debugger: GO - sanity test; Form: Go - very basic test (create sample project, run. create new)	none of issues above were solved. 154573 occurs but you can continue in work
Jara Uhrik	Maven	Go	155731, 155734
Petr Dvorak	Java, J2SE Projects	GO - No new issues found, "Editing Files" plugin issue was fixed, problem with refreshing Options dialog seems not to be specific for ergonomic build...

Motivation

The NetBeans IDE grows with new functionality every release and soon it will be like a swiss knife. It will do everything everyone can ever need. On the other hand most of the time we are using knifes and even swiss knifes to cut - the same way most of us use the NetBeans IDE. We use it for Java, PHP or C/C++ most of the time and if we need some other functionality, it is good to have it around and available. It just shall not make the knife too big and heavy.

Since the introduction of NetBeans Big IDE - the only IDE you ever need - we are facing various problems resulting from the IDE size. With every release we are growing and slowing down while occupying more and more memory and CPU. We have a dedicated effort to prevent these inevitable regressions, and already fixed many issues, however if the roots are rotten, nobody can plant a beautiful tree on top of them. The root of problems related to NetBeans IDE is that it is too big for single person use. In long term, the all-in-one IDE will become unsustainable.

This page provides evidence of this claim and offers solution that will return the NetBeans IDE its fitness and will allow the IDE to keep its fitness forever.

Issues of the Big IDE

Let's see some typical voice of customer examples first:

• "IDE is getting big and heavy. Some improvements on performance could be."
• "The only drawback I see with NetBeans is the amount of memory being consumed. If I could minimize it i dare to do it."
• "I would like to make the base IDE lighter. It takes too much time on my system to load. I would prefer to load the base IDE 1st and then other plugins etc after editor."
• ”MyEclipse feels snappy... NB feels constipated... ”

All the troubles we focus on here are actually issues caused by unused features. It becomes apparent that just having the features in the IDE hurts, even if never used by the user. These features may affect overall performance, expose scalability issues, introduce bugs, make UI more complicated, etc. The problem show up in various ways. Here are some typical examples.

Startup

The full IDE starts 30% longer than the basic Java SE distribution. This is a start of an empty IDE with no project opened, i.e. none of the additional features is used. Would not be a problem if the total time was insignificant, but that is not the case. E.g. in Eclipse the difference is unnoticeable.

More to that, startup with a Java SE project opened takes longer than the empty startup, but the difference is bigger in the full IDE (with the same project). Before a partial fix was made, the additional time was even 3s bigger in full IDE (measurements of 6.5 dev on 1st Sep 2008).

Number of loaded classes

Additional features generate classes that must be loaded, obviously. But it's bad that many classes are loaded even if the feature itself is not used at all. During 6.5 development we've been watching classes loaded in startup, there were about 10-20 new classes every week. Many of them were fixable regressions, many could not be fixed due to infrastructure issues (e.g. in action system, windowing system - like IZ 147352), various registrations causing eager loading (e.g. project lookup - like IZ 150194, debugger types, queries - like IZ 148177), duplicities in code (e.g. in GSF - like IZ 148235, 148181, 148179, 148079), etc.

The impact is not only related to startup. As shown in more details below, if a Java SE project is opened in full IDE, 1000 more classes are loaded compared to when opened in Java SE IDE, even though the same features are used in both cases.

Bugs

The big IDE is naturally more error prone. It's expected that more features mean more bugs, but not if they manifest themselves even if the particular feature is not used at all. The usual pattern is that there are various providers/hooks/listeners registered on various places which may misbehave. The typical problems are memory leaks, but also strange slowdowns of some operations (package expansion, refactoring, etc). We've been facing various issues during last year that only showed up in the big IDE while using just a basic functionality for which the smaller Java SE IDE would be enough.

Some examples:

• Long pauses when using "Go To Type..." IZ 145851
• 38s wasted while expanding 51 files IZ 146618
• Find Usages takes extremely long to complete IZ 143851
• Memory leak via UML reverse engineer action IZ 124051
• Big .uml directory created on first startup IZ 125597

Despite that we find most the bugs and fix them eventually, there are always some present, cannot be completely avoided. People could be saved from all these problems if they used the right size of the IDE needed for their work.

The More Code the More Bugs!

An example to demonstrate the problems arising due to the all-in-one approach is to measure how much code gets executed when doing similar tasks in small and big IDEs. Let's measure the regular edit/compile/debug cycle on JavaSE project.

Configuration	Activated Classes	%	Total Classes	Comment
Base	0	0%	2913	Start and exit JavaSE edition
JavaSE	2963	100%	5915	Edit/compile/debug in JavaSE edition
Full	3862	130%	6780	The same edit/compile/debug in full IDE

The report excludes classes that are out of our control (java, org.apache, com.sun, sun, etc.), which clearly shows the amount of code that we maintain ourselves and need to repair ourselves. As can be seen, regardless of the scenario being the same, there is 30% more code executed in the Big IDE. With a bit of oversimplification this means what we can eliminate 1/4 of our bugs if we convince majority of JavaSE users to use the JavaSE edition and not the Big IDE one.

Navigation

Responsiveness of common navigation actions like selection changes and also popup menu invocation is significantly affected in the full IDE. Typical examples are switching between tabs in editor, moving selection in Explorer (via keyboard) or showing context menu on a project. The window/node selection changes perform worse due to more listeners registered in the full IDE (these are usually various actions). It can be partially mitigated (like we did for editor tabs), but fixes of this kind are fragile. Fixing all the listeners one by one is laborious and again prone to regressions. And this will only get worse with more technologies and features added to the IDE.

UI clutter

More features is not only more memory, CPU, bugs, but also more UI. Is that bad? From certain point, yes. More menu items, windows, configuration options, file types in new wizard, etc definitely makes the UI is more complex and harder to use for people who actually want to use only a limited set of features.

Summary

Despite our effort, in long term we seem not able to fix the negative impact of the "unused features" entirely. It is both due to infrastructure limitations as well as bugs in the features implementation. Even if every single issue is fixable, the amount of work to "fix everything" is enormous. Also new features comes faster than we are able to incorporate them properly. But there are not only technical issues, the problem of UI clutter, too many features in the UI is not solvable without hiding/removing some features or creating separate distribution.

An important point here is that majority of our users download and install the full IDE distribution, so facing the described issues. They are not aware of the penalty of the all-in-one IDE on performance and quality. They just think NetBeans is a big slow hog...

Solution

The basic idea is as follows: Lets start the IDE in "empty" mode letting our users to choose what features shall be enabled for them individually. This is similar approach to the one taken by Visual Studio. Start slowly, showing only general overview and most important entry points to each functionality. As the users require more, observe and adjust. At the end create a perfectly tailored environment for each individual developer. More on the workflow in the UI spec.

The user would still download and install "full IDE", but with most of modules (clusters) disabled. The entry points are selected parts of UI indicating that more features can be available than those just visible in the IDE. A typical example is the New Project wizard - it would list all projects possible to create and enable the necessary modules once such a project is requested. Similarly for open project wizard; and there are some more entry points to be specified. So for a typical user the IDE would be light and clean, no UI clutter etc, but allowing to easily enable more functionality when needed. Note the approach is significantly different from the original feature on demand proposal as it exposes only several entry points and does not try to mimic complete UI to make the IDE look like all features are there.

The technical side of this proposal does not require any new ground breaking technologies. We have in place everything that is necessary. All those little tunes are there. We have Plugin Managers and its API, we have pluggable UI elements, etc. We just need to look at all these pieces from a completely new point of view and orchestrate them into a symphony not yet revealed and seen as a whole. In the name of effectiveness and quality, let's undertake this journey. We'll get IDE ready to grow and be fit forever.

Technical Details

There is one module ide.ergonomics placed in separate cluster ergonomics1. In addition to regular files, this cluster also contains config/Module/xyz-module.xml files to disable all the modules from other clusters (except platform, ide and gsf cluster). This cluster is about to be included in Full IDE. Also the same approach of of auto enabling on a subset of features will be applied in the Java IDE distribution (which is still pretty big). All other distributions are not going to contain the ergonomics1 cluster at all, as such they will work as expected.

How do I enable 'Full IDE'?

The answer depends on what point of view you have. In case you are QE tester, you might want to comment out cluster ergonomicsX from etc/netbeans.cluster file. In case you are writing simpletest functional tests, it is enough to use NbModuleSuite.allModules(...) which enabled all the functionality of Full IDE. In case you are user missing your favorite functionality, you can always open Tools/Plugins manager and enable (or disable) functionality that you wish.

Why this change is safe?

The common worry seems to be whether this change is not too complicated? Whether it is safe? Is it not better to stick with our old good bloated IDE and try to fix it? These are valid questions and they deserve their answers now:

Nothing New

The most obvious reason why the whole system shall work is: We are doing nothing new. All the enabling/disabling of modules/kits/clusters has been present for ages and we are just adding different UI to trigger it. As such, all the bugs that will be found, are already part of the codebase and can be simulated without our project. Indeed, our projects makes them more visible, but that does not mean we introduce new bugs. The bugs have been there and they deserve to be fixed.

Reuse Existing Distributions

Another reason why this is as safe change as possible is that it reuses our previous investments in quality, bug fixing, etc. The set of functionalities that user can enable matches closely the distribution downloads:

• Java SE
• Java Web and EE
• Java ME
• Ruby
• C/C++
• PHP
• SOA
• Groovy (+)
• API Support (+)

These distributions are properly tested and known to work. This is good starting point. Users can use Tools/Plugins to extend these distributions and this is exactly what we want to do. Again, by reusing tested functionality that has been available for years, we are lowering risks associated with ergonomics IDE project.

The system always enables clusters as a whole - e.g. if the user is using some UI present in one cluster, then all regular modules in clusters that this cluster depends on will be enabled as well. This follows the cluster boundaries and mimics what happens after download of appropriate editions.

Mitigation Plan

The ergonomics functionality is distributed in its own cluster ergonomics1 and is present only when necessary. The current plan includes the cluster in full IDE and all java only. There no dependencies from other clusters to the ergonomics functionality. As such the switch from bloated IDE as we know it towards the ergonomic one is just a matter of including/excluding this cluster from the build. This means we can safely turn the functionality on (as soon as QE confirms there are no P1, P2 bugs), let our users report bugs, and fix them. In the unlikely case we found an unfixable blocker, we can safely remove the cluster before 6.7 release. The work done will not be lost, we will still keep bugfixes made meanwhile (just like IZ 153797), thus get better IDE. Of course, this is just a mitigation plan. The team is confident that this feature can work in 6.7 timeframe.

Static Analysis and Dynamic Verification

The basic trick that ergonomic IDE project is using, is to statically analyse the functionality provided by the real modules and mock it up without the modules. As soon as the real functionality is needed, the system triggers the right modules and delegates the execution to them.

With a trivial implementation this might lead to infamous two version of the truth problem. Having the same definitions in two places which can get async would indeed be undesirable and needs to be avoided. The current system uses two approaches to eliminate the problem:

• Analyse - where ever possible the JAR files of existing modules are scanned for their definitions in their layer files, this information extracted and morphed into purposes of ergonomics infrastructure
• Verify - commit validation contains a test that compares the data seen by the real IDE and the ergonomics IDE and verifies that they match as expected.

Using these methodologies allows us to minimize work needed by individual module owners (they can still stick to their own habits and care only about their own module) in most frequent cases and in the rest yield a warning in case the views ergonomics and bloated view of the world does not match. Moreover the expectation is that due to introduction of @annotations we will be able to statically analyse more and more cases:

Entry Point	Static Analysis	Verification
New Project wizard	Extracted from Layers	All templates attributes need to be correct
New File dialog	Extracted from Layers	All templates attributes need to be correct
Attach Debugger	Extracted from annotation registration	Check the # and names of attach types match
Open Project dialog	Extracted from @AntBasedProjectRegistration, plus manually described in ide.ergonomics/*.properties files	Check the list of registered ProjectFactories matches. Check the list of registered AntBasedProjectTypes is the same
VCS Checkout	same as open project dialog	same as open project dialog
Attach Profiler...	Extracted Actions from Layer	verify actions in Menu/Profile folder
Import/Eclipse	Extracted Actions from Layer	verify actions in Menu/File/Import folder
Open File dialog	Extracted from Layers	verify that mime types for each extension are the same

Need Restart

Currently there are some modules that need restart. See needs.restart.txt. From what can be seen (look at commit messages) introducing these changes, the original purpose of needs restart flag - e.g. to restart when classpath of the IDE itself changes has been significantly broaden in favor of workaround bugs that should have been fixed anyway. As IZ 153797 shows there are parts of the IDE which are not written properly and the need restart flag is just workarounding these problems. We are ready to help to identify such poorly design areas and help improving them. We believe that this is completely aligned with one of the major themes of NetBeans 6.7 - improving quality of the IDE.

The Future

The Ergonomics IDE seems to be the only way for us to support two diverging promises. If we want to have NetBeans as the only IDE you ever need and at the same time we want our users to see it not as the bloated IDE you never want. We need a system that really embrace the user, in similar or more advanced way as Ergonomics project does.

One of the more advanced way is a suggestion to always start with minimum functionality and always enable just what is necessary. E.g. make the module enablements done by ergonomics transient. This is unlikely a topic for 6.7 release, but we can take on it in future, if we find way to:

1. do not write modulesystem state to userdir if enabled by ide.ergonomics watchdog module
2. turning on needs to be faster or semi-persistent (e.g. support for all opened projects enabled before main window is shown)
3. we need more triggers, e.g. menu items.

Attachments