OutOfDateIndexes

Reported as Issue 168237

VOCs

Has it ever happened to you that code completion, or Go To Type dialog did not list something that you knew they should? Was it a result of some external changes for example using your version control systems from command line or generating/modifying some files by an external tool? Did you have to restart the IDE to fix the problem? If so, please share your experience with us by editing this page. You can include links to your blog, issues in IssueZilla, email conversations, etc. We would like to know about your scenario in order to decide how many of these problems can be fixed by improving external changes detection (i.e. how many of the reported problems would be fixed by a simple restart)?

These are the situations that we are currently aware of:

Added Type

If you work on a project (opened in the IDE) which is using another project as library (not opened) and you modify the sources of the library from command line (eg. by using your VCS client) and add a new file (eg. *.java file with new classes defined in it). When invoking "Go To Type", the classes from the new file are likely to be missing.

Broken Source

Similar two project setup as above. There is one file from the opened project open in the editor. You update your sources by using a command line VCS client (eg. svn update). There was a new method added to one of the classes in the library project and the change is pulled from the VCS. This library class is referenced from the class opened in the editor and the changes pulled from VCS add code to the edited class that calls the new method in the library class. The editor content is properly refreshed, but marked as erroneous.

To reproduce the problem:

  1. get indices-out-of-date_OutOfDateIndexes.zip, unzip it into empty directory and invoke hg update 0.
  2. start NetBeans and open the app project (it depends on library project, but leave that one closed)
  3. open app.Main in editor, wait for indexing to finish and execute
  4. choose Team/Update and update to revision 1
  5. The source in editor is marked as invalid
  6. never mind and execute - everything works
  7. the source is however still considered invalid, see the picture below
  8. restart fixes the problem

Image:indices-out-of-date_OutOfDateIndexes.png

Missing PHP classes

While working with PHP project (Projects view not expanded yet) you open some file in the editor. Then use an external tool to generate additional PHP files. Return back to the IDE and try "Go To File" or "Go To Type". The newly generated files are likely not to be present.

Using external tools which generate/change source files

See Issue 150687:

There are a few important case, when user needs to invoke reindexing sources. It's usually when the project is in a CVS and the sources are updated externally. Next example is that a part of sources are regenerated by an external tool, like UML or some tools like Propel or Hibernate.

The problem is that if there is not created FileObject for the source, which is changed externally then is not reindexed after the change and all features like cc, Go To .... work with the old version. User has to restart the IDE to get it work correctly in such case.

This is not problem only in PHP but for every project in NetBeans.

General Problem

The NetBeans IDE tries to automagically observe external changes. However not all of them are currently noticed. When a change is not noticed, it results in inconsistency in the internal indexes. The fix is to force the "up-to-date" check and rescan the changes - the simplest way how to do this now is to restart the IDE.

Fix It!

There are two general directions to improve this situation:

  1. improve the automagical detection - this can lead to degradation of performance on slow file systems (nfs, smbfs, sshfs), especially after main window is activated (when the check is running)
  2. provide users means of force the up-to-date check - this may be overused, hard to understand, and it somehow indicates we have resigned on getting the automgical solution right

The actual fix is likely to be some combination of these two approaches.

Tentative UI Specification

After discussions with our UI guru, Ondřej Langr, it seems that the preferred solution lays in tuning the automagical detection performance and reducing its intrusiveness, as well as, giving users an intuitive way of asking for the up-to-date check and rescan in situations when this actually could help.

Automagical Refresh

This is still going to be done after the main window activation. If it takes longer than 10s (not likely on a local disk, possible with network filesystems), it will show a progress bar and let the user cancel the task. There should be a way how to disable the automagic detection completely (eg. if the refresh is canceled by the user three times in a row (without ever fully finishing) the system will disable it until the end of the NetBeans IDE session.

The refresh will be made less intrusive - e.g. it will run with "delays" - so the system is not fully occupied by background I/O and has some resources to process user requests. This makes the check slightly longer, but less intrusive.

Refresh "button"

Various UI places in the IDE will be enhanced with a manual trigger to force the refresh. These places could be:

  • Go To Type dialog - if no types found, there will a be result called "Are you sure this item exists? Shall I search more deeply?"
  • Go To File dialog - if no files found, there will be a similar message
  • Code compiletion result - tbd.
  • Broken source in editor - if there is a broken source in the editor due to missing symbols (classes or methods), the system offers a hint to perform the up-to-date check of projects that the edited source file depends on. This may be too annoying, more discussion needed.

There are many questions and uncertainty regarding this feature and it's likely to be discussed further and refined. Some of the many questions are:

  • What exactly does the "refresh" mean? What is the IDE giong to do when user chooses to refresh - eg. just up-to-date check and rescan of changes (faster, but may not fix some problems) or just full rescan of all indexes (slow, but accurate)? What will be the exact wording? Won't the users feel betrayed if the up-to-date check doesn't fix the problem?
  • Incorrect error badge - likely the most visible indication of a problem is the presence of incorrect error badges in the Projects tab. What is our strategy? What will be the UI gesture for refreshing in this case?

Try It!

There is a branch indices-168237 with our changes and Hudson builder which contains the automagical refresh enhancements. Please download appropriate latest artifact and try it by the end of Aug 2009 (the sooner the better) and let us know how intrusive it is (especially on non-local disks). Typical scenario to test for:

I started the IDE with -J-Dorg.netbeans.modules.masterfs.REFRESH.level=0 so I would be able to attach the IDE log with useful information to the issue.

My test case was SQE, a midsize module suite (213
*.java in 54 modules), using a binary platform with a source association (release67). I set "Dependencies in Java
Tasklist" to "Enabled". What works:

1. Edit a project file in Emacs, switch to NB, project marked modified.

2. Edit release67/openide.windows/.../TopComponent.java to be non-public, switch to NB, several modules marked with
error badges. Restore TC.java, switch to NB again, badges disappear.

3. Edit SQEManager.getDefault to be non-public, switch. Calling modules marked w/ errors. Fix, switch, errors go away.

Upon switching back to the IDE, if there are in fact external changes, you get a few seconds of high CPU activity as
projects are rescanned. If there are no external changes, there is a blip of high CPU - maybe 1sec - visible in the
process monitor, but not enough to turn the laptop fan on. So for this size project, the fully accurate mode is quite
reasonable.

Approximate memory usage after startup and initial scan:
no flag -> 100/123; after GC -> 43/148
kind=all -> 86/121; after GC -> 47/151


Thank you all for your help!

Measurements

The table below shows data that we measured for various projects and on various filesystems with the patched IDE. The data are the number of FileObjects tracked by the IDE, memory footprint, the amount of time needed to check all FileObjects for external changes when simply switching from and back to the IDE without actually modifying any files.

The hardware configuration that these numbers where measured on was:

  • Sun Ultra 24 Workstation
  • Intel Core 2 Duo CPU E4400 @ 2.00GHz
  • 4GB RAM.
  • Windows XP Professional Version 2002 Service Pack 3 (Win XP), Ubuntu 9.04 "Jaunty Jackalope" (Ubuntu)
  • Java 1.6.0 Update 11
Project NTFS (Win XP) ext3 (Ubuntu) samba on Ubuntu (Win XP) NFS (Ubuntu)
LimeWire (J2SE)
(1 source root, 2532 files)
trunk 757 FOs 235-375ms
patch 785 FOs 313-484ms
trunk 754 FOs 423ms
patch 769 FOs 552ms
trunk 750 FOs 4359-5735ms
patch 762 FOs 9500-11331ms
trunk 739 FOs 331-370ms
patch 754 FOs 725-772 ms
web.core.syntax (apisupport)
(176 source roots, 9982 files)
trunk 399-443 FOs 85ms
patch 2488-2518FOs 950ms
trunk 454 FOs 64ms
patch 2371 FOs 740ms
trunk 419 FOs 2650ms
patch 2389 FOs 37500ms
trunk 438-481 FOs 374-890ms
patch 2388-2416 FOs
6889-12872ms
Nigel's project (freeform)
(1 source root, 18281 files)
trunk 912 FOs 16-63ms
patch 10592 FOs 3135-4001ms
trunk 921 FOs 126-267ms
patch 10593 FOs 2077-3067ms
trunk 857 FOs 3563-3719ms
patch 1434 FOs 26906-27906ms
trunk 1467 FOs 775-1399 ms
patch 6374 FOs 5405-7098ms
ACE + TAO (C\C++)
(2 source roots, 51947 + 11 files)
... trunk 14121 FOs 1858-2236ms ... trunk 28211 FOs 7504-15408ms


  1. Contains values of 5 measurements, either the average or min-max values.
  2. Samba on Ubuntu running on different subnet.
  3. NFS - Sun's internal home mounted.

TODO: remeasure NFS on the same machine as Samba to compare these services. (expecting NFS still faster)

Report: Scanning goes on for ages

I tried the php test version 090820. I noticed that after I close a project it is still scanned for external changes. Even half an hour after I closed it the project still gets scanned with every external changes check.

Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo