How to make scanning/indexing faster?

See also: ScanOnDemand

Faster Scanning Via a VPN Connection

Scanning projects can be very slow over a VPN connection on Windows 7. One solution is to disable the Remote Differential Compression.

Use fast filesystem/disk

Scanning files means a lot of disk activity and the faster your disk or filesystem is the better. The performance killer for scanning are network drives or disks mounted over a network using protocols such as nfs, samba, ssh, etc. The particular protocol does not really matter these mounted drives are just slow and if at all possible please refrain from using them.

It's vitally important to store not only your source files on a fast filesystem, but also the userdir used by your IDE. The userdir is the place where the indexed data is stored and there is a lot of disk activity in these cache folders as well while scanning.

Even if on fast local drive, also check if your sources or the userdir are not subject of intrusive virus scanning or drive indexing.

Use different Garbage Collector strategy

We've had several reports from users that choosing 'Concurrent Mark And Sweep' garbage collector improves scanning performance. This may or may not make a difference on your system. The problem is that people use different hardware, different versions of JDK and they have different default GC algorithm chosen by their JVM.

If you want to see what GC your IDE uses please look in the IDE's messages.log file and look for Garbage collector: XYZ messages. They should be near to the beginning of the file and there should be two of them. The second one is the more important one.

You can tell the JVM running your IDE what GC to use by adding special flags to <nb-inst>/etc/netbeans.conf file. These flags for Concurrent Mark And Sweep GC are listed below and you should add them to netbeans_default_options= string in the netbeans.conf file.

-J-XX:+UseConcMarkSweepGC -J-XX:+CMSClassUnloadingEnabled -J-XX:+CMSPermGenSweepingEnabled

Configure your freeform projects correctly

There are several good FAQ entries about freeform projects here. Please read them through and make sure that you understand the stuff described there. Specifically, 'Can I have multiple source roots in a freeform project?' is very important and you should definitely read it.

As far as the scanning is concerned freeform projects are just like any other projects. In fact scanning does not understand projects at all. All that scanning does is looking at the classpath roots registered in the system. These roots (at least in case of java) are of two types - source roots and binary roots. The scanning is primarily interested in source roots, because that's where your source files are and that's what you edit. Scanning has keep up with your modification, otherwise things like code completion, go to type, etc would not show stuff that you've just typed.

The tricky part is that dependencies between projects are expressed in the form of dependencies on project outputs. For example if project-A produces A.jar and project-B depends on project-A, it in fact depends on the A.jar file. That is project-B has A.jar on its compilation classpath. This is not suitable for scanning, because scanning needs source files not binary jars. Luckily, correctly configured projects provide a way how their outputs can be translated to sources that were used for producing these outputs - ie. project-A provides SourceForBinaryQuery which allows to map A.jar to the sources folder in project-A.

Now, as pointed out before this all works for correctly configured projects and that is where you come in, because configuring freeform projects is solely up to you.

Configuring Binaries to Sources translation

This is done on 'Java Sources' and 'Output' tabs in the freeform project properties dialog. The Java Sources tab is pretty straightforward and if you've read 'Can I have multiple source roots in a freeform project?' you should know how to configure this.

The Output tab is a little bit more complicated. The problem here is that you have to have your project built otherwise its outputs (eg. jars) don't exist. If your project is built then simply assign project jars or folders with *.class files to their respective source folders.

Configuring project dependencies

This is the usual source of problems. It is absolutely critical not to use source folders here. As explained before projects depend on outputs of other projects and so on this tab you are expected to enter *.jar files or *.class folders produced by the project that the project you are configuring depends on.

Again the problem usually is that the projects you are working with are not built and so you don't see their outputs on the disk and has nothing to enter. And although you might be tempted to enter the main folder of the project itself or the sources folder these are not going to work. Moreover, the IDE has only a very little chance to correct your error and will basically let you enter anything. So, be careful you are on your own and any mistake here is likely to have severe consequences.

Java compilation errors detection

This is only loosely related to scanning and applies only to java sources. The IDE indicates compilation errors in your java sources by adding special red badges to source file nodes, package and project nodes. This works even for files that are not opened in the editor. And since your java classes usually use each other making modifications in one class may break another class or on the contrary modifying a class may fix errors in other classes.

In order for that to work the IDE has to find classes that depend on the one you modified and recompile them. You can imagine that if a class is used by many (eg. hundreds or even thousands) other classes this process can take a while.

There is a setting in Tools-Options -> Editor -> Hints -> Java called Dependency Scanning which you can use to limit the scope where the IDE looks for the dependent classes or even to turn this feature off completely. The available options are:

  • Source Root - search only in a source root where the modified class lies
  • Project only - search in all source roots located in a project where the modified class lies
  • All Projects - search in all source roots in all opened projects (default value)

If you don't need error badges or are bothered too much with 'Project scanning' after saving java source files you may try to limit the scope for this feature.

(Please note that in versions prior and including 6.8beta this setting used to be located in Tools-Options -> Editor -> Tasklist -> Java and was called Dependencies in Java Tasklist.)

File Descriptor Limit On UNIX

By default the UNIX OS has a limit 1024 concurrently opened file descriptors per single process. Because of this limit the NetBeans IDE limits number of opened cache segments to 400. Opening the cache segment may slow down the Go To Type, Go To Symbol and Go To File features.

If you experience this kind of problem you can increase the number of concurrently opened cache files by adding the following option -J-DIndexCache.size=1000 (or any number greater than 400) to netbeans.conf or command line. It's also suggested to increase the per process file limit on your OS otherwise the IDE may throw IOException regarding missing file handles.

Changing file descriptor limit on Linux (requires root login)

To increase the limit (to 4096 concurrently opened files) for all users edit the file /etc/security/limits.conf and add the following lines

*        soft        nofile        4096 
*        hard        nofile       4096

To increase the limit for just for given user (me) edit the file /etc/security/limits.conf and change the following lines

me        soft        nofile        4096 
me        hard        nofile       4096

Changing file descriptor limit on Solaris (requires root login)

The description is available on http://docs.sun.com/app/docs/doc/820-4289/abeja?l=en&a=view&q=File+Descriptor

This problem does not affect Windows and Mac OS X.

Detecting external changes in files

This again is only loosely related to scanning. Netbeans use their own abstraction for accessing files called Netbeans Filesystems. There is not many differences between Netbeans Filesystems and simple java.io.File when accessing files on a local disk. The only big difference are change notifications (events). In other words Netbeans Filesystems allow listeners to be attached to files, folders or even the whole folder hierarchies that are then notified about changes/modifications done to the files in those folders or folder hierarchies.

The long standing problem with Netbeans Filesystems listeners has been detection of changes that are done externally (i.e. outside of the IDE) to the files that are being watched. In early versions of Netbeans the Filesystems periodically synchronized their state with the disk. As of approx. Nb 4.0 this was changed to only check for external changes when the IDE's main window gains focus and to only check files that have their FileObject instance in memory. This was better, but highly inaccurate when watching for changes in folder hierarchies.

In 6.9 we improved the detection mechanism algorithm to detect changes in all files/folders scanned by the IDE reliably. You may have already seen the extra progress bar that says 'Checking for external changes' when you come back to the IDE from another app. This is when Netbeans Filesystems are getting in sync with your disk. If there are changes detected the Filesystems will fire appropriate events, which may result in scanning sources.

For some projects this all is not interesting, because they don't use external tools and their sources are only changed from within the IDE. For these projects 'Checking for external changes' can most likely be turned off without any adverse effect. There is a setting in Tools-Options -> Miscellaneous -> Enable auto-scanning of sources which can be used for controlling this feature.

How to make Go To Type (Symbol) faster?

In the NetBeans 6.9 there is an experimantal option which improves the speed of the Go To Type and Go To Symbol. When the option is enabled the IDE consumes more memory which is used to cache disk indexes. To enable the memory cache you have to add the following option to the netbeans_default_options= string in the <nb-inst>/etc/netbeans.conf file.


Applies to: All versions, but scanning/indexing behavior and effect of these hints may vary between versions
Platforms: All
See also: Other performance FAQs, Freeform projects, ScanOnDemand, FaqScanningAndIndexingIssues

Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo