EditorIndexing68

Indexing improvements plans for 6.8 release

This page contains planning and other info related to the scanning and indexing improvements that should happen in the 6.8 time frame.

Introduction

What is done during scanning/indexing

  • Initial scan
  • Dependencies among roots are collected
  • Roots are topologically sorted
  • Binary roots are indexed
  • Source roots are indexed
  • General steps of each source root indexing
  • Collecting all files to index
  • Grouping collected files by their mime-types
  • Finding and running all indexers registered with these mime-types
  • Steps specific to JavaCustomIndexer
  • Files are compiled into the signature files that are stored in cache necessary for the Java infrastructure to speed up compilation of files opened in editor (javac compiles against the signature files instead of sources)
  • Compiled sources are scanned for usages of types/methods/fields and the information is stored in Lucene index
  • Information about all errors and warnings (requested later by Task List) is stored
  • If there are changes to class/method/field signatures (MD5 checksums are stored for each file), a set of dependent files that need to re-indexed is collected (currently, only files within the same source root are collected, however there are requests to re-index all dependent files across source roots (e.g. Issue 158218))
  • Notifications about added/removed/modified types are fired

When do we scan/index

What user/system generated events trigger (re-)scanning

  • In 6.5
  • file created, modified, deleted
  • class-path root added/removed
  • document is modified (does not trigger immediately, but document is marked dirty and can trigger the scanning even when quering index)
  • last focused editor component changed and the deactivated document was modified
  • changes to a class/method/field signature "sometimes" trigger rescan on dependent files within the same source root
  • source level change
  • external compiler options change
  • turning-on task list since dependent files are not reindexed while task list is switched off
  • In 6.7
  • file created, modified, deleted
  • class-path root added/removed
  • document is modified (does not trigger immediately, but document is marked dirty and can trigger the scanning even when quering index)
  • editor focus is lost and the document is modified
  • changes to a class/method/field signature trigger rescan on dependent files within the same source root.
  • basically every switch between files opened in editor (see Issue 167866 for more details)
  • source level change
  • In 6.8 (trunk)
  • file created, modified, deleted
  • class-path root added/removed
  • document is modified (does not trigger immediately, but document is marked dirty and can trigger the scanning even when quering index)
  • editor focus is gained - rescan is called on the previously focused editor component if it is modified
  • changes to a class/method/field signature trigger rescan on dependent files within the same source root.
  • source level change

Source roots that have on a jar on any translated classpath (after applying SourceForBinaryQuery) are rebuilt from scratch when the jar changes.

What we scan (how is the set of things to be re-scanned determined)

What the user can do when the scanning is happenning on the background (and what not)

Since indexing runs as a special task using the new ParsingAPI and infrastructure, it is mutually excluded with all other tasks that access parser results (trees, types, etc.) via the Parsing API. These tasks include: code completion, find usages and refactoring, editor hints, folds, semantic highlighting, navigator etc. The only feature that is not completely blocked by scanning now is Go To Type/Symbol.

Links

EditorPlan68, NB68PerfPlan, PostNB67PerfTopics, ScanOnDemand, FitnessViaSamples

Internal (for now)

Performance dashboard, Testing projects

Planned for 6.8, but not done

  • (M2) Issue 133943 Make the IDE usable during scanning (Scanning and Data Usage) (Vita 10)
  • (M2) Issue 159756 Allow modules to bundle preindexed data (Vita 3)

Ideas

  • Issue 161564 Rebuilding across source roots: cannot be done eagerly by default, but maybe we could compute what will be needed, and repair each source root on next update of said source root?
  • Issue 168043 Indexing should be interruptible

Done

  • Out of date indexes IZ 168237
  • Issue 152749 Scanning of sources and compiling should be removed (Scan On Demand) (Vita 5 (evaluation: done: will be on UC))
  • Refresh action: Issue 150687
  • Issue 166340 mime type resolution should be faster during scanning (Vita)
  • Issue 42683 Should avoid rebuilding shared subprojects: faster building of complex (enterprise) projects - with multiple dependencies on a common project. The current compilation with ant exposes the problem with bigger projects - does not scale. (Jesse)
  • Speeding up indexing jobs:
  • Issue 166800 Dusan changed order of files passed to JavaIndexer which significantly improves the performance on large projects.
  • We have fixed a memory leak in the JavaIndexer => less javac => performence improvent.
  • Preventing unnecessary rescans:
  • Issue 167866 Unnecessary reindexing while switching files opened in editor. (Dusan)
  • Reindexing is no more triggered by simple editor focus lost. Now, it is triggered by changes of the last focused editor component only, (Dusan)
  • Making large rescans less intrusive
  • Tomas have changed the locking of lucene, now the lock is held only when the lucene is modified => index sensitive actions have access to index during the time of scan.
  • (M1) Create sample projects together with performance team to (Vita with P. Flaska)
  • measure performance
  • define supported project setups
  • More automated tests to measure scanning performance - more situations, different projects
  • A lot of 'error badges' issues fixed. In Tools-Options there are several new options for detecting errors in java files - Disabled, Enabled within a source root (default, == 6.7), Enabled within a project, Enabled (across projects).
  • (M2) Bundle the latest version of Lucene (2.4.1) (Tomas 1)
  • (M2) Issue 134805 File system events coalescing (dependency on core (FS) change)
  • (M2) Issue 167098 Suspend scanning for VCS updates when many changes are coming in for long time period and repeated rescans would just slow everything down. (Vita 5)
  • Checksums over jar files content could be used to eliminate the need to always rebuild all dependening source roots (Tomas).
Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo