GsfToParsingAndIndexingApiMigration

Migrating GSF-based language plugins to Parsing & Indexing API

Contents


After the major overhaul of the java editor in Netbeans 6.0 (project Retouche) a couple of other frameworks for writing language support plugins sprang up to life - Schliemann and GSF. Both of those used different approach for providing similar services which in the end lead to inefficiencies and code duplication. This was not an ideal state and during Netbeans 6.5 we decided to consolidate and unify those frameworks.

In order to reduce code duplication and improve maintainability we created the Parsing & Indexing API, which took the best from all three frameworks in terms of parser support and indexing so that it could become a new building block that these three frameworks could use. The next step was to rewrite the three frameworks to use that building block.

Retouche, Schliemann and now even GSF has already been rewritten and this document is meant to serve as a reference guide for migrating GSF-based language plugins to use the new Parsing and Indexing API. Although the document attempts to explain various aspects of the new API as you will need them for rewriting your modules, it should not be considered an API documentation.

Overview

Since there is quite a lot of GSF-based language plugins and it would be hard to migrate them all at once we decided to fork the GSF framework and create its new variant that is based on Parsing & Indexing API. This new variant is called CSL and lives in the csl.api module. This means that both GSF and CSL frameworks as well as GSF and CSL based language plugins can live together in the same IDE session.

The main goal of creating CSL was to rewrite it to use the new Parsing and Indexing API. Nothing more, nothing less. Particularily we didn't strive to improve the framework itself or its API.

The original GSF framework is going to be deprecated and its use discouraged in favor of CSL. Despite of that there still have been work going on in GSF module in Netbeans 7.0 and in order to not loose this work we will retrofit any changes done in GSF between when we forked off CSL and Netbeans 7.0 release. After Netbeans 7.0 is released we will no longer guarantee that any fixes or changes in GSF will be propagated to CSL.

What does CSL stand for?

It's a shame, but nobody seems to really know. It most likely originated from 'Common Scripting Language support', which was an alternative name for GSF (Generic Source Framework) used back in Netbeans 6.0. We think the name is not good, the word 'Scripting' is definitely not correct and it's probably hard to remember. Unfortunately we don't have a better name. If you can find some, please let us know.

Development Tools

The Parsing and Indexing API has been part of Netbeans codebase (trunk) since releasing Netbeans 6.5. That said, all the standard tools/channels to support Netbeans community development are available for this API as well. Specifically, there is:

Notes on Migration

Parser and tasks

  • The XML layer registration of CSL-based plugins is the same except for the main folder where the registrations are stored. The folder is now called CslPlugins, please see the example below.
  <folder name="CslPlugins">
    <folder name="text">
      <folder name="javascript">
        <file name="language.instance">
          <attr name="instanceClass" stringvalue="org.netbeans.modules.javascript.editing.JsLanguage"/>
        </file>
      </folder>
    </folder>
  </folder>
  • Pre-6.9 versions only: The special ant task org.netbeans.modules.gsf.GsfJar used for enhancing language-plugin's XML layer before it's compressed to a jar file is now called org.netbeans.modules.csl.CslJar and works exactly the same as before. However, we may abandon this concept in favor of @ServiceProvider-like annotations. Here is the taskdef element for your build script.
  <taskdef name="csljar" classname="org.netbeans.modules.csl.CslJar" classpath="${nb_all}/csl.api/anttask/build/cslanttask.jar:${nb_all}/nbbuild/nbantext.jar"/>
  • 6.9 version and later: Since the introduction of java annotations and adding annotation processors support to Netbeans 6.9 we have been able to drop the ugly CslJar ant task and replace it with @LanguageRegistration annotation. You should add this annotation to your language class (eg. the one that subclasses org.netbeans.modules.csl.spi.DefaultLanguageConfig). The annotation itself is very simple, it has only two parameters - mimetype of your language and useCustomEditorKit to support languages with custom layer registrations. Typically, you should end up with something like the example below. Also, if your build script uses GsfJar or CslJar ant task simply remove this build step. The @LanguageRegistration and its annotation processor provided by the CSL infrastructure will enhance your module's XML layer the same way as the ant task used to do.
@LanguageRegistration(mimeType="text/x-your-mimetype")
public class YourLanguage extends DefaultLanguageConfig {
  ...
}
  • Majority of CSL API classes are almost the same as GSF API classes and often the only difference is the package where the class is stored.
  • All tasks in Parsing & Indexing API are classes (not interfaces) and therefore some services in CSL were changed to classes as well (eg. OccurencesFinder, SemanticAnalyzer).
  • ParserResult vs. CompilationInfo - a typical service implementation in a language plugin obtained CompilationInfo, which was produced by the plugin's parser and provided access to AST, etc. The same role now plays org.netbeans.modules.csl.spi.ParserResult. In general ParserResult is a result of parsing a snapshot of a document or its embedded parts. You can get hold of the parsed text by calling PR.getSnapshot().getText(). You can also access the Document or the FileObject by calling PR.getSnapshot().getSource().getDocument() or PR.getSnapshot().getSource().getFileObject() respectively.
  • ParserResult.invalidate() - make sure that you don't take ParserResult or anything parser related outside of "The Loop" (ie. outside of a task's run method). The infrastructure invalidates ParserResults after a task's run method finishes and accessing data on an invalid ParserResult will cause an exception. GSF has far more relaxed policies for its ParserResults. Also, it is a good idea to implement invalidate in your ParserResult subclass and release all unneeded resources.
  • No parser phases are supported on a generic level. The reason is that this is too language specific a concept to make up a useful abstraction for it. Parsers for different languages are likely to have different phases (many of them having only one phase at all). The phases must be understood by the caller and as such there are no generally useful phases that would be applicable to all languages. CSL itself does not need to use parser phases at all. In general, if your parser supports phases make yourself an API to give your clients control over them. If your parser is used by a phases-unaware client simply parse as much as you can (slow, but correct). All features (tasks) written particularly for your language can (and should) communicate the required phase to the parser. If needed you can use task priorities to run less demanding tasks first.
  • There is no PositionManager anymore. It's been replaced by ElementHandle.getElementOffset(ParserResult). Please implement and use this method instead.
  • How to get hold of the caret offset? In a SchedulerTask that's registered with CURSOR_SENSITIVE_TASK_SCHEDULER you simply read it from CursorMovedSchedulerEvent. You should not need it or should already have it in UserTasks, afterall it's you who starts that task. You should not need it anywhere else either, however, some parsers try to sanitize parsed source in order to make an external parser produce at least some AST. They use cursor position as an approximation of the last edit and try to sanitize source in this area. If you do this you might find useful GsfUtilities.getLastKnownCaretOffset(Snapshot, EventObject).
  • Conversely CslTestBase.enforceCaretOffset(Source, int) can be used in tests to set an offset for a given document. All code using GsfUtilities.getLastKnownCaretOffset() will get that offset.
  • There is a org.netbeans.modules.csl.spi.support.ModificationResult class which might be useful for refactoring plugins. However, the class does not really belong to CSL and may end up somewhere else.

Embedded languages

  • EmbeddingProvider vs. EmbeddingModel and TranslatedSource - The support for embedded languages is slightly different in Parsing API than what it used to be in GSF. There is no TranslatedSource. Its role is played by a Snapshot and there are two types of snapshots - a snapshot of the whole document (ie. top-level snapshot) and a snapshot of some embedded sections (ie. embedded snapshot). There is virtually no difference between them and the infrastructure treats them equally. The embedded snapshots are created by EmbeddingProviders, which are ordinary tasks run by the infrastructure whenever it is needed and when it is appropriate. These tasks are supplied to the system by a TaskFactory registered in MimeLookup for the mime type of documents, where the TaskFactory's EmbeddingProvider can recognize embedded sections. The EmbeddingProvider can provide as many Embeddings (even of different mime types) as it likes.
  • Offsets translation - Like TranslatedSource in GSF, the Snapshot in Parsing API provides methods for translating offsets from a snapshot's coordinate space to the document's coordinate space and vice versa - getOriginalOffset(int) and getEmbeddedOffset(int). For a top-level snapshot these two methods are an identity (ie. getEO(x) == x and getOO(x) == x), but for snapshots of embedded sections these methods try to provide coordinates translation from the snapshot to the original document and vice versa. Due to the nature of snapshots (ie. depending on what snapshots your EmbeddingProvider creates) this translation may not always be possible.

Indexers

  • The GSF's Indexer is replaced by org.netbeans.modules.parsing.spi.indexing.EmbeddingIndexer, which is supplied from EmbeddingIndexerFactory. The functionality of Indexer.isIndexable() can be moved to EmbeddingIndexerFactory.createIndexer(Indexable, Snapshot). Otherwise the indexers work the same.
  • Indexers are integrated to the parsing infrastructure, which means that they operate on Snapshots and Parser.Results. This ensures that an indexer that can index documents of a given mime type can be used transparently for both top-level and embedded snapshots. Therefore indexers does not have to care for embedding in any special way. This may for example render some tests useless. Although I humbly admit that using words test and useless in the same sentence may sound like a heresy.
  • SearchScope - There is no SearchScope class anymore. The scope of a search in an index is determined by the roots that you search through (ie. the roots that you use in QuerySupport.forRoots()). There are two new methods - GsfUtilities.getRoots(FileObject, ...) and GsfUtilities.getRoots(Project, ...) - which might be useful when determining roots for an index search. The search roots typically depend on the project you are in, the classpath IDs your language files belong to and the actual registered classpath instances.
  • NameKind - There is no NameKind class anymore. It's been removed in favor of QuerySupport.Kind from Parsing & Indexing API.
  • Due to the removal of NameKind you are likely to have to update golden files for your code completion tests. Typically you will have to do a simple search and replace operation with the following texts:
  -(QueryType=COMPLETION, NameKind=PREFIX)
  +(QueryType=COMPLETION, prefixSearch=true, caseSensitive=true)

Annotations

Do not use annotations from org.netbeans.modules.csl.api.annotations in csl.api. These will be removed soon. Add api.annotations.common and use annotations in package org.netbeans.api.annotations.common.

Old GSF Annotation New Common Annotation
@org.netbeans.modules.gsf.api.annotations.NonNull @org.netbeans.api.annotations.common.NonNull
@org.netbeans.modules.gsf.api.annotations.CheckForNull - method @org.netbeans.api.annotations.common.CheckForNull
@org.netbeans.modules.gsf.api.annotations.CheckForNull - other occurences @org.netbeans.api.annotations.common.NullAllowed
@org.netbeans.modules.gsf.api.annotations.Nullable @org.netbeans.api.annotations.common.NullAllowed


Already migrated plugins

As a proof of concept we migrated all javascript modules (ie. javascript.editing, javascript.hints and javascript.refactoring) and they are now based on CSL and Parsing & Indexing API. There are several other modules whose migration has already started and they are at least partially migrated; for example HTML editing modules and Groovy editing modules.


Known Issues

  • No reliable way how to determine that a Snapshot is a top-level snapshot (ie. snapshot of the whole document vs. snapshot of embedded parts). CSL needs to run some tasks on top-level snapshots only. Also, some refactorings can only be performed on top-level snapshots.
  • No support for incremental parsing so far.
  • How to update the status of a file (ie. the little top-right corner square)?
  • No way how to determine if there are EmbeddingProviders registered for a certain mime-type and if they recognize an embedding of a given mime-type; needed in refactoring; eg. in order to skip files that can't contain embeddings that we might otherwise have to refactor.

Limitations

  • No interoperability between GSF and CSL based modules.
Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo