EditorSupportChangesProposal

Editor Support Changes Proposal

The documents contains some ideas regarding editing which have come to my mind recently when struggling with 6.0 and during current 6.1 work.

Current State of Parsing Support in Editors

HTML

  • properietary hancoded html parser
  • runs on top of tokens
  • used for code completion, braces matching
  • identifies embedded css and javascript and creates the embedding dynamically
  • Schlieman parser
  • used for folding, navigator
  • provides features for embedded css/javascript

JSP

  • Jasper - a third party parser taken from tomcat
  • doesn't use our tokens - does its own lexical analysis
  • parses particular pages together with their context - the web project with all jars, libraries, tlds etc.
  • doesn't provide a good parse tree for the page itself
  • used for error checking, partially for code completion
  • Properietary hancoded jsp parser
  • runs on top of tokens
  • used for code completion, jsp braces matching
  • Properietary hancoded html parser
  • runs on top of tokens
  • used for html code completion, html braces matching
  • identifies embedded css and javascript and creates the embedding dynamically
  • Schlieman parser
  • used for jsp/html folding, html navigator
  • provides features for embedded css/javascript
  • Javac
  • provides java features in java scriptlets

RHTML

  • The same parsers like in the case of pure HTML
  • Ruby parser by GSF

PHP

  • The same parsers like in the case of pure HTML
  • PHP part??? TBD

Erlang

  • Schlieman parser
  • used for folding, navigator
  • GSF
  • used for classpath and indexing

Scala

  • Parser generated by Rats!
  • GSF
  • used for navigator, highlighting, indentation, formatter ...

GSP

TBD

Others?

TBD

Problems

  • ad hoc parsers integration - no straightforward way how to integrate a parser for new users
  • mostly features separatedly listens on document or token hierarchy changes and directly triggers the parsers
  • parsers often depends on each other but runs in variable order - may lead to duplicated parsing
  • tasks prioritization cannot be simply implemented (eg. cancel all less important parser tasks if user wants code completion now)
  • parsers are started on each modification of the document not just when "their" language is affected
  • no standart way how to define parsers dependency.
  • no standart way how to listen on parser stages for clients/features
  • no standart way how to create and maintain index files
  • lack of generic AST tree??? I am not sure if we really need this. We may be able to implement some simple features on such generic trees but the majority will require the original parser's nodes

jb01 personaly I don't think we need generic AST. We either have parser and consequently AST or we don't have parser, then we can use Schliemann's ASTs

  • lack of generic classpath
  • lack of generic library support

Solution

  • Generic parser SPI/API
  • allows to plug-in a parser implementation and provides unified access to it and to its results
  • provides a virtual source model of an embedded language
  • provides a unified translation between virtual and real positions
  • defines the parsers dependencies in one file
  • restarts only parsers affected by the document change - the translation model handles the nodes shifts of the omited parsers
  •  ??? provides a unified AST tree for each language
  • the results of the parsers/analyzers are not joined in any way e.g. there is no one big ast for the whole document. Even if we had the generic AST for each language it is not technically possible to make one tree from them, the merge result would be a graph
  • Generic "wheel" - the code handling source tasks
  • handles the parsers dependencies, their stages, user tasks prioritization and
   scheduling for opened document 
  • needs to somehow cooperate also with incremental parsers
  • Generic repository updater
  • allows reindexing of the files on classpath when externally modified
  • when reparsing the modified files the same rules for parser deps. like for the wheel needs to be taken into account

In fact we have something like the Generic parser S/API, the wheel and the updater in Tor's GSF but it doesn't solve all the mentioned problems. Schlieman also has its own solutions for most of the problems but it is IMHO too much closed and doesn't allow others to plug in well.

OK, so how do I think the whole stuff should work?

Well ... abandon the proprietary solutions and use the proposed infrastructure and APIs, but ... implemenent them first :-)

  • plug-in the parsers into the parser A/SPI
  • use existing rich third party parsers where possible (rhino, groovy ...) instead of coding our own
  • let features like folding, navigator, semantic highlighs work with the wheel instead of listening on the document or lexer
  • let Schlieman also implement and use the API so it can be more easy integrated with native/handcoded parsers and handcoded features.

What it is all good for if it somehow works now and there is no P1?

  • new users will easily understand the concept and use it
  • better performance (task pririties, parser dependencies)
  • faster implementation of new features
  • nicer code, less bugs, better maintainability ... you know

Notes

Virtual source model - more details

  • simplifies parsing of embedded languages using either our tokens or just char streams by providing a virtual source of an embedded language
  • the model is provided by the superordinate language which knows the context
  • there might be more virtual sources for one language
  • is language - path based
  • can provide either "virtual token sequence" for parsers using our tokens or a reader backed up by the document
  • the virtual source may contain more code than what is just in the file itself. The virtual source should be a "parseable unit" so it is possible to run a parser on the source like it was a standalone source.
  • there is a mechanism for identification of the origin of virtual source parts - "from file/imported/generated/..."
  •  ? the virtual source may also contain included or imported files e.g. linked external js scripts in html.
Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo