(Difference between revisions)
Line 105: Line 105:
* NativeProcessBuilder -> ProcessBuilder
* NativeProcessBuilder -> ProcessBuilder
** <strike>NativeProcessBuilder available as ProcessBuilder</strike>
** <strike>NativeProcessBuilder available as ProcessBuilder</strike>
** setShell
** setShell - not now
** environment -> ProcessBuilder.getEnvironment()
** environment -> ProcessBuilder.getEnvironment()
** process listeners
** process listeners
** '''properties''' - very weak API, we have to do something with that
** '''properties''' -> capabilities
* Connection
* Connection
** '''TBD'''
** Separate module
==Javadoc and code==
==Javadoc and code==
The code is developed on execution-merge branch ([http://hg.netbeans.org/cnd-main/rev/bfaa410ff432 initial commit]) in cnd-main repository. The builder is [http://deadlock.netbeans.org/hudson/job/prototypes-execution-merge/ on deadlock].
The code is developed on execution-merge branch ([http://hg.netbeans.org/cnd-main/rev/bfaa410ff432 initial commit]) in cnd-main repository. The builder is [http://deadlock.netbeans.org/hudson/job/prototypes-execution-merge/ on deadlock].

Revision as of 15:06, 12 November 2012

Execution API merge


Problem statement

  • There is a need of common public API for executing external processes - either on local host or remote host.
  • There are several flavors of related APIs:
    • dlight.nativeexecution - non public (friend) API used by C/C++ modules and many modules in Studio ( 119 friends in total); All the code that uses this API is under our control;
    • extexecution - PUBLIC API for executing remote processes. It is plug-able and allows to pass own ProcessBuilder. As API is public, there are users of the API that are not under our control... (54 modules in nb codebase use it)
    • richexecution (within lib.terminalemulator)

dlight.nativeexecution is intensively used in CND. This module provides functionality for runnig processes on different platforms (*nix, windows, macos), it deals with processes starting/terminating, I/O, different terminals, signals, and so on. So it provides good functionality coverage, but has several big problems:

  • nowadays it requires serious cleanup;
  • it's implementation has many places where call to some method initiates a connection, which is unexpected in most cases
  • it is ssh-oriented. It's fundamental API object, ExecutionEnvironment, doesn't provide a way for specifying a way of accessing remote host
  • it's error-handling/flow is not well structured and not always clear

cleaned-up dlight.nativeexecution can be used as an implementation module with extexecution API, but extexecution should be extended in this case to support features proposed by the dlight.nativeexecution.

Scope ot the review

  • I would like to review an API (cleaned-up) of nativeexecution (not binded anyhow to extexecution) with the main aim to demostrate
    • what functionality this module provides (all this functionality was needed for CND's remote support)
    • how the functionality affected some API design decisions
  • Then I would like to make conclusions on what is missing in the extexecution (mostly this is about ExternalProcessBuilder), how it could be extended in a compatible way to make it possible for users to use full range of functionality provided by nativeexecution.

From here by term 'nativeexecution' I mean a new rewrittent API/implementation which is a result of clean-up of existent dlight.nativeexecution module. The possible way of dealing with modules (as I see it) is to:

  1. Deprecate dlight.nativeexecution
  2. Introduce nativeexecution (API/SPI/impl)
    [phejl] Maybe such API won't be needed afterall. Depends on how much we can integrate to extexecution.
    [akrasny] Right. But I would like to do a cleanup of dlight.nativeexecution as well.. So this 'new' nativeexecution API/SPI/impl could be much more clear. Maybe this is out of scope task and could be done separately.
  3. Extend extexecution and start using (2) as a implementation layer
  4. Move existent CND modules on the new APIs (extexecution/nativeexecution)
  5. Get rid of the 'old' nativeexecution.

Goal of the review

To provide an API (most likely extend extexecution) so that API users will be able to execute commands remotely and have access to all needed (as experience with CND shows) functionality.

All requirements (not ordered)


  • connection initiation/cancellation/termination to a remote environment
  • connection error handling
  • track changes in connection state


  • parallel execution
    [phejl] Is there a problem with that? Why is it mentioned explicitly?
    [akrasny] There were some implementation issues with intensive parallel execution (when many processes are run at the same time) in remote... Also some parts in rfs work sequentially... But right.. this is all implementation stuff... From the API point of view this is obvious requirement for the execution framework.
  • environment variables handling (initial get, ways to modify)
  • start / terminate processes
  • tracking process state changes
  • get process info (at least some - duration/start time etc..)
  • get process exit status (exit status or if terminated by signal - signal number; if core was dumped - need to know this)
  • signals sending
  • OS-neutral
  • helpers for handling special variable names (like LD_PRELOAD/DYLD_INSERT_LIBRARIES)
  • helpers to work with PATH-like variables
  • 'easy' processes start a.k.a ProcessUtils
  • encodings handling
  • pseudo-terminal, terminal I/O
  • internal/external terminals
  • unbuffering
    [phejl] What's this?
    [akrasny] Again, is an implementation detail.. There is a problem of getting process' output in time if buffering is used. Usually output is buffered (especially when I/O is not attached to any terminal). Our 'solution' for this is ld-preloaded library that forcibly sets buffer size to zero. This is rather serious intervention, IMO... that is why we have an API for controlling this behavior. I have mentioned it here because I wanted to discuss this with you.
  • X11 forwarding
  • authentication - password/ssh key/agents
  • errors Handling / flow
  • info caching (?)
    [phejl] What's this?
    [akrasny] Again, implementation thing that I wanted to discuss. There is always a tradeoff - caching vs. calculation. There should be a clear understanding of this question - what information (about remote/local execution environment) is cached and what is calculated each time. Like info about remote host, user's environment variables, etc..
  • shell commands support
  • ShellSession (a helper session for running small tasks (?))
  • trace-ability
  • I/O redirection


  • sensitive information protection
  • caches/temp files handling..

Additional possible usages that are NOT covered by CND/SolStudio use cases, but could be considered as well

  • 'Remote host' is not necessary a host

Here I mean that remote execution could be considered as an execution in 'other context'. Which could be, for exeample, running some command on a remote farm in the context of some ADE-view;

How to do that [phejl]

As this is pretty large task I think we have to come up with some basic plan. The target state should be extexecution module providing API with nativeexecution as implementation module. We will see what's doable.

The merge should be done upside down so we won't copy methods and classes at first place, but rather we will search for missing functionality and we will explore a ways how to add it to extexecution enriching it in a compatible way.

We should progress in steps starting from core functionality. This does not mean such steps should be visible in the VCS trunk though a named branch would be ideal solution for cooperation. The name is execution-merge.

As the new temporary nativeexecution API has no clients, it is hard to understand the usage and workflow (at least for extexecution maintainer). The nativeexecution API usage should be demonstrated at least on one client.

These would be necessary steps:

  1. Provide a core functionality in extexecution:
    • Connection handling
    • NativeProcess if needed
  2. Extend a functionality to handle things like:
    • Enviroment variables
    • Authentication
    • X11 Forwarding
    • ...
  3. Going further to cover all other requirements

Obviously this will need a bunch of iterations and will take some considerable amount of time before being commit ready.


  • ProcessUtils
    • move execution methods to ExecutionService in extexecution
  • NativeProcess -> java.lang.Process
    • ProcessId
    • ProcessCharset
    • ProcessState
    • SignalUtils -> ProcessSignal in extexecution
  • NativeProcessBuilder -> ProcessBuilder
    • NativeProcessBuilder available as ProcessBuilder
    • setShell - not now
    • environment -> ProcessBuilder.getEnvironment()
    • process listeners
    • properties -> capabilities
  • Connection
    • Separate module

Javadoc and code

The code is developed on execution-merge branch (initial commit) in cnd-main repository. The builder is on deadlock.

Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo