This topic summarizes prototyping efforts started in 2007.

This is not a plan document.

See EmbeddedBrowserPlan68 for the most recent plans.

Introduction

With the advent of Web 2.0 it is obvious that a first grade support for HTML/CSS/Javascript application development is required in NetBeans IDE. One of the critical components (among others) for such a support is a first grade component that supports the current standards in HTML/CSS/Javascript. Such a component needs to be most likely available as a Java component so that it will integrate well with the Netbeans Platform and IDE. It is conceivable to develop such a component from scratch, however it will be a massive undertaking. It may require a state of the art expertise in many areas such as HTML, CSS, Javascript and windowing systems and so on... on all the platforms and OSes on which NetBeans Platform and IDE runs. As it turns out there are several browser component technologies available. For example:

• XULRunner - the technology/component at the heart of leading browsers such as Firefox and Mozilla (opensource)
• Webkit - the technology/component at the heart of leading browsers such as Safari (opensource)
• IE browser component - the technology/component at the heart of leading browser Microsoft's IE (obvious)
• ICEBrowser (commercial)
• ...

Some of these are open source technologies. Out of these the XULRunner is the most promising technology as it is supported on all the platforms and OSes on which Netbeans Platform and IDE runs. Being at the heart of the very popular Firefox Browser it supports the latest HTML, CSS and Javascript standards. It is likely to do so in the future. Thus a safe horse to bet on.

Embedding XULRunner

It turns out that XULRunner component technology is designed to be embedded in other applications. The XULRuner Embedding API is available with multiple language bindings through the technology called XPCOM. The Java based binding to XPCOM is also available and is called JavaXPCOM. In fact, there are several projects based on XPCOM, JavaXPCOM or an abstraction layer above these for embedding XULRunner is a Java application. For example:

• Webclient
• JRex
• JDIC
• WebRenderer
• MozSwing (datasheet)
• ...

The first three are open source projects. The last one is a commercial product and thus is out of question or at least least preferred. That leaves us with the first three. There is an issue with these components. They are all based on embedding XULRUnner in a so called heavyweight component such as java.awt.Canvas. This is so because the embedding of XULRunner is based on the native window handle. And heavyweight components do have a native window handle. There is a problem in terms of embedding in the Netbeans Platform and IDE, because Netbeans is based on Java Swing components which are so called lightweight components. There are well known issues with respect to mixing heavyweight and lightweight components as described here. Therefore we have started investigating other approaches to embed XULRunner in a lightweight Swing component.

Using the native window handle of the top level java.awt.Window

(AFAIK) The approach taken by the projects such as Webclient, JRex and JDIC has been to embed XULRunner inside a java.awt.Canvas. This is has been so for the want of a native window handle - which is a requirement for embedding XULRunner. However this causes problems for embedding those inside Swing components based applications. It turns out that every swing applications is ultimatley hosted in or subclasses of java.awt.Window or java.applet.Applet both of which are heavyweigh components and thus have an associated native window handle. The first approach is based on exploiting that fact. In other words making use of the native window handle of the top level heavyweight container of a Swing application to embed the XULRunner but somehow restricting the rendering of the embedded XULRunner to ther sub area that corresponds to some lightweight Swing component. It remains to be seen if such an approach could work. However if it works we think that is the most preferred approach as this approach has several advantages.

Actually it turns out this approach may not work as is. This is because the embedded XULRunner simply uses the top level window's native window handle as the parent of a native window that it creates. This native window is heavy weight. See the "Transparent window based approach" below.

I am not sure if it will work but the following needs to be tried out - by hacking the native code of XULRunner is it possible to tell it to not create it's own window but instead make it use the real estate of the native window handle of the top level window of embedding component.

Off screen window based approach

This approach is based on embedding XULRunner inside an offscreen heavyweight component and copying the rendered output to the onscreen lightweight Swing component. This of course involves forwarding of coordinate translated input events from the onscreen lightweight component to the offscreen heavyweight component in which the XULRunner is embedded. This is a less preferred approach for obvious reasons however has a higher likelyhood of working because in some sense it is based on an approach that is used by remote desktop trechnologies such as VNC, Remote desktop, NetMeeting application sharing or WebEx.

Transparent window based approach

This approach is basically a hybrid of the first approach and the approach taken by embedded IE project. In this approach the embedded browser window is made a child window of the top level window's native window. The embedded browser window is positioned at the same location as the embedding lightweight container where it is embedded. The size and location of the embedded browser window is updated as the size and location of the embedding lightweight container changes. The native painting of the embedded browser window is turned off and/or it's window (and it's children recursively) is(are) made transparent. When the embedding lightweight container's paint(Graphics g) method is called the embedded browser window is asked to paint itself in a BuffredImage and that BuffredImage is drawn in embedding lightweight container's graphics. With this approach no input handling is required as the embedded browser is actually there at the right location.

This approach depends on the ability to:

• make the native painting of the embedded browser window turned off and/or it's window (and it's children recursively) is(are) made transparent.
• make the embedded browser window paint in a BuffredImage e.g. PrintWindow Win32 API

With this approach there is still a concern with respect to input event handling for any other lightweight components (such as drop down menus) that paint on top of embedding lightweight container. That is because even though the transparent heavyweight window will make it possible to show the other light weight components (such as drop down menus) that paint on top of embedding lightweight container it will block the input events from being received.

For the above reason we may have to try out yet another approach.

Behind window approach (for lack of better name)

In this approach the XULRunner is embedded inside a non-modal JDialog with no decorations that is positioned exactly at the same location/size as the embedding lightweight container. The JDialog is always hidden behind the top level window of the embedding lightweight container. This ofcourse means that input event forwarding will have to be performed by the embedding lightweight container. So in effect this is similar to Off screen window based approach above. The only advantage is that because the embedded XULRunner is exactly at the same location/size as the embedding lightweight container a simple bitblit of it's window will work. We will not have to worry about the possibility of not being able to take XULRunner screenshot because it is offscreen. Moreover the mouse coordinates will also match. Thus in essence we are keeping the control of rendering as well as input event handling on the Java side of embedding.

I am proposing JDialog for embedding XULRunner because:

• JDialogs do not show in the Taskbar on Windows.
• We can set the undecorated property. (Well this is not specific to JDialog).
• Keyboard input works on JDialog (unlike java.awt.Window).

In fact I have already implemented such a component - which has a JDialog hosted shadow component that can track the location and size of itself at a specified (x,y) offset. Download the attached DopplegangerPanel NetBeans project and try it. HOORAY! I got some basic input event forwarding working.

NOTE: The form submission does/may not work in JEditorPane if it is invoked using javascript. Therefore Google Search will not work. Only hyperlinks work.

We have been able to get the embedded XULRunner working - only the rendering part - and only on windows. Next step is to implement input event forwarding for XULRunner. This will have to be done at native level.

The window on the top is showing the embedding lightweight panel inside a NetBeans RCP which is using the lightweight rendering as evident from the overlapping Dropdown Menu. The window at the bottom, without any window decorations is the actual Webclient based embedded XULRunner in a model less decorationless JDialog. In the real implementation this window will not be visible to the user. Instead it will be always behind the embedding application window. It is being shown here for debugging purposes only.

It has been pointed out or observed that this approach may run into several issues:

• How to capture the Popup windows such as the drop down of a <select>? That is because such dropdown are separate native heavyweight window. Thus they are not captured by the API taking the screenshot of XULRunner. Moreover such dropdowns may go outside the bounds of XULRunner and may even show from under the edges of embedding RCP. To that I propose the following solution. Simply - use them directly. That is let us use such pop up (drop donw) windows directly by making them show over the embedding application window. Thus the embedding application window will be sadwiched between the XULRunner window and the pop up (drop down) window. This may require some native windowing level tricks to change the zorder of these overlapped windows to achieve the desired result.
• How to handle the cursor changes? It is unclear if it is possible to monitor cursor changes to the native window of XULRunner and apply an analogous (as Java level cursors may not match the native level cursors) cursor change to the embedding lightweight panel.

Behind (when inactive) and Front (when active) window approach (for lack of better name)

This is yet another approach we are trying. In fact the following zip implements this approach:

embeddedbrowserrcpsuite.zip

In this approach the XULRunner is embedded in an undecorated non-modal dialog (just like the previous approach). This dialog is behind the main application window at exactly the same location and of same size of the embedding lightweight container. So far this is same as the last approach. The only difference is that as soon as the user clicks in the embedding lightweight container the dialog is brought to the front and the mouse event is forwarded to it. A similar thing happens when the embedding lightweight container gains the focus due to tabbing. (that is the reason the input events also work in this approach). Then, the XULRunner continues to handle the input events until such time that the user clicks elsewhere e.g. user clicks on a help menu. Then the XULRunner dialog is pushed back again. Now the lightweight drop down menu works correctly as the heavyweight XULRunner dialog is out of the way again. There is a little bit of flicker when the dialog moves to the front and back. Another drawback with this approach is that as the heavy weight XULRunner is in front when the user is interacting with it - the transparent component based editing layer may not work. However if the transparent editing layer is going to handle all the input events then we could leave the XULRunner dialog in the back. Some of the issues of cursor image management do apply to this approach also. Some of these issues could be solved by detecting the cursor position and querying the DOM to see if the cursor is over a hyperlink and change the cursor accordingly to Hand cursor. Another issue will be how to make the key board accelerators for menu and other actions (e.g. switching multiview tabs) as well as menu item mnemonic work when the dialog is in the front. Yet another issue will be how the bringing to front interacts with other non-modal children of main window.

Here is a screen shot of the running NetBeans RCP. It also shows the DOM in a tree view. It must be noted that the DOM was obtained using direct Java API (in this case Webclient) and can be programmatically manipulated.

Visualization of XULRunner with Firebug extension embeded inside NetBeans

Visualization of tight integration of XULRunner with inside NetBeans IDE

(Extremely) Simplified Architecture

visualweb.next.embeddedxulrunnnerplusfirefox.pdf

XULRunner Javascript debugger research notes

Here is the list of Phobos Debugger APIs:

• AttachedDebugEvent.java
• ComplexDebugObject.java
• DebugEvent.java
• DebugEventVisitor.java
• DebugExceptionInfo.java
• DebugFrame.java
• DebuggerContext.java
• Debugger.java
• DebuggerListener.java
• DebugObject.java
• DebugScript.java
• DetachedDebugEvent.java
• ExceptionThrownDebugEvent.java
• FrameEnteredDebugEvent.java
• FrameExitedDebugEvent.java
• LineChangedDebugEvent.java
• SimpleDebugObject.java

Phobos Debugger API

Here is the list of XULRUnner Debugger APIs:

• jsdICallHook
• jsdIContext
• jsdIContextEnumerator
• jsdIDebuggerService
• jsdIEphemeral
• jsdIErrorHook
• jsdIExecutionHook
• jsdIFilter
• jsdIFilterEnumerator
• jsdINestCallback
• jsdIObject
• jsdIProperty
• jsdIScript
• jsdIScriptEnumerator
• jsdIScriptHook
• jsdIStackFrame
• jsdIValue

XULRunner Javascript Debugger API

Even from the names you can tell some of them are analogous.

Webclient needs to expose the second set as Java API through Webclient's BrowserControl.

Then basically we have to adapt (wrap) those set of classes into the first set if we want to make use of the work done by Phobos team for plugging into NetBeans debugger infrastructure. That implementation is here:

Phobos Javascript Debugger integration into NetBeans IDE

Ed Burns is working on accessing the Firebug debugger interface using Java XPCOM. Once he has that in place he will wrap it in Phobos debugger API so that it will be easily integrated into NetBeans debugger UI.

Status

Ed Burns is trying out Sandip's HWND idea on webclient. You can read about it at http://developer.mozilla.org/en/docs/Webclient_Roadmap

Embedded Browser Discussion

EmbeddedBrowserMeeting10172007

EmbeddedBrowserMeeting10182007

During the discussions in this meeting with AWT team members Oleg and Artem, we learned that JDK7 in fact has a multiplatform (Windows, Linux and Solaris) solution for mixing heavyweight and lightweight components. We tried it and it works. I have built the RCP zip for you to try. Just download the following RCP zip, download the JDK7 (build b22 and above) and try the RCP using the --jdkhome command line option. This is the real deal - there is no separate window hosting the XULRunner. The XULRunner is actually embedded in NetBeans RCP. According to Oleg it should be possible to port it back to JDK 6 also.

Reference

ATF/Mozilla

XULRunner Reference

Venkman Javascript Debgugger Guide

Contacts

• Sandip Chitale
• Joelle Lam
• Devananda Jayaraman

MozSwing Requirements

• [#1] There is a problem on Unix (Linux and Solaris) to correctly synchronize access of X window resources by the AWT/Swing thread and Mozilla thread. There are more approaches how to do it. Ideally by correct locking mechanism in Swing, which got introduced some issues from JDK 6 and higher.
• [#2] There is a 'standard' level which can be delivered without changes, and might be sufficient. If the one would be not sufficient, there can be enhanced the Mozilla by patch, which would allow comprehensive support. The patch is needed for the Mozilla binaries, or it would be needed to push the changes into the Mozilla project directly (preferrable).
• [#3] requires to implement window reparting and api to keep the mozilla area alive after removeNotify (bug #18705)
• [#4] On Linux, a signal handler registered by mozilla user profile code randomly receives signal #11 (SIGSEGV) and aborts the program. If unregistring the signal handler, everything works. Should be checked with devs from JDK team why the signal is raised from JVM and what are consequences of unregistering the handler. (bug #18255)

Use Cases for MozSwing requirement

• [#1] Ability to specify base URL: In order to support Page (HTML, JSF, PHP etc) design, the page (.html, .php) is parsed and a DOM is generated (called Source DOM). This DOM may contain tags that can not be displayed by the Embedded Browser. This DOM is rendered (the non displayable node is converted in to suitable displayable HTML element) in to another DOM (called rendered DOM). The rendered DOM is not created from URL but from data as it is transiently manipulated . When the Source DOM is rendered in to this Rendered DOM, we don't see a way to set the "base URL" (RenderedDOM.setDocumentUri() throws exception), so that relative URLs in that DOM can be resolved.
• [#2] API to parse URL: This is to create the Source DOM as explained above. Source DOM will not be associated with any window to display. Mainly we want to use Mozilla built-in parser, because it can support HTML source which JAXP parser can not handle.
• [#3] Abilito set "UserData" to node: In order to make a cross reference between Source DOM and Rendered DOM, the Source node may be added to the rendered DOM as "UserData". Now I'm getting "Unsupported Operation" Exception.
• [#4] Ability to access JavaXPCOM API: While MozSwing provides good support for UI related artifacts, we require much more than that. That is to manipulate CSS, Javascript and more as supported by JavaXPCOM. So we may need the ability to access JavaXPCOM API directly. Our entry point is MozSwing. So it should provide a window to reach these API.

Attachments

DopplegangerPanel.zip		13165 bytes
Embedded XULRunner.png		64783 bytes
EmbeddedBrowserAPI.zip		22660250 bytes
EmbeddedBrowserSuite.zip		50858194 bytes
Firebug.png		119806 bytes
Integrated.png		155873 bytes
MozSwing datasheet.pdf		508908 bytes
NetBeansEmbeddedXULRunnerWithFirebug.png		126081 bytes
embeddedbrowserrcpsuite.zip		12047686 bytes
embeddedbrowserrcpsuiteJDK7.zip		16827456 bytes
embeddedbrowsersuite.zip		13021860 bytes
embeddedxulrunner.jpg		83272 bytes
embeddedxulrunnerfrontback.jpg		65093 bytes
visualweb.next.embeddedxulrunnnerplusfirefox.pdf		124364 bytes