CND68RemoteFastSyncDesign
Remote Fast Synchronization Design
The idea
The core idea is that files should be copied to remote host by demand.
For this purpose, some layer should present on remote host that interferes somehow with file system on remote host and makes copying occur on the right moment. We will call this layer XVFS (though it can be, for example, as simple as just a LD_PRELOAD library that intercepts some file system related calls).
Here is a draft picture
http://wiki.netbeans.org/attach/CND68RemoteFastSyncDesign/D30_3102_CND68RemoteFastSyncDesign.jpg
Main components are:
- XVFS. A bridge between file system and other our components. XVFS interferes with remote host file system; with the help of the Remote Controller requires data from local host. Redirects read/write requests as such to some cache directory (which might be the same directory remote builds runs in).
- Remote Controller. Processes XVFS requests, redirects it to Local Controller via socket (which is forwarder via JSch connection), redirects responses from Local Controller to XVFS, performing necessary conversions. Most likely, physically it is a part of the same native component as XVFS, but logically it's different. For example, if we decide to change XVFS, controller will not be changed.
- Local Controller. Processes Remote Controller requests, initiates file transfers. The idea behind controllers is to a separate massive data transfer operations and control, so that control is more reliable and responsive.
- File Sender and File Receiver. Send and receive files over JSch connection. They may be (in 1-st prototype) as simple as the existent copying files via JSch. But most likely to get it fast we have to change this somehow, for copying files one by one via calling remote command for each is very inefficient.
XVFS
FUSE
Pros: it's a standard for Linux. It is included in Linux kernel by default starting from 2.6.16. It has WebDAV support (might be of interest later).
Cons: is not installed on Solaris (there is a port, but it's status is rather vague). It also needs some setup actions on remote host: user should be added to "fuse" group.
The availability of the FUSE can be checked automatically.
AVFS
API is very similar to FUSE one. Can work over FUSE or via LD_PRELOAD.
Cons: for Solaris, supposed to work via LD_PRELOAD; but this does not actually work (and even does not compile). It was tested on old (2.7 or 2.8) Solarises only. I fixed compilation, but haven't yet got it working.
On Linux, can work via FUSE. LD_PRELOAD does not work on Linux due to impossibility of intercepting stat call (and probably some other calls) in the recent Linux glibc (glibc > 2.1).
GVFS
Not a very good candidate for they are going to replace it.
Own LD_PRELOAD module
Pros: it is the most simple way (in the case it works) - we have to intercept only few calls, and just delegate all other calls, in contrary to AVFS / FUSE, where we have to implement more than 20 functions (including read, write - and all these should be efficient!)
Cons: Though it's the most simple way for Solaris, unfortunately, on recent Linuxes intercepting stat call via LD_RELOAD is impossible. So the only way to get it work on Linux is to copy all files initially with zero length. This will probably work... but who knows what kind of issues will we get in this case (what if some tool that is involved in build needs correct file length?)
Transferring data
In the 1-st prototype we'll use the existent way of copying files one by one. However, to not make build too slow we'll have to change this. The ways of speeding this up are
- Don't copy each file via a separate command that is invoked on the remote host. Use a single receiver program that will get file by file and extract to the remote host file system,
- Probably using some predictive algorythms - copying some files in background
- ???
