FileSystems
Improvements in FileSystems
Contents |
Performance
Performance of FileSystem library is often subject of claims especially if used in network environment. This page describes improvements done as part of performance work on NetBeans 6.1 release.
The poor performance is caused because of: API design, implementation, and usage...
FS API design + performance
Performance of MasterFS impl. is limited by API design. The most important API limits:
- API for the VCS Generic infrastructure (dropped for 6.1)
- only valid instances of FileObject are issued from FileSystem
- for unique file path is issued only one shared valid instance of FileObject
Usage of FS API + performance
Wrong usage from performance point of view:
- not use FS API for writing, modifying and then call refresh (not calling refresh at all is even worse and may cause real bug!)
- uselessly convert File to FileObject to get, read some information provided by java.io.File
- spending a lot of time in in FileChangeListener impl.
- refreshing the whole filesystem if not necessary
Fixing, improving performance
Improving performance of masterfs is goal for 6.1 (part of post-6.0 performance work)
List of assumptions how to achieve better performance:
- drop support for VCS Generic
- eliminate all useless disk touches
- less instances of FileObject in memory => faster refresh
- not instantiate all parents (up to the root) for one instance of FileObject
- not reference children from parents (and vice versa), keep weak - to let garbage collect not necessary instances
- lightweight locks - not to create additional files for locking if not necessary
Steps/state:
- developed on branch masterfs_novcs and merged into trunk |6.1 milestone M1|
- focused on backward compatibility |6.1 milestone M2|
- provided guide how to fix potential problems caused by API changes
Guide API changes for 6.1
List of API changes in MasterFileSystem for 6.1
- Fileobject instances are not guaranteed to be issued after external changes ( see details)
- One drive == one instance of filesystem on windows ( see details)
- Simplify MasterFS, do not delegate on other embedded FS
Justification:
- simplify masterfs #123542
- perf.improvements
FileObject instances are not guaranteed to be issued after external changes
Justification:
- performance improvements; FS doesn't need to test again and again whether File really exists
- only implementation detail (never documeneted)
Potential problems:
- methods in FS API may return null although the file really exist
- methods in FS API should return null but return old cached instance instead although the file was deleted or moved
Affected methods:
- FileObject.getFileObject
- FileObject.getChildren
- FileSystem.findResource
- FileUtil.toFileObject
Known issues:
- #125090 (FIXED) Debugger doesn't stop at breakpoint on Windows
- #125575 (FIXED) Safe delete Restored file is not visible in project view after undo
- #126038 (NEW) java.io.SyncFailedException: C:\Documents and Settings\Krystyna Polomski\My Documents\NetBeansProjects\WebApplication803\web\resources\stylesheet.css
- #125720 (NEW) Custom tasks like JPDAStart, ... may use inconsistent MasterFS
How to fix other related bugs (for developers):
- read how to use properly FS API
- see related thread on nbdev
- find motivation in already fixed issues
Patterns of right usage
- create new file
File f = ...; FileObject fo = FileUtil.createData(f);
- create new folder
File f = ...; FileObject fo = FileUtil.createFolder(f);
- run external command (modifying files layout)
File foldForGen =...; FileObject foFoldForGen = FileUtil.toFileObject(foldForGen); ProcessBuilder pb = new ProcessBuilder(command); Process process = pb.start(); process.waitFor(); foFoldForGen.getFileSystem().refresh(); FileObject fo = FileUtil.toFileObject(new File(foldForGen, "newfile.txt"));
java.io.File and its FS API equivalents
| java.io.File | FS API equivalents |
|---|---|
| mkdirs | FileUtil.createFolder(java.io.File) |
| mkdir | FileObject.createFolder |
| createNewFile | FileObject.createData |
| mkdirs + createNewFile | FileUtil.createData(java.io.File) |
| delete | FileObject.delete (can be used to delete the whole folder recursively) |
| rename | FileObject.rename |
How NOT to fix:
File f = ...;
FileObject fo = FileUtil.toFileObject(f);
if (fo == null) {
fs.refresh();
fo = FileUtil.toFileObject(f);
}
How to identify possible problems (for developers and QA):
- watch ide log for WARNING printed from FS impl. if inconsistency between files on disk and FS caches. Should be reported, evaluated and fixed whenever appears (even if no real obvious bug)
- grep your code for occurrences of suspicious calls like (createNewFile, mkdir, mkdirs) and check if really called properly
Patterns of wrong usage:
- create new file
File f = ...; f.createNewFile(); //FS doesn't know nothing about newly created file FileObject fo = FileUtil.toFileObject(f);
- create new folder
File f = ...; f.mkdir(); //FS doesn't know nothing about newly created folder FileObject fo = FileUtil.toFileObject(f);
- run external command (modifying files layout)
ProcessBuilder pb = new ProcessBuilder(command); Process process = pb.start(); process.waitFor(); //FS doesn't know nothing about newly generated files, folders FileObject fo = FileUtil.toFileObject(generatedFile);
Mixing java.io, FS API, external tools
Files can be created, deleted, renamed, .. - simply modified by:
- FS API
- java (java.io, ...)
- external tools like ant, rails, ...
There is no problem to use all of them interchangeably. Even there is no problem not to use FS API at all as long as not needed. But as soon as the code needs to modify files layout to create, delete or modify files then is necessary to know that FS API is the communication door (about file modifications) to the other APIs in Netbeans and thus if filesystems are not informed about these changes then also datasystem, nodes, project view, versioning ... won't know about them.
See also:
One drive == one instance of filesystem on windows
- #125426 (FIXED) FileObject.getPath() incompatible with NB 6.0 version
- #125203 (FIXED) Files and folders not refreshed in IDE
Justification:
makes possible to refresh individual drives on Windows separately on demand (especially if network shared folders mapped as windows drives)
How to identify possible problems (for developers):
- grep your code to see if method FileObject.getPath is used properly according to javadoc.
- grep your code to see that method FileSystem.refresh is called on the right instance of FileSystem. There is no one FS instance for all reachable files.
How to fix other related bugs (for developers):
- Find motivation in already fixed issues #125426 getPath() and #125203 FileSystem.refresh()
See also commits:
