FileSystems

Revision as of 17:08, 18 February 2010 by Tpavek (Talk | contribs)
(diff) ← Older revision | Current revision (diff) | Newer revision → (diff)

Improvements in FileSystems

Contents


Performance

Performance of FileSystem library is often subject of claims especially if used in network environment. This page describes improvements done as part of performance work on NetBeans 6.1 release.

The poor performance is caused because of: API design, implementation, and usage...

FS API design + performance

Performance of MasterFS impl. is limited by API design. The most important API limits:

  1. API for the VCS Generic infrastructure (dropped for 6.1)
  2. only valid instances of FileObject are issued from FileSystem
  3. for unique file path is issued only one shared valid instance of FileObject


Usage of FS API + performance

Wrong usage from performance point of view:

  • not use FS API for writing, modifying and then call refresh (not calling refresh at all is even worse and may cause real bug!)
  • uselessly convert File to FileObject to get, read some information provided by java.io.File
  • spending a lot of time in in FileChangeListener impl.
  • refreshing the whole filesystem if not necessary

Fixing, improving performance

Improving performance of masterfs is goal for 6.1 (part of post-6.0 performance work)


List of assumptions how to achieve better performance:

  • drop support for VCS Generic
  • eliminate all useless disk touches
  • less instances of FileObject in memory => faster refresh
  • not instantiate all parents (up to the root) for one instance of FileObject
  • not reference children from parents (and vice versa), keep weak - to let garbage collect not necessary instances
  • lightweight locks - not to create additional files for locking if not necessary


Steps/state:

  • developed on branch masterfs_novcs and merged into trunk |6.1 milestone M1|
  • focused on backward compatibility |6.1 milestone M2|
  • provided guide how to fix potential problems caused by API changes


Guide API changes for 6.1

List of API changes in MasterFileSystem for 6.1

  1. Fileobject instances are not guaranteed to be issued after external changes ( see details)
  2. One drive == one instance of filesystem on windows ( see details)
  3. Simplify MasterFS, do not delegate on other embedded FS


Justification:

  • simplify masterfs #123542
  • perf.improvements


FileObject instances are not guaranteed to be issued after external changes

Justification:

  • performance improvements; FS doesn't need to test again and again whether File really exists
  • only implementation detail (never documeneted)


Potential problems:

  • methods in FS API may return null although the file really exist
  • methods in FS API should return null but return old cached instance instead although the file was deleted or moved


Affected methods:

  • FileObject.getFileObject
  • FileObject.getChildren
  • FileSystem.findResource
  • FileUtil.toFileObject


Known issues:

  • #125090 (FIXED) Debugger doesn't stop at breakpoint on Windows
  • #125575 (FIXED) Safe delete Restored file is not visible in project view after undo
  • #126038 (NEW) java.io.SyncFailedException: C:\Documents and Settings\Krystyna Polomski\My Documents\NetBeansProjects\WebApplication803\web\resources\stylesheet.css
  • #125720 (NEW) Custom tasks like JPDAStart, ... may use inconsistent MasterFS


How to fix other related bugs (for developers):

  • read how to use properly FS API
  • see related thread on nbdev
  • find motivation in already fixed issues


Patterns of right usage

  • create new file
File f = ...;
FileObject fo = FileUtil.createData(f);
  • create new folder
File f = ...;
FileObject fo = FileUtil.createFolder(f);
  • run external command (modifying files layout)
File foldForGen =...;
FileObject foFoldForGen = FileUtil.toFileObject(foldForGen);
ProcessBuilder pb = new ProcessBuilder(command);
Process process = pb.start();
process.waitFor();
foFoldForGen.getFileSystem().refresh();
FileObject fo = FileUtil.toFileObject(new File(foldForGen, "newfile.txt"));


java.io.File and its FS API equivalents

java.io.File FS API equivalents
mkdirs FileUtil.createFolder(java.io.File)
mkdir FileObject.createFolder
createNewFile FileObject.createData
mkdirs + createNewFile FileUtil.createData(java.io.File)
delete FileObject.delete (can be used to delete the whole folder recursively)
rename FileObject.rename


How NOT to fix:

File f = ...;
FileObject fo = FileUtil.toFileObject(f);
if (fo == null) {
 fs.refresh();
 fo = FileUtil.toFileObject(f);
}


How to identify possible problems (for developers and QA):

  • watch ide log for WARNING printed from FS impl. if inconsistency between files on disk and FS caches. Should be reported, evaluated and fixed whenever appears (even if no real obvious bug)
  • grep your code for occurrences of suspicious calls like (createNewFile, mkdir, mkdirs) and check if really called properly


Patterns of wrong usage:

  • create new file
File f = ...;
f.createNewFile();
//FS doesn't know nothing about newly created file
FileObject fo = FileUtil.toFileObject(f);
  • create new folder
File f = ...;
f.mkdir();
//FS doesn't know nothing about newly created folder
FileObject fo = FileUtil.toFileObject(f);
  • run external command (modifying files layout)
ProcessBuilder pb = new ProcessBuilder(command);
Process process = pb.start();
process.waitFor();
//FS doesn't know nothing about newly generated files, folders
FileObject fo = FileUtil.toFileObject(generatedFile);


Mixing java.io, FS API, external tools
Files can be created, deleted, renamed, .. - simply modified by:

  • FS API
  • java (java.io, ...)
  • external tools like ant, rails, ...


There is no problem to use all of them interchangeably. Even there is no problem not to use FS API at all as long as not needed. But as soon as the code needs to modify files layout to create, delete or modify files then is necessary to know that FS API is the communication door (about file modifications) to the other APIs in Netbeans and thus if filesystems are not informed about these changes then also datasystem, nodes, project view, versioning ... won't know about them.

See also:


One drive == one instance of filesystem on windows

  • #125426 (FIXED) FileObject.getPath() incompatible with NB 6.0 version
  • #125203 (FIXED) Files and folders not refreshed in IDE


Justification: makes possible to refresh individual drives on Windows separately on demand (especially if network shared folders mapped as windows drives)
How to identify possible problems (for developers):

  • grep your code to see if method FileObject.getPath is used properly according to javadoc.
  • grep your code to see that method FileSystem.refresh is called on the right instance of FileSystem. There is no one FS instance for all reachable files.


How to fix other related bugs (for developers):


See also commits:

Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo