Fitness Against Forgetfulness
Improvements in loaders' scalability done for NetBeans 6.5 release.
Quick howto: Register Your Loader Properly
- Remove any definition of your loader from manifest of your module
- Define a MIME Type for your filetype (you can use File Type wizard to do that)
- Register your loader in Loader/yourmime/yourtype/Factories in your layer file
Don't forget what you have just computed!
The computers are getting faster and faster and as such even extensive algorithms get over time faster and grow into acceptable shape. Many developers do not appreciate and try as hard as possible make the algorithms worse and worse so the effect of faster processors and more memory does not apply. However this is quite a hard job. It is tough to invent new and new ways to slow down an application. It often needs quite inventive tricks! However one of the most promising ones is to often forget what has been computed and compute that once again. If you do this multiple times per second, the result can be quite perfect, the program really becomes slow and there does not seem to be an easy way to fix that. Unless we find a way to make your program remember.
Improve Transfer of Knowledge
The Data System API with its Loader is an example of such "forgetfulness". It's task is to convert files of certain type, usually mime type, into instances of DataObjects. In most of the cases that is single mapping between the mimetype and the DataObject type. In some situations, the mapping is not 1:1, but still the amount of DataObject types per one mimetype is limited.
So what is the best way to exhaust as much of CPU? Use mimeresolvers to compute the right mimetype and immediatelly forget it and then traverse all the existing DataObjects once again and ask them whether they want to file or not!
This really slows down everything. Even, after identifying the mimetype of a file, we almost certainly know which loader shall be in charge of that file, we rather iterate through the tens of registered loaders to find it. This indeed results in quite extensive disk access, CPU computation, memory allocations, garbage collector's load.
This is so excessive that performance team cannot stand it anymore. We need to create a fix - create better ways to register data loaders!
Scalability is highly needed
The following table shows how unscalable our current forgetfulness is. It shows that the number of disk touches, e.g. operations increases in 6.1 version quite significantly.
However it also shows that there is a way to solve this issue. It is possible to apply these little changes to Java loader registrations and the system suddenly starts to be much more scalable. Any reason not to apply similar changes to other loaders?
We want to implement this feature for release 6.5. It looks acceptable to do it in three steps.
Prototype and Review
- by May 22, 2008
- put your comments here or into the issue 91665
- UI spec. is here. It also addresses the open with problem as wished by PHP and many others
Integrate the Infrastruture
- Integrate before M1.
- Fix all loaders in big IDE.
- All teams need to check and verify.
- UI later during M2.
- API support later during M2
TCRs and TCAs
- cd164f315c3c: TCR - Compatibility is important: Have the old (manifest-registered) loaders at the beginning of the recognition chain (before mime types).
- 3dea1f96e573: TCR: Agreed to remove any special support for XML: What to do with text/ant+xml? Just fallback to making content/unknown a fallback category for all loaders".
- TCA - someone should change it all... also because UI
- d0d56c95bf44: TCA - use DataObject.Factory
- TCA: Provide infrastructure so all loaders placed in layer files should work in unit tests if utils, filesystem and loaders are on classpath - as long as given mime type is resolved.
- 4 yes - approved
- Will do next week without UI changes (later).
Additional Notes and Requirements about the UI
JG02 - UI in Options: Not clear how this is going to work for non-trivial cases. Assuming list of file extensions maps to mime type directly.
- TCR: All mime resolvers mapping that are plain extension-based. E.g. ant has one - for 'ant', and couple of more complicated associations that need not be shown. Complicated cases hidden from GUI. (So e.g. xml associations.) When the user wants to register their own file extension to a mime type/loader association, all mime types must be available to choose from.
- 82319770e503 Set of registered extensions is populated from Services/MIMEResolver/* and set of available MIME types from union of Services/MIMEResolver/* and Loaders/*.
- TCR: User customization will get a new resolver created when needed. No edits of original ones (e.g. if user adding a new extension for ant). And positioned as first in the chain of loaders.
- 82319770e503 New resolver is created in userdir/config/Services/MIMEResolver and position attribute set to fixed value 10.
- TCR: In UI "Opens With" is misleading - listing possible loaders - remove it.
- 82319770e503 Removed.
- TCR: "File Extensins and Patterns" - remove pattern - resolvers only work with extensions. Don't use *.ext, only list extensions.
- 82319770e503 Only extension is used.
- TCR: Context menu for unrecognized files - have special "Open As" action, "Open" should just open in text editor - so doubleclick always work immediately.
- 82319770e503 "Open" and "Open As" coexist for unknown files. It is a question if action "Open in System" is still needed.
- TCA: Investigate how easy should it be to screw up the IDE - e.g. by registering something strange for java extension? We should avoid happening that by accident (not noticing). Compare to other apps - e.g. Firefox.
- It seems nothing horrible happens when you change resolver for java files (one exception from editor was thrown). Anyway user can revert wrong setting using the Default button in the File Associations panel.
- Remove localized labels from mime resolvers? Not useful, garbage. Plus change API support not to generate them. But probably not needed. TCR: Fix the API support.