FitnessForNBMs

(Difference between revisions)
(Open Questions)
(Open Questions)
 
Line 45: Line 45:
Do we need changes in apisupport's new library wrapper wizard? To detect whether library can be compressed by [[Pack200]] or not?
Do we need changes in apisupport's new library wrapper wizard? To detect whether library can be compressed by [[Pack200]] or not?
-
----
+
''The simplest way is to compress only contained JAR files, not the whole NBM.''
-
 
+
-
"The simplest way is to compress only contained JAR files, not the whole NBM."
+
But text files, especially XML files, can take up a good deal of space.  If the one of the major goals is to reduce size, then wouldn't it make sense to compress the text files too?  Even a general compression scheme like GZip should easily be able to deflate text files by 80% or more with very little processing overhead.
But text files, especially XML files, can take up a good deal of space.  If the one of the major goals is to reduce size, then wouldn't it make sense to compress the text files too?  Even a general compression scheme like GZip should easily be able to deflate text files by 80% or more with very little processing overhead.

Current revision as of 23:00, 22 January 2010

The NBM format remains the same since 1998 and does not use latest enhancements in JAR compressions. This adds unnecessary additional cost to every NBM download in terms of transmitted bits as well as download time.

The goal of this performance related project is to design new, smaller format for NBM files to save bandwidth and download time by compressing them files using Pack200.

The status is tracked in Issue 84852

Contents

Goals & Constraints

  • reduce the typical NBM download size by 70%
  • no additional load on NetBeans release engineering group
  • somehow share and unify the experience/format between NBI and NBM files

Design

The simplest way is to compress only contained JAR files, not the whole NBM. E.g. having

Info/info.xml
config/Modules/org-netbeans-libs-jsr223.xml
netbeans/modules/ext/script-api.jar.pack.gz
netbeans/modules/org-netbeans-libs-jsr223.jar.pack.gz

inside org-netbeans-libs-jsr223.nbm. The tasks and AU need to be modified to recognize and properly handle such file.

The MakeNBM task needs additional parameter usepack200. By default (for backward compatibility) this parameter will be false. The harness/common.xml will be modified to set it to true (by default). As we can't pack signed jars as well as some others (e.g. mysql-connector with JDK5, and a bunch of jars in JavaFX SDK, and files which goes together with .jad descriptor), one will be allowed to override this in nbprojec/project.properties and disable use of Pack200. There are some tools in NBI to do packaging and further verification, they can be reused in AU. This brings the NBI and NBM closer together.

Maven relies on NBMs heavily during development. Slowing the build/run/edit cycle by using Pack200 is not desirable, thus Maven will employ the optional parameter to disable usepack200. Also the Maven 6.9 and future repositories will contains unpacked NBMs. Those can be built locally with ant build-nbms -Dusepack200=false.

AutoUpdateTask needs to be updated to handle packaged JARs properly.

Auto Update itself needs to be update to handle Pack200 NBMs.

Maven will reuse the AutoUpdateTask for processing NBMs. The latest version of the task allows one to specify local fileset of NBMs, so it can be used without any connection to internet. This shields Maven from necessity to understand NBM files and behavior and creates a standardized API for maven plugin to install NBM files.


Pack200 is not memory leak free, so we need to make sure to not run out of available memory or leave (too much) garbage after its use. If there are no static references in Pack200 implementation, then it is enough to throw instances of the packager/unpackager away every time a single NBM is processed. If there are some static fields we need to resort to reflection to clear them after processing fixed number of NBMs.

Open Questions

How to build the new Maven plugin as there is no Maven repository that hosts the AutoUpdateTask right now?

Do we need changes in apisupport's new library wrapper wizard? To detect whether library can be compressed by Pack200 or not?

The simplest way is to compress only contained JAR files, not the whole NBM.

But text files, especially XML files, can take up a good deal of space. If the one of the major goals is to reduce size, then wouldn't it make sense to compress the text files too? Even a general compression scheme like GZip should easily be able to deflate text files by 80% or more with very little processing overhead.

If the entire NBM file will be a ZIP archive (as is currently) the case, you can ignore this comment. However, it was not clear from reading this page what would be used to enclose the individual files within the NBM.

Resolved Questions

  • mkleint: in maven the nbms are used as means of composing the application. On assumption that pack is slower than regular zip, I'd rather have the pack stuff implemented just optionally, so that it can be applied on the maven projects when releasing the final bits and not for regular development. Since the app is composed from packed nbms I hope it will work not only for update center, but also for modules in default distribution.
    • Yes, use of Pack200 will be optional.
  • mkleint: not sure what what the AutoUpdateTask does, but anything that downloads stuff from nb update centers needs to be dropped.
  • One needs to be careful with foreign signed JARs like JavaHelp; easiest to leave these intact. You could explode and repack the META-INF/* files which determine the signature, but you could not run Pack200 on these classfiles anyway - unless the JAR has already been "normalized" before signing (*), which it would be possible to detect from a tool at some build-time cost.
    • One can configure its nbproject/project.properties to disable use of Pack200 for such JARs.
Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo