How do I set the JVM's heap size?
Open the configuration file etc/netbeans.conf in a text editor. (The 'etc' directory under the directory where NetBeans is installed, which on Mac means inside the .app, somewhere like "NetBeans 7.1.app/Contents/Resources/NetBeans/etc/"). In the netbeans_default_options setting, change or add the following options: -J-Xmx... (e.g. -J-Xmx1600M) or -J-XX:MaxPermSize=... (e.g. -J-XX:MaxPermSize=500M). Save the file and restart the IDE.
UI responsiveness may be affected when the heap utilization gets close to its limit. Should you encounter an OutOfMemoryError, you need to increase Xmx or XX:MaxPermSize.
NetBeans 6.x+ note
Since version 6.0, NetBeans defaults to dynamically setting its Xmx heap size limit to something like 1/3 or 1/4 of the RAM installed on the system. For that reason there's no -J-Xmx value set in the default netbeans.conf. If you find that the automatically selected limit is too little/too much, you can of course still add an appropriate -J-Xmx... option to your netbeans.conf) to override the automatically selected limit.
Running the 32-bit JVM
Also note that if you're running the 64-bit JVM, NetBeans will be using almost twice as much memory as it will if you run the 32-bit JVM. To make NetBeans use the 32-bit JVM, you can again make some changes in the netbeans.conf file mentioned above. Either uncomment and set the netbeans_jdkhome variable to the path of the 32-bit JVM or, if you don't have separate 64-bit and 32-bit JVMs on your machine (e.g. OS X Lion), then try adding -J-d32 to the netbeans_default_options line to make the JDK use a 32-bit mode (if available).
Note that if run Netbeans with a differnt JVM, the default Java Platform for your applications will also change. If you want to build/run your applications with a different JDK, you can specify it in the Libraries section of the project properties (may need to define a Java Platform using Manage Platforms button).
WARNING! If you select the 32-bit JVM, you may find that NetBeans will not start if you specify too big a heap size using -J-Xmx. If you're starting NetBeans from the command line you will see the following message (or if you're starting it by clicking an icon you may not see any message):
Error occurred during initialization of VM Could not reserve enough space for object heap Could not create the Java virtual machine.
Using Apple's JVM on OS X Lion with -J-d32 the maximum -J-Xmx seems to be about -J-Xmx1820M. For the 32-bit JVM from Oracle (HotSpot) the limit is closer to 3 GB.
Applies to: NetBeans 4.x, 5.x, 6.x, 7.x