Note that for Rails Project there is not the JVM Arguments field yet (see issue 146551). But you might edit the JVM Arguments by hand. Go to the Files View (Ctrl+2) navigate to nbproject directory and open project.properties and add line like:
jvm.args=-Xmx=1024m
export JRUBY_EXTRA_VM_ARGS=-Xmx=1024m