By default, JRuby will be run with the same Java Platform (JVM or JDK) as the one running NetBeans (which you in turn can control with the --jdkhome flag). You can control which Java Platform is used for JRuby by using the System property jruby.java.home. Edit your etc/netbeans.conf file and add the following -J-Djruby.java.home=directory, e.g:
-J-Djruby.java.home=/System/Library/Frameworks/JavaVM.framework/Versions/1.5/Home
It is currently impossible via NetBeans UI. To be fixed in the future.