First you should verify that you do have java 6 on you system. The fastest way is to run this command in a terminal.
you should get something that looks like this:
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03-211)
Java HotSpot(TM) 64-Bit Server VM (build 11.3-b02-83, mixed mode)
If you don't then there is your first problem. The command above was taken from a line in the nsm.lax file and looks like this
I didn't touch it; it was already set.
Now at the bottom of the file (NSM.lax) is a section that looks like this:
# the install directory
NSDIR=/Users/Bird/Desktop/Network and Security Manager
In my case i have the NSM sitting on my desktop. Set that line and run this ./NSM in the directory and you should be good to go.
If you still have problems then i'm not sure because that's all i did to get it working. In fact i didn't have to even use the JAVA_HOME variable. What version of OS X are you using? I'm using 10.5.7 and i don't remember when JAVA 6 was installed via updates but i don't think it was that long ago. 6 months maybe?