[GE users] SDM 6.2U3 won't start

cbyun cbyun at ll.mit.edu
Wed Jul 1 16:02:02 BST 2009


Hi Michal,

I always run the command as a root user.

Interesting thing is that, after commenting out the following line in the policy file in the SDM spool directory as Richard suggested,

    /* permission javax.management.MBeanPermission "*", "runtime"; */

I can start cs_vm manually as a root user using the start_sc_vm.sh file that I captured previously from the SDM's tempory spool directory. However, the sdmadm suj -j cs_vm command always fails.

I guess the question is why the start_sc_vm.sh created by the sdmadm suj -j cs_vm command worked manually but the command itself failed to start up cs_vm.

Here is my manually start up using the start_cs_vm.sh script:

# /tmp/start_cs_vm.sh
Waiting for pid file
Waiting for pid file

# sdmadm sj
name  host      state      used_mem  max_mem   message
---------------------------------------------------------------------------------------
cs_vm llgriddev STARTED          17M      878M

So are you suspecting that the sdmadm command switch to the Sdm admin user, sge, before executing the start up script?

- Chansup

> -----Original Message-----
> From: Michal.Bachorik at sun.com [mailto:Michal.Bachorik at sun.com]
> Sent: Wednesday, July 01, 2009 10:56 AM
> To: users at gridengine.sunsource.net
> Subject: Re: [GE users] SDM 6.2U3 won't start
>
> Ok Chansup,
>
> try to start it as root user please :) ..the same command I sent you
> before (with log settings). And send me the results, please.
>
> Regards,
>
> Michal
>
> cbyun wrote:
> > Here you go:
> >
> > # sdmadm -s sdm62u3 -log
> com.sun.grid.grm.bootstrap.ParentStartupService=FINE suj -j cs_vm
> > DEBUG: Add /opt/sdm/lib/sdm-cloud-adapter.jar to system classpath
> > DEBUG: Add /opt/sdm/lib/sdm-security.jar to system classpath
> > DEBUG: Add /opt/sdm/lib/sdm-common.jar to system classpath
> > DEBUG: Add /opt/sdm/lib/sdm-ge-adapter.jar to system classpath
> > DEBUG: Add /opt/sdm/lib/sdm-starter.jar to system classpath
> > DEBUG: Add /opt/sdm/lib/ext/jsr173_1.0_api.jar to system classpath
> > DEBUG: Add /opt/sdm/lib/ext/jaxb-impl.jar to system classpath
> > DEBUG: Add /opt/sdm/lib/ext/activation.jar to system classpath
> > DEBUG: Parameter for "cs_vm"
> > DEBUG: Commandline: ""/usr/java/jdk1.6.0_13/jre/bin/java"   -
> Djava.security.manager=java.rmi.RMISecurityManager -
> Djava.security.policy=/var/spool/sdm/sdm62u3/security/java.policy -
> Djava.security.auth.login.config=/var/spool/sdm/sdm62u3/security/jaas.conf
> ig -Dcom.sun.grid.grm.bootstrap.systemname=sdm62u3 -
> Dcom.sun.grid.grm.bootstrap.jvmname=cs_vm -
> Dcom.sun.grid.grm.bootstrap.localspool="/var/spool/sdm/sdm62u3" -
> Dcom.sun.grid.grm.bootstrap.dist="/opt/sdm" -
> Dcom.sun.grid.grm.bootstrap.csInfo="llgriddev:6447" -
> Dcom.sun.grid.grm.bootstrap.preferencesType="SYSTEM" -
> Djava.util.logging.manager=com.sun.grid.grm.util.GrmLogManager -
> Djava.library.path="/opt/sdm/lib/lx-amd64::/opt/sdm/lib/lx-amd64:" -
> Dcom.sun.grid.grm.bootstrap.isCS="true" -cp "/opt/sdm/lib/sdm-cloud-
> adapter.jar:/opt/sdm/lib/sdm-security.jar:/opt/sdm/lib/sdm-
> common.jar:/opt/sdm/lib/sdm-ge-adapter.jar:/opt/sdm/lib/sdm-
> starter.jar:/opt/sdm/lib/ext/jsr173_1.0_api.jar:/opt/sdm/lib/ext/jaxb-
> impl.jar:/opt/sdm/lib/ext/activation.jar" -
> Djava.rmi.server.codebase="file:/opt/sdm/lib/sdm-cloud-adapter.jar
> file:/opt/sdm/lib/sdm-security.jar file:/opt/sdm/lib/sdm-common.jar
> file:/opt/sdm/lib/sdm-ge-adapter.jar file:/opt/sdm/lib/sdm-starter.jar
> file:/opt/sdm/lib/ext/jsr173_1.0_api.jar file:/opt/sdm/lib/ext/jaxb-
> impl.jar file:/opt/sdm/lib/ext/activation.jar " -
> Djava.endorsed.dirs="/opt/sdm/lib/ext/endorsed" -
> Djava.rmi.server.hostname="llgriddev"  ${SDM_MEMORY_HEAP_SIZE_cs_vm} -
> Dcom.sun.grid.grm.management.connectionTimeout=60
> com.sun.grid.grm.bootstrap.JVMImpl  &"
> > DEBUG: User: "sge"
> > jvm   host      result message
> > -----------------------------------------------------
> > cs_vm llgriddev ERROR  JVM: cs_vm died during
> >                        startup.
> > Error: Command has generated error.
> >
> >
> > Thanks,
> > - Chansup
> >
> >
> >
> >> -----Original Message-----
> >> From: Michal.Bachorik at sun.com [mailto:Michal.Bachorik at sun.com]
> >> Sent: Wednesday, July 01, 2009 10:35 AM
> >> To: users at gridengine.sunsource.net
> >> Subject: Re: [GE users] SDM 6.2U3 won't start
> >>
> >> Chansup,
> >>
> >> please try to start only cs vm with these options:
> >>
> >> sdmadm -s sdm62u3 -log
> >> com.sun.grid.grm.bootstrap.ParentStartupService=FINE suj -j cs_vm
> >>
> >> and send us the output from screen, please.
> >>
> >> Regards,
> >>
> >> Michal
> >>
> >>
> >> cbyun wrote:
> >>
> >>> I found the following err file on /tmp:
> >>>
> >>> # cat err.2730.44
> >>> java.security.policy: error adding Permission,
> >>>
> >> javax.management.MBeanPermission:
> >>
> >>>         java.lang.IllegalArgumentException: Invalid permission:
> runtime
> >>>
> >>> This might be the cause of the start up failure.
> >>> Anyway, Below is the stack trace that I got:
> >>>
> >>> # /bin/sh -x /opt/sdm/bin/sdmadm -d -s sdm62u3 suj
> >>> + PATH=/bin:/usr/bin:/sbin:/usr/sbin
> >>> + CLASS=com.sun.grid.grm.cli.SdmAdm
> >>> ++ dirname /opt/sdm/bin/sdmadm
> >>> + BASEDIR=/opt/sdm/bin
> >>> ++ cd /opt/sdm/bin
> >>> ++ pwd
> >>> + BASEDIR=/opt/sdm/bin
> >>> ++ dirname /opt/sdm/bin
> >>> + SDM_DIST=/opt/sdm
> >>> + JVM_ARGS=
> >>> + for arg in '"$@"'
> >>> + '[' -d = -D ']'
> >>> + for arg in '"$@"'
> >>> + '[' -s = -D ']'
> >>> + for arg in '"$@"'
> >>> + '[' sdm62u3 = -D ']'
> >>> + for arg in '"$@"'
> >>> + '[' suj = -D ']'
> >>> + '[' /usr/java/default = '' ']'
> >>> + JAVA=/usr/java/default/bin/java
> >>> + '[' '!' '(' -x /usr/java/default/bin/java -a '!' -d
> >>>
> >> /usr/java/default/bin/java ')' ']'
> >>
> >>> ++ /opt/sdm/util/arch
> >>> + ARCH=lx26-amd64
> >>> + export ARCH
> >>> + case $ARCH in
> >>> + ARCH=lx-amd64
> >>> + JVM_ARGS=' -Djava.library.path=/opt/sdm/lib/lx-amd64'
> >>> + JVM_ARGS=' -Djava.library.path=/opt/sdm/lib/lx-amd64 -
> >>>
> >> Djava.endorsed.dirs=/opt/sdm/lib/ext/endorsed'
> >>
> >>> + JVM_ARGS=' -Djava.library.path=/opt/sdm/lib/lx-amd64 -
> >>>
> >> Djava.endorsed.dirs=/opt/sdm/lib/ext/endorsed -
> >> Dcom.sun.grid.grm.management.connectionTimeout=20'
> >>
> >>> + /usr/java/default/bin/java -Djava.library.path=/opt/sdm/lib/lx-amd64
> -
> >>>
> >> Djava.endorsed.dirs=/opt/sdm/lib/ext/endorsed -
> >> Dcom.sun.grid.grm.management.connectionTimeout=20 -jar
> /opt/sdm/lib/sdm-
> >> starter.jar com.sun.grid.grm.cli.SdmAdm -d -s sdm62u3 suj
> >>
> >>> jvm         host            result message
> >>> -----------------------------------------------------------------
> >>> cs_vm       llgriddev.local ERROR  JVM: cs_vm died during
> >>>                                    startup.
> >>> com.sun.grid.grm.GrmException: JVM: cs_vm died during startup.
> >>>         at
> com.sun.grid.grm.bootstrap.ParentStartupService.start(Unknown
> >>>
> >> Source)
> >>
> >>>         at
> com.sun.grid.grm.ui.component.StartJVMCommand.execute(Unknown
> >>>
> >> Source)
> >>
> >>>         at com.sun.grid.grm.ui.impl.CommandServiceImpl.execute(Unknown
> >>>
> >> Source)
> >>
> >>>         at
> >>>
> >> com.sun.grid.grm.cli.cmd.components.StartJVMCliCommand.execute(Unknown
> >> Source)
> >>
> >>>         at com.sun.grid.grm.cli.AbstractCli.run(Unknown Source)
> >>>         at com.sun.grid.grm.cli.SdmAdm.main(Unknown Source)
> >>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>         at
> >>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> >> 39)
> >>
> >>>         at
> >>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >> pl.java:25)
> >>
> >>>         at java.lang.reflect.Method.invoke(Method.java:597)
> >>>         at
> com.sun.grid.grm.util.MainWrapper$SystemRunThread.run(Unknown
> >>>
> >> Source)
> >>
> >>> executor_vm llgriddev.local ERROR  Timeout. Pid file:
> >>>                                    /var/spool/sdm/sdm62u3/run/exe
> >>>                                    cutor_vm at llgriddev.local for
> >>>                                    JVM: executor_vm not found.
> >>> com.sun.grid.grm.GrmException: Timeout. Pid file:
> >>>
> >> /var/spool/sdm/sdm62u3/run/executor_vm at llgriddev.local for JVM:
> >> executor_vm not found.
> >>
> >>>         at
> com.sun.grid.grm.bootstrap.ParentStartupService.start(Unknown
> >>>
> >> Source)
> >>
> >>>         at
> com.sun.grid.grm.ui.component.StartJVMCommand.execute(Unknown
> >>>
> >> Source)
> >>
> >>>         at com.sun.grid.grm.ui.impl.CommandServiceImpl.execute(Unknown
> >>>
> >> Source)
> >>
> >>>         at
> >>>
> >> com.sun.grid.grm.cli.cmd.components.StartJVMCliCommand.execute(Unknown
> >> Source)
> >>
> >>>         at com.sun.grid.grm.cli.AbstractCli.run(Unknown Source)
> >>>         at com.sun.grid.grm.cli.SdmAdm.main(Unknown Source)
> >>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>         at
> >>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> >> 39)
> >>
> >>>         at
> >>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >> pl.java:25)
> >>
> >>>         at java.lang.reflect.Method.invoke(Method.java:597)
> >>>         at
> com.sun.grid.grm.util.MainWrapper$SystemRunThread.run(Unknown
> >>>
> >> Source)
> >>
> >>> rp_vm       llgriddev.local ERROR  JVM: rp_vm died during
> >>>                                    startup.
> >>> com.sun.grid.grm.GrmException: JVM: rp_vm died during startup.
> >>>         at
> com.sun.grid.grm.bootstrap.ParentStartupService.start(Unknown
> >>>
> >> Source)
> >>
> >>>         at
> com.sun.grid.grm.ui.component.StartJVMCommand.execute(Unknown
> >>>
> >> Source)
> >>
> >>>         at com.sun.grid.grm.ui.impl.CommandServiceImpl.execute(Unknown
> >>>
> >> Source)
> >>
> >>>         at
> >>>
> >> com.sun.grid.grm.cli.cmd.components.StartJVMCliCommand.execute(Unknown
> >> Source)
> >>
> >>>         at com.sun.grid.grm.cli.AbstractCli.run(Unknown Source)
> >>>         at com.sun.grid.grm.cli.SdmAdm.main(Unknown Source)
> >>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>         at
> >>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> >> 39)
> >>
> >>>         at
> >>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >> pl.java:25)
> >>
> >>>         at java.lang.reflect.Method.invoke(Method.java:597)
> >>>         at
> com.sun.grid.grm.util.MainWrapper$SystemRunThread.run(Unknown
> >>>
> >> Source)
> >>
> >>> Error: com.sun.grid.grm.GrmException: Command has generated error.
> >>>         at
> >>>
> >> com.sun.grid.grm.cli.cmd.components.StartJVMCliCommand.execute(Unknown
> >> Source)
> >>
> >>>         at com.sun.grid.grm.cli.AbstractCli.run(Unknown Source)
> >>>         at com.sun.grid.grm.cli.SdmAdm.main(Unknown Source)
> >>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>         at
> >>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> >> 39)
> >>
> >>>         at
> >>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >> pl.java:25)
> >>
> >>>         at java.lang.reflect.Method.invoke(Method.java:597)
> >>>         at
> com.sun.grid.grm.util.MainWrapper$SystemRunThread.run(Unknown
> >>>
> >> Source)
> >>
> >>> + exit 2
> >>>
> >>>
> >>> Also, I changed the logging level but I don't get cs_vm log.
> >>>
> >>> Here is what I have in the logging.properties:
> >>>
> >>> # Default global logging level.
> >>> # Loggers and Handlers may override this level
> >>> .level=FINE
> >>> javax.level=SEVERE
> >>>
> >>> com.sun.grid.grm.service.level=FINE
> >>>
> >>> # --- grid.grm logging level ---
> >>> com.sun.grid.grm.level=FINE
> >>>
> >>> # --- FileHandler ---
> >>> java.util.logging.FileHandler.level=ALL
> >>>
> >>>
> >>
> java.util.logging.FileHandler.formatter=com.sun.grid.grm.util.GrmFormatter
> >>
> >>> java.util.logging.FileHandler.pattern=${log_dir}/${jvm_name}-%g.log
> >>> java.util.logging.FileHandler.append=true
> >>> <cure here>
> >>>
> >>> # find /var/spool/sdm/sdm62u3/log -ls
> >>> 4406418    4 drwxr-xr-x   2 sge      root         4096 Jun 30 11:24
> >>>
> >> /var/spool/sdm/sdm62u3/log
> >>
> >>> 4406510    0 -rw-r--r--   1 root     root            0 Jun 30 11:09
> >>>
> >> /var/spool/sdm/sdm62u3/log/executor_vm.stderr
> >>
> >>> 4406511    0 -rw-r--r--   1 root     root            0 Jun 30 11:09
> >>>
> >> /var/spool/sdm/sdm62u3/log/executor_vm-0.log.lck
> >>
> >>> 4406516    4 -rw-r--r--   1 root     root          330 Jun 30 11:08
> >>>
> >> /var/spool/sdm/sdm62u3/log/executor_vm-0.log.1
> >>
> >>> 4406515    0 -rw-r--r--   1 root     root            0 Jun 30 11:07
> >>>
> >> /var/spool/sdm/sdm62u3/log/executor_vm-0.log.1.lck
> >>
> >>> 4406512   96 -rw-r--r--   1 root     root        90359 Jun 30 11:10
> >>>
> >> /var/spool/sdm/sdm62u3/log/executor_vm-0.log
> >>
> >>> 4406505    0 -rw-r--r--   1 root     root            0 Jun 30 11:09
> >>>
> >> /var/spool/sdm/sdm62u3/log/executor_vm.stdout
> >>
> >>> There is not much valuable information in executor_vm-0.log. It is
> >>>
> >> waiting for cs_vm and failed.
> >>
> >>> Any further suggestions are welcome.
> >>>
> >>> Thanks,
> >>> - Chansup
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Michal.Bachorik at sun.com [mailto:Michal.Bachorik at sun.com]
> >>>> Sent: Tuesday, June 30, 2009 10:06 AM
> >>>> To: users at gridengine.sunsource.net
> >>>> Cc: Byun, Chansup
> >>>> Subject: Re: [GE users] SDM 6.2U3 won't start
> >>>>
> >>>> also, you should increase the logging level (in
> >>>> <sdp_spool>/logging.properties).
> >>>>
> >>>> regards,
> >>>>
> >>>> m.
> >>>>
> >>>> easymf wrote:
> >>>>
> >>>>
> >>>>> chansup,
> >>>>>
> >>>>>
> >>>>> try to start it with "sdmadm -d -s sdm62u3 suj". it will print a
> >>>>> stacktrace .. and also, look at the log files in sdm spool dir.
> >>>>> as all of your jvms has "died" (refused to start), it seems to be a
> >>>>> problem with installation or configuration.
> >>>>>
> >>>>> regards,
> >>>>>
> >>>>> michal
> >>>>>
> >>>>> cbyun wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I just installed SDM from SGE 6.2u3 release without any errors.
> >>>>>> Then, I tried to start JVMs but none of them started with the
> >>>>>>
> >> following
> >>
> >>>> error:
> >>>>
> >>>>
> >>>>>> # sdmadm -s sdm62u3 suj
> >>>>>> jvm         host            result message
> >>>>>> -----------------------------------------------------------------
> >>>>>> cs_vm       llgriddev.local ERROR  JVM: cs_vm died during
> >>>>>>                                    startup.
> >>>>>> executor_vm llgriddev.local ERROR  Timeout. Pid file:
> >>>>>>                                    /var/spool/sdm/sdm62u3/run/exe
> >>>>>>                                    cutor_vm at llgriddev.local for
> >>>>>>                                    JVM: executor_vm not found.
> >>>>>> rp_vm       llgriddev.local ERROR  JVM: rp_vm died during
> >>>>>>                                    startup.
> >>>>>> Error: Command has generated error.
> >>>>>>
> >>>>>> I used Java 1.6.
> >>>>>>
> >>>>>> # echo $JAVA_HOME
> >>>>>> /usr/java/latest
> >>>>>>
> >>>>>> # /usr/java/latest/bin/java -version
> >>>>>> java version "1.6.0_13"
> >>>>>> Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
> >>>>>> Java HotSpot(TM) 64-Bit Server VM (build 11.3-b02, mixed mode)
> >>>>>>
> >>>>>> Any suggestions for further troubleshooting?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> - Chansup
> >>>>>>
> >>>>>> ------------------------------------------------------
> >>>>>>
> >>>>>>
> >>>>>>
> >>
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId
> >>
> >>>> =204401
> >>>>
> >>>>
> >>>>>> To unsubscribe from this discussion, e-mail: [users-
> >>>>>>
> >>>>>>
> >>>> unsubscribe at gridengine.sunsource.net].
> >>>>
> >>>>
> >>>>> ------------------------------------------------------
> >>>>>
> >>>>>
> >>>>>
> >>
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId
> >>
> >>>> =204552
> >>>>
> >>>>
> >>>>> To unsubscribe from this discussion, e-mail: [users-
> >>>>>
> >>>>>
> >>>> unsubscribe at gridengine.sunsource.net].
> >>>>
> >>>> ------------------------------------------------------
> >>>>
> >>>>
> >>
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId
> >>
> >>>> =204605
> >>>>
> >>>> To unsubscribe from this discussion, e-mail: [users-
> >>>> unsubscribe at gridengine.sunsource.net].
> >>>>
> >>>>
> >>> ------------------------------------------------------
> >>>
> >>>
> >>
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId
> >> =204625
> >>
> >>> To unsubscribe from this discussion, e-mail: [users-
> >>>
> >> unsubscribe at gridengine.sunsource.net].
> >>
> >> ------------------------------------------------------
> >>
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId
> >> =204805
> >>
> >> To unsubscribe from this discussion, e-mail: [users-
> >> unsubscribe at gridengine.sunsource.net].
> >>
> >
> > ------------------------------------------------------
> >
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId
> =204808
> >
> > To unsubscribe from this discussion, e-mail: [users-
> unsubscribe at gridengine.sunsource.net].
> >
>
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId
> =204813
>
> To unsubscribe from this discussion, e-mail: [users-
> unsubscribe at gridengine.sunsource.net].

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=204814

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list