[GE users] SDM 6.2U3 won't start

easymf michal.bachorik at sun.com
Wed Jul 1 15:45:21 BST 2009


chansup,

the behavior is correct, the template file from distribution dir
(/opt/sdm/) is being copied to local spool directories on master and
managed hosts.

but the faulty policy file should not cause your kind of troubles
(except the message in err file).

regards,

m.


cbyun wrote:
> It seems that during the SDM installation, the Java policy template was copied into the SDM spool directory as is:
>
> # diff /opt/sdm/util/templates/java.policy.template /var/spool/sdm/sdm62u3/security/java.policy
>
> What is the expected behavior here?
>
> - Chansup
>
>
>> -----Original Message-----
>> From: cbyun [mailto:cbyun at ll.mit.edu]
>> Sent: Wednesday, July 01, 2009 9:28 AM
>> To: users at gridengine.sunsource.net
>> Subject: RE: [GE users] SDM 6.2U3 won't start
>>
>> Hi Richard,
>>
>> I'm using exactly the same JVM:
>> # java -version
>> java version "1.6.0_13"
>> Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
>> Java HotSpot(TM) 64-Bit Server VM (build 11.3-b02, mixed mode)
>>
>>
>> I'm running the CentOS 5.3 built by Rocks 5.2.
>>
>> # cat /etc/*release
>> CentOS release 5.3 (Final)
>> Rocks release 5.2 (Chimichanga)
>> # uname -a
>> Linux llgriddev 2.6.18-128.1.6.el5 #1 SMP Wed Apr 1 09:10:25 EDT 2009
>> x86_64 x86_64 x86_64 GNU/Linux
>>
>> However, majority of MBean policy was set correctly in my java.policy:
>>
>> # grep MBean /var/spool/sdm/sdm62u3/security/java.policy
>>      permission javax.management.MBeanTrustPermission "register";
>>    permission javax.management.MBeanServerPermission "createMBeanServer";
>>    permission javax.management.MBeanTrustPermission "register";
>>    permission javax.management.MBeanPermission "*", "registerMBean";
>>    permission javax.management.MBeanPermission "*", "unregisterMBean";
>>    permission javax.management.MBeanPermission "*", "getDomains";
>>    permission javax.management.MBeanPermission "*", "getObjectInstance";
>>    permission javax.management.MBeanPermission "*", "queryMBeans";
>>    permission javax.management.MBeanPermission "*", "queryNames";
>>    permission javax.management.MBeanPermission "*", "getAttribute";
>>    permission javax.management.MBeanPermission "*", "getMBeanInfo";
>>    permission javax.management.MBeanPermission "*",
>> "addNotificationListener";
>>    permission javax.management.MBeanPermission "*",
>> "removeNotificationListener";
>>    permission javax.management.MBeanPermission "*", "isInstanceOf";
>>    permission javax.management.MBeanPermission "*", "getObjectInstance";
>>    permission javax.management.MBeanPermission "*", "invoke";
>>      /* permission javax.management.MBeanPermission "*", "runtime"; */
>>    permission javax.management.MBeanServerPermission "createMBeanServer";
>>    permission javax.management.MBeanTrustPermission "register";
>>    permission javax.management.MBeanPermission "*", "registerMBean";
>>    permission javax.management.MBeanPermission "*", "unregisterMBean";
>>    permission javax.management.MBeanPermission "*", "getDomains";
>>    permission javax.management.MBeanPermission "*", "getObjectInstance";
>>    permission javax.management.MBeanPermission "*", "queryMBeans";
>>    permission javax.management.MBeanPermission "*", "queryNames";
>>    permission javax.management.MBeanPermission "*", "getAttribute";
>>    permission javax.management.MBeanPermission "*", "getMBeanInfo";
>>    permission javax.management.MBeanPermission "*",
>> "addNotificationListener";
>>    permission javax.management.MBeanPermission "*", "isInstanceOf";
>>    permission javax.management.MBeanPermission "*", "getObjectInstance";
>>    permission javax.management.MBeanPermission "*", "invoke";
>>    permission javax.management.MBeanPermission
>> "com.sun.grid.grm.ui.ConfigurationService#run", "getMBeanInfo";
>>    permission javax.management.MBeanPermission
>> "com.sun.grid.grm.ui.ConfigurationService#run", "invoke";
>>    permission javax.management.MBeanPermission
>> "com.sun.grid.grm.ui.ConfigurationService#state", "getAttribute";
>>
>> Even if I commented out the one line which  you recommended, cs_vm still
>> doesn't start.
>>
>> Any further suggestions?
>>
>> Thanks,
>> - Chansup
>>
>>
>>
>>
>>> -----Original Message-----
>>> From: Richard.Hierlmeier at sun.com [mailto:Richard.Hierlmeier at sun.com]
>>> Sent: Wednesday, July 01, 2009 3:19 AM
>>> To: users at gridengine.sunsource.net
>>> Subject: Re: [GE users] SDM 6.2U3 won't start
>>>
>>> Hi Chansup,
>>>
>>> SDM uses its own java policy file. After the installation it is stored
>>>
>> at
>>
>>>    <local_spool_dir>/security/java.policy
>>>
>>> It seems that there is something wrong with this policy file. The action
>>> "runtime" for the javax.management.MBeanPermission does not exist.
>>>
>>> Please remove the following line from the java.policy file. You can
>>> comment it
>>> out with C-style comments:
>>>
>>> <    permission javax.management.MBeanPermission "*", "runtime";
>>>  > /* permission javax.management.MBeanPermission "*", "runtime"; */
>>>
>>> You can also remove/comment out this line in the
>>> <dist-dir>/util/templates/java.policy.template. This file is used as
>>> source for
>>> the java.policy file during the installation.
>>>
>>> The strange thing is that the problem does not occur in our environment.
>>> We
>>> tested it also with java6u13. On my opensolaris box I tested it with
>>> 1.6.0_13-b03:
>>>
>>> % java -version
>>> java version "1.6.0_13"
>>> Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
>>> Java HotSpot(TM) Server VM (build 11.3-b02, mixed mode)
>>>
>>> What is the exact version of your jvm?
>>>
>>> Richard
>>>
>>>
>>> cbyun wrote:
>>>
>>>> I found the following err file on /tmp:
>>>>
>>>> # cat err.2730.44
>>>> java.security.policy: error adding Permission,
>>>>
>>> javax.management.MBeanPermission:
>>>
>>>>         java.lang.IllegalArgumentException: Invalid permission:
>>>>
>> runtime
>>
>>>> This might be the cause of the start up failure.
>>>> Anyway, Below is the stack trace that I got:
>>>>
>>>> # /bin/sh -x /opt/sdm/bin/sdmadm -d -s sdm62u3 suj
>>>> + PATH=/bin:/usr/bin:/sbin:/usr/sbin
>>>> + CLASS=com.sun.grid.grm.cli.SdmAdm
>>>> ++ dirname /opt/sdm/bin/sdmadm
>>>> + BASEDIR=/opt/sdm/bin
>>>> ++ cd /opt/sdm/bin
>>>> ++ pwd
>>>> + BASEDIR=/opt/sdm/bin
>>>> ++ dirname /opt/sdm/bin
>>>> + SDM_DIST=/opt/sdm
>>>> + JVM_ARGS=
>>>> + for arg in '"$@"'
>>>> + '[' -d = -D ']'
>>>> + for arg in '"$@"'
>>>> + '[' -s = -D ']'
>>>> + for arg in '"$@"'
>>>> + '[' sdm62u3 = -D ']'
>>>> + for arg in '"$@"'
>>>> + '[' suj = -D ']'
>>>> + '[' /usr/java/default = '' ']'
>>>> + JAVA=/usr/java/default/bin/java
>>>> + '[' '!' '(' -x /usr/java/default/bin/java -a '!' -d
>>>>
>>> /usr/java/default/bin/java ')' ']'
>>>
>>>> ++ /opt/sdm/util/arch
>>>> + ARCH=lx26-amd64
>>>> + export ARCH
>>>> + case $ARCH in
>>>> + ARCH=lx-amd64
>>>> + JVM_ARGS=' -Djava.library.path=/opt/sdm/lib/lx-amd64'
>>>> + JVM_ARGS=' -Djava.library.path=/opt/sdm/lib/lx-amd64 -
>>>>
>>> Djava.endorsed.dirs=/opt/sdm/lib/ext/endorsed'
>>>
>>>> + JVM_ARGS=' -Djava.library.path=/opt/sdm/lib/lx-amd64 -
>>>>
>>> Djava.endorsed.dirs=/opt/sdm/lib/ext/endorsed -
>>> Dcom.sun.grid.grm.management.connectionTimeout=20'
>>>
>>>> + /usr/java/default/bin/java -Djava.library.path=/opt/sdm/lib/lx-amd64
>>>>
>> -
>>
>>> Djava.endorsed.dirs=/opt/sdm/lib/ext/endorsed -
>>> Dcom.sun.grid.grm.management.connectionTimeout=20 -jar /opt/sdm/lib/sdm-
>>> starter.jar com.sun.grid.grm.cli.SdmAdm -d -s sdm62u3 suj
>>>
>>>> jvm         host            result message
>>>> -----------------------------------------------------------------
>>>> cs_vm       llgriddev.local ERROR  JVM: cs_vm died during
>>>>                                    startup.
>>>> com.sun.grid.grm.GrmException: JVM: cs_vm died during startup.
>>>>         at
>>>>
>> com.sun.grid.grm.bootstrap.ParentStartupService.start(Unknown
>>
>>> Source)
>>>
>>>>         at
>>>>
>> com.sun.grid.grm.ui.component.StartJVMCommand.execute(Unknown
>>
>>> Source)
>>>
>>>>         at com.sun.grid.grm.ui.impl.CommandServiceImpl.execute(Unknown
>>>>
>>> Source)
>>>
>>>>         at
>>>>
>>> com.sun.grid.grm.cli.cmd.components.StartJVMCliCommand.execute(Unknown
>>> Source)
>>>
>>>>         at com.sun.grid.grm.cli.AbstractCli.run(Unknown Source)
>>>>         at com.sun.grid.grm.cli.SdmAdm.main(Unknown Source)
>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>         at
>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>>
>>> 39)
>>>
>>>>         at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>>
>>> pl.java:25)
>>>
>>>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>>>         at
>>>>
>> com.sun.grid.grm.util.MainWrapper$SystemRunThread.run(Unknown
>>
>>> Source)
>>>
>>>> executor_vm llgriddev.local ERROR  Timeout. Pid file:
>>>>                                    /var/spool/sdm/sdm62u3/run/exe
>>>>                                    cutor_vm at llgriddev.local for
>>>>                                    JVM: executor_vm not found.
>>>> com.sun.grid.grm.GrmException: Timeout. Pid file:
>>>>
>>> /var/spool/sdm/sdm62u3/run/executor_vm at llgriddev.local for JVM:
>>> executor_vm not found.
>>>
>>>>         at
>>>>
>> com.sun.grid.grm.bootstrap.ParentStartupService.start(Unknown
>>
>>> Source)
>>>
>>>>         at
>>>>
>> com.sun.grid.grm.ui.component.StartJVMCommand.execute(Unknown
>>
>>> Source)
>>>
>>>>         at com.sun.grid.grm.ui.impl.CommandServiceImpl.execute(Unknown
>>>>
>>> Source)
>>>
>>>>         at
>>>>
>>> com.sun.grid.grm.cli.cmd.components.StartJVMCliCommand.execute(Unknown
>>> Source)
>>>
>>>>         at com.sun.grid.grm.cli.AbstractCli.run(Unknown Source)
>>>>         at com.sun.grid.grm.cli.SdmAdm.main(Unknown Source)
>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>         at
>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>>
>>> 39)
>>>
>>>>         at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>>
>>> pl.java:25)
>>>
>>>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>>>         at
>>>>
>> com.sun.grid.grm.util.MainWrapper$SystemRunThread.run(Unknown
>>
>>> Source)
>>>
>>>> rp_vm       llgriddev.local ERROR  JVM: rp_vm died during
>>>>                                    startup.
>>>> com.sun.grid.grm.GrmException: JVM: rp_vm died during startup.
>>>>         at
>>>>
>> com.sun.grid.grm.bootstrap.ParentStartupService.start(Unknown
>>
>>> Source)
>>>
>>>>         at
>>>>
>> com.sun.grid.grm.ui.component.StartJVMCommand.execute(Unknown
>>
>>> Source)
>>>
>>>>         at com.sun.grid.grm.ui.impl.CommandServiceImpl.execute(Unknown
>>>>
>>> Source)
>>>
>>>>         at
>>>>
>>> com.sun.grid.grm.cli.cmd.components.StartJVMCliCommand.execute(Unknown
>>> Source)
>>>
>>>>         at com.sun.grid.grm.cli.AbstractCli.run(Unknown Source)
>>>>         at com.sun.grid.grm.cli.SdmAdm.main(Unknown Source)
>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>         at
>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>>
>>> 39)
>>>
>>>>         at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>>
>>> pl.java:25)
>>>
>>>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>>>         at
>>>>
>> com.sun.grid.grm.util.MainWrapper$SystemRunThread.run(Unknown
>>
>>> Source)
>>>
>>>> Error: com.sun.grid.grm.GrmException: Command has generated error.
>>>>         at
>>>>
>>> com.sun.grid.grm.cli.cmd.components.StartJVMCliCommand.execute(Unknown
>>> Source)
>>>
>>>>         at com.sun.grid.grm.cli.AbstractCli.run(Unknown Source)
>>>>         at com.sun.grid.grm.cli.SdmAdm.main(Unknown Source)
>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>         at
>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>>
>>> 39)
>>>
>>>>         at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>>
>>> pl.java:25)
>>>
>>>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>>>         at
>>>>
>> com.sun.grid.grm.util.MainWrapper$SystemRunThread.run(Unknown
>>
>>> Source)
>>>
>>>> + exit 2
>>>>
>>>>
>>>> Also, I changed the logging level but I don't get cs_vm log.
>>>>
>>>> Here is what I have in the logging.properties:
>>>>
>>>> # Default global logging level.
>>>> # Loggers and Handlers may override this level
>>>> .level=FINE
>>>> javax.level=SEVERE
>>>>
>>>> com.sun.grid.grm.service.level=FINE
>>>>
>>>> # --- grid.grm logging level ---
>>>> com.sun.grid.grm.level=FINE
>>>>
>>>> # --- FileHandler ---
>>>> java.util.logging.FileHandler.level=ALL
>>>>
>>>>
>> java.util.logging.FileHandler.formatter=com.sun.grid.grm.util.GrmFormatter
>>
>>>> java.util.logging.FileHandler.pattern=${log_dir}/${jvm_name}-%g.log
>>>> java.util.logging.FileHandler.append=true
>>>> <cure here>
>>>>
>>>> # find /var/spool/sdm/sdm62u3/log -ls
>>>> 4406418    4 drwxr-xr-x   2 sge      root         4096 Jun 30 11:24
>>>>
>>> /var/spool/sdm/sdm62u3/log
>>>
>>>> 4406510    0 -rw-r--r--   1 root     root            0 Jun 30 11:09
>>>>
>>> /var/spool/sdm/sdm62u3/log/executor_vm.stderr
>>>
>>>> 4406511    0 -rw-r--r--   1 root     root            0 Jun 30 11:09
>>>>
>>> /var/spool/sdm/sdm62u3/log/executor_vm-0.log.lck
>>>
>>>> 4406516    4 -rw-r--r--   1 root     root          330 Jun 30 11:08
>>>>
>>> /var/spool/sdm/sdm62u3/log/executor_vm-0.log.1
>>>
>>>> 4406515    0 -rw-r--r--   1 root     root            0 Jun 30 11:07
>>>>
>>> /var/spool/sdm/sdm62u3/log/executor_vm-0.log.1.lck
>>>
>>>> 4406512   96 -rw-r--r--   1 root     root        90359 Jun 30 11:10
>>>>
>>> /var/spool/sdm/sdm62u3/log/executor_vm-0.log
>>>
>>>> 4406505    0 -rw-r--r--   1 root     root            0 Jun 30 11:09
>>>>
>>> /var/spool/sdm/sdm62u3/log/executor_vm.stdout
>>>
>>>> There is not much valuable information in executor_vm-0.log. It is
>>>>
>>> waiting for cs_vm and failed.
>>>
>>>> Any further suggestions are welcome.
>>>>
>>>> Thanks,
>>>> - Chansup
>>>>
>>>>
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Michal.Bachorik at sun.com [mailto:Michal.Bachorik at sun.com]
>>>>> Sent: Tuesday, June 30, 2009 10:06 AM
>>>>> To: users at gridengine.sunsource.net
>>>>> Cc: Byun, Chansup
>>>>> Subject: Re: [GE users] SDM 6.2U3 won't start
>>>>>
>>>>> also, you should increase the logging level (in
>>>>> <sdp_spool>/logging.properties).
>>>>>
>>>>> regards,
>>>>>
>>>>> m.
>>>>>
>>>>> easymf wrote:
>>>>>
>>>>>> chansup,
>>>>>>
>>>>>>
>>>>>> try to start it with "sdmadm -d -s sdm62u3 suj". it will print a
>>>>>> stacktrace .. and also, look at the log files in sdm spool dir.
>>>>>> as all of your jvms has "died" (refused to start), it seems to be a
>>>>>> problem with installation or configuration.
>>>>>>
>>>>>> regards,
>>>>>>
>>>>>> michal
>>>>>>
>>>>>> cbyun wrote:
>>>>>>
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I just installed SDM from SGE 6.2u3 release without any errors.
>>>>>>> Then, I tried to start JVMs but none of them started with the
>>>>>>>
>>> following
>>>
>>>>> error:
>>>>>
>>>>>>> # sdmadm -s sdm62u3 suj
>>>>>>> jvm         host            result message
>>>>>>> -----------------------------------------------------------------
>>>>>>> cs_vm       llgriddev.local ERROR  JVM: cs_vm died during
>>>>>>>                                    startup.
>>>>>>> executor_vm llgriddev.local ERROR  Timeout. Pid file:
>>>>>>>                                    /var/spool/sdm/sdm62u3/run/exe
>>>>>>>                                    cutor_vm at llgriddev.local for
>>>>>>>                                    JVM: executor_vm not found.
>>>>>>> rp_vm       llgriddev.local ERROR  JVM: rp_vm died during
>>>>>>>                                    startup.
>>>>>>> Error: Command has generated error.
>>>>>>>
>>>>>>> I used Java 1.6.
>>>>>>>
>>>>>>> # echo $JAVA_HOME
>>>>>>> /usr/java/latest
>>>>>>>
>>>>>>> # /usr/java/latest/bin/java -version
>>>>>>> java version "1.6.0_13"
>>>>>>> Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
>>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 11.3-b02, mixed mode)
>>>>>>>
>>>>>>> Any suggestions for further troubleshooting?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> - Chansup
>>>>>>>
>>>>>>> ------------------------------------------------------
>>>>>>>
>>>>>>>
>> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId
>>
>>>>> =204401
>>>>>
>>>>>>> To unsubscribe from this discussion, e-mail: [users-
>>>>>>>
>>>>> unsubscribe at gridengine.sunsource.net].
>>>>>
>>>>>> ------------------------------------------------------
>>>>>>
>>>>>>
>> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId
>>
>>>>> =204552
>>>>>
>>>>>> To unsubscribe from this discussion, e-mail: [users-
>>>>>>
>>>>> unsubscribe at gridengine.sunsource.net].
>>>>> ------------------------------------------------------
>>>>>
>>>>>
>> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId
>>
>>>>> =204605
>>>>>
>>>>> To unsubscribe from this discussion, e-mail: [users-
>>>>> unsubscribe at gridengine.sunsource.net].
>>>>>
>>>> ------------------------------------------------------
>>>>
>>>>
>> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId
>>
>>> =204625
>>>
>>>> To unsubscribe from this discussion, e-mail: [users-
>>>>
>>> unsubscribe at gridengine.sunsource.net].
>>>
>>>
>>> --
>>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>
>> -
>>
>>> -
>>> Richard Hierlmeier           Phone: ++49 (0)941 3075-223
>>> Software Engineering         Fax:   ++49 (0)941 3075-222
>>> Sun Microsystems GmbH
>>> Dr.-Leo-Ritter-Str. 7      mailto: richard.hierlmeier at sun.com
>>> D-93049 Regensburg           http://www.sun.com/grid
>>>
>>> Sitz der Gesellschaft:
>>> Sun Microsystems GmbH, Sonnenallee 1, D-85551 Kirchheim-Heimstetten
>>> Amtsgericht Muenchen: HRB 161028
>>> Geschaeftsfuehrer: Thomas Schroeder, Wolfgang Engels, Dr. Roland Boemer
>>> Vorsitzender des Aufsichtsrates: Martin Haering
>>>
>>> ------------------------------------------------------
>>>
>>>
>> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId
>>
>>> =204763
>>>
>>> To unsubscribe from this discussion, e-mail: [users-
>>> unsubscribe at gridengine.sunsource.net].
>>>
>> ------------------------------------------------------
>> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId
>> =204803
>>
>> To unsubscribe from this discussion, e-mail: [users-
>> unsubscribe at gridengine.sunsource.net].
>>
>
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=204807
>
> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].
>

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=204810

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list