[GE users] dbwriter errors

weicfa awei1231 at yahoo.com.tw
Thu Nov 26 10:57:20 GMT 2009


I have the same problem too. My dbwriter run on solaris 10 , with 10G ram ,sge 6.2u1, mysql5.1.34.

I install a new ARCO from 2009/07  and it run well  until the end of Oct.   It show the same messages when dbwriter crash.
 I had tried the heap space by setting the environment value JVMARGS e.g. to "-Xmx256M" or "-Xmx512" or  "-Xmx1024" before calling the start script. But the dbwriter still crash.  
 I run dbwriter with "-debug" option and got the error messgages from console:
---------------------------------------------
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  SIGSEGV (0xb) at pc=0xffffffff7e5d873c, pid=29336, tid=17
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_07-b03 mixed mode)
# Problematic frame:
# V  [libjvm.so+0x5d873c]
#
# An error report file with more information is saved as hs_err_pid29336.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#
Abort - core dumped


--------------------------------------------


And here are the content of hs_err_pid29336.log:

--------------------------------------------

#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  SIGSEGV (0xb) at pc=0xffffffff7e5d873c, pid=29336, tid=17
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_07-b03 mixed mode)
# Problematic frame:
# V  [libjvm.so+0x5d873c]
#

---------------  T H R E A D  ---------------

Current thread (0x00000001009003a0):  JavaThread "Poller SunPKCS11-Solaris" daemon [_thread_in_vm, id=17]

siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x0000000000000000

Registers:
 O0=0x0000000000000000 O1=0x000000000000f1f8 O2=0x0000000000000074 O3=0x0000000080000000
 O4=0x0000000000000006 O5=0x0000000000001ffc O6=0xffffffff4acfe241 O7=0x0000000000000000
 G1=0x0000000000012c00 G2=0x0000000000012800 G3=0x0000000000000000 G4=0x0000000000000040
 G5=0x000000000000f000 G6=0x0000000000000000 G7=0xffffffff7f508200 Y=0x0000000000000000
 PC=0xffffffff7e5d873c nPC=0xffffffff7e5d8740


Top of Stack: (sp=0xffffffff4acfea40)
0xffffffff4acfea40:   0000000000000001 ffffffff7eaafb10
0xffffffff4acfea50:   ffffffff7ea0e000 0000000000010938
0xffffffff4acfea60:   ffffffff7eaafb08 0000000000010800
0xffffffff4acfea70:   0000000000012e90 0000000000012a80
0xffffffff4acfea80:   ffffffff7ea9af20 00000001009003a0
0xffffffff4acfea90:   0000000000000000 0000000000000040
0xffffffff4acfeaa0:   ffffffff7eab65c8 0000000000000003
0xffffffff4acfeab0:   ffffffff4acfe311 ffffffff4b40ca54
0xffffffff4acfeac0:   ffffffff4acfe311 ffffffff4b40ca30
0xffffffff4acfead0:   0000000000000002 0000000000012c00
0xffffffff4acfeae0:   ffffffff7ea0e000 0000000000000000
0xffffffff4acfeaf0:   0000000100b8f4d0 0000000000000000
0xffffffff4acfeb00:   0000004000001ffc ffffffff7eab0da4
0xffffffff4acfeb10:   0000000000000074 00000001009003a0
0xffffffff4acfeb20:   0000000000000040 0000000100b8f4d0
0xffffffff4acfeb30:   00000001009004e8 000000000000007e 

Instructions: (pc=0xffffffff7e5d873c)
0xffffffff7e5d872c:   80 a1 20 00 06 40 00 4e de 5b a8 b7 b6 06 80 04
0xffffffff7e5d873c:   d4 5b e0 00 da 02 a0 10 80 a6 c0 0d 18 40 00 48 

Stack: [0xffffffff4ac00000,0xffffffff4ad00000),  sp=0xffffffff4acfea40,  free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x5d873c]
C  [libj2pkcs11.so+0xca5c]
C  [libj2pkcs11.so+0x7780]
C  [libj2pkcs11.so+0x76a8]
j  sun.security.pkcs11.wrapper.PKCS11.C_GetSlotInfo(J)Lsun/security/pkcs11/wrapper/CK_SLOT_INFO;+774366328
j  sun.security.pkcs11.wrapper.PKCS11.C_GetSlotInfo(J)Lsun/security/pkcs11/wrapper/CK_SLOT_INFO;+0
j  sun.security.pkcs11.SunPKCS11.initToken(Lsun/security/pkcs11/wrapper/CK_SLOT_INFO;)V+12
v  ~C2IAdapter
J  sun.security.pkcs11.SunPKCS11$TokenPoller.run()V
v  ~OSRAdapter
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
V  [libjvm.so+0x29c188]
V  [libjvm.so+0x3d03b4]
V  [libjvm.so+0x3eece4]
V  [libjvm.so+0x3e9204]
V  [libjvm.so+0x7a9bd0]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  sun.security.pkcs11.wrapper.PKCS11.C_GetSlotInfo(J)Lsun/security/pkcs11/wrapper/CK_SLOT_INFO;+0
j  sun.security.pkcs11.SunPKCS11.initToken(Lsun/security/pkcs11/wrapper/CK_SLOT_INFO;)V+12
v  ~C2IAdapter
J  sun.security.pkcs11.SunPKCS11$TokenPoller.run()V
v  ~OSRAdapter
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x00000001007478e0 JavaThread "statistic" [_thread_blocked, id=20]
  0x00000001007e09f0 JavaThread "derived" [_thread_blocked, id=18]
=>0x00000001009003a0 JavaThread "Poller SunPKCS11-Solaris" daemon [_thread_in_vm, id=17]
  0x0000000100625d50 JavaThread "MySQL Statement Cancellation Timer" daemon [_thread_blocked, id=16]
  0x000000010090e4c0 JavaThread "dbwriter" [_thread_blocked, id=15]
  0x000000010029d2a0 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=13]
  0x000000010029c220 JavaThread "CompilerThread1" daemon [_thread_blocked, id=12]
  0x000000010029b0c0 JavaThread "CompilerThread0" daemon [_thread_blocked, id=11]
  0x0000000100299f60 JavaThread "AdapterThread" daemon [_thread_blocked, id=10]
  0x0000000100298a10 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=9]
  0x0000000100297490 JavaThread "JDWP Event Helper Thread" daemon [_thread_blocked, id=8]
  0x0000000100295230 JavaThread "JDWP Transport Listener: dt_socket" daemon [_thread_in_native, id=7]
  0x0000000100274fc0 JavaThread "Finalizer" daemon [_thread_blocked, id=6]
  0x0000000100274760 JavaThread "Reference Handler" daemon [_thread_blocked, id=5]
  0x000000010011ee30 JavaThread "main" [_thread_blocked, id=1]

Other Threads:
  0x00000001002709f0 VMThread [id=4]
  0x0000000100196530 WatcherThread [id=14]

VM state:synchronizing (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x000000010011dff0/0x000000010011e050] Threads_lock - owner thread: 0x00000001002709f0

Heap
 PSYoungGen      total 84800K, used 75583K [0xffffffff6a800000, 0xffffffff70800000, 0xffffffff75400000)
  eden space 75584K, 99% used [0xffffffff6a800000,0xffffffff6f1cfff8,0xffffffff6f1d0000)
  from space 9216K, 0% used [0xffffffff6f1d0000,0xffffffff6f1d0000,0xffffffff6fad0000)
  to   space 9216K, 0% used [0xffffffff6ff00000,0xffffffff6ff00000,0xffffffff70800000)
 PSOldGen        total 352256K, used 352256K [0xffffffff55000000, 0xffffffff6a800000, 0xffffffff6a800000)
  object space 352256K, 100% used [0xffffffff55000000,0xffffffff6a800000,0xffffffff6a800000)
 PSPermGen       total 24576K, used 11169K [0xffffffff50000000, 0xffffffff51800000, 0xffffffff55000000)
  object space 24576K, 45% used [0xffffffff50000000,0xffffffff50ae8730,0xffffffff51800000)

Dynamic libraries:
0x0000000100000000 	/usr/jdk/instances/jdk1.5.0/bin/sparcv9/java
0xffffffff7f700000 	/lib/64/libthread.so.1
0xffffffff7f200000 	/lib/64/libdl.so.1
0xffffffff7ef00000 	/lib/64/libc.so.1
0xffffffff7ee00000 	/platform/SUNW,Sun-Fire-V445/lib/sparcv9/libc_psr.so.1
0xffffffff7e000000 	/usr/jdk/instances/jdk1.5.0/jre/lib/sparcv9/server/libjvm.so
0xffffffff7de00000 	/lib/64/libsocket.so.1
0xffffffff7ed00000 	/usr/lib/64/libsched.so.1
0xffffffff7dc00000 	/usr/lib/64/libCrun.so.1
0xffffffff7da00000 	/lib/64/libm.so.1
0xffffffff7d800000 	/lib/64/libnsl.so.1
0xffffffff7d600000 	/lib/64/libm.so.2
0xffffffff7d400000 	/lib/64/libscf.so.1
0xffffffff7d200000 	/lib/64/libdoor.so.1
0xffffffff7d000000 	/lib/64/libuutil.so.1
0xffffffff7ce00000 	/lib/64/libgen.so.1
0xffffffff7cc00000 	/lib/64/libmd.so.1
0xffffffff7ca00000 	/platform/SUNW,Sun-Fire-V445/lib/sparcv9/libmd_psr.so.1
0xffffffff7c800000 	/lib/64/libmp.so.2
0xffffffff7c600000 	/usr/jdk/instances/jdk1.5.0/jre/lib/sparcv9/native_threads/libhpi.so
0xffffffff7c400000 	/usr/jdk/instances/jdk1.5.0/jre/lib/sparcv9/libjdwp.so
0xffffffff7c200000 	/usr/jdk/instances/jdk1.5.0/jre/lib/sparcv9/libverify.so
0xffffffff7c000000 	/usr/jdk/instances/jdk1.5.0/jre/lib/sparcv9/libjava.so
0xffffffff7be00000 	/usr/jdk/instances/jdk1.5.0/jre/lib/sparcv9/libzip.so
0xffffffff4f600000 	/usr/jdk/instances/jdk1.5.0/jre/lib/sparcv9/libdt_socket.so
0xffffffff4c400000 	/usr/jdk/instances/jdk1.5.0/jre/lib/sparcv9/libnet.so
0xffffffff4c200000 	/usr/jdk/instances/jdk1.5.0/jre/lib/sparcv9/libnio.so
0xffffffff4c000000 	/lib/64/librt.so.1
0xffffffff4be00000 	/lib/64/libaio.so.1
0xffffffff4b900000 	/home/grid-home/lib/sol-sparc64/libjuti.so
0xffffffff4b400000 	/usr/jdk/instances/jdk1.5.0/jre/lib/sparcv9/libj2pkcs11.so
0xffffffff4b200000 	/usr/lib/sparcv9/libpkcs11.so
0xffffffff4b000000 	/usr/lib/64/libcryptoutil.so.1
0xffffffff4ae00000 	/usr/lib/security/64/pkcs11_softtoken.so

VM Arguments:
jvm_args: -Xmx512M -Xdebug -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 -Djava.library.path=/home/grid-home/lib/sol-sparc64
java_command: com/sun/grid/reporting/dbwriter/ReportingDBWriter -pid /home/grid-home/rdgrid/spool/dbwriter/dbwriter.pid -logfile /home/grid-home/rdgrid/spool/dbwriter/dbwriter.log
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/usr
PATH=/home/grid-home/RT/wrapper:/home/grid-home/bin/sol-sparc64:/oracle/mysql/mysql-5.1.34-solaris10-sparc-64bit/bin:/usr/sfw/bin:/usr/sfw/sbin:/opt/csw/bin:/usr/local/bin:/opt/coolstack/php5/bin:/usr/sbin:/usr/bin:/usr/ccs/bin:/usr/openwin/bin:/usr/dt/bin:/usr/platform/SUNW,Sun-Fire-V445/sbin:/opt/sun/bin:/etc/vx/bin:/opt/VRTS/bin:/etc/vx/bin:/opt/VRTSob/bin:/opt/VRTSvlic/bin:/opt/SUNWvts/bin:/opt/SUNWexplo/bin:/opt/SUNWsneep/bin:/opt/CTEact/bin
LD_LIBRARY_PATH=/usr/jdk/instances/jdk1.5.0/jre/lib/sparcv9/server:/usr/jdk/instances/jdk1.5.0/jre/lib/sparcv9:/usr/jdk/instances/jdk1.5.0/jre/../lib/sparcv9:/usr/openwin/lib
SHELL=/sbin/sh
HOSTTYPE=sun4
OSTYPE=solaris
MACHTYPE=sparc

Signal Handlers:
SIGSEGV: [libjvm.so+0x83f99c], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGBUS: [libjvm.so+0x83f99c], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGFPE: [libjvm.so+0x3713f8], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0x3713f8], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0x3713f8], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGHUP: [libjvm.so+0x7aaf00], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: [libjvm.so+0x7aaf00], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGTERM: [libjvm.so+0x7aaf00], sa_mask[0]=0xffbffeff, sa_flags=0x00000004


---------------  S Y S T E M  ---------------

OS:                       Solaris 10 11/06 s10s_u3wos_10 SPARC
           Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                           Assembled 14 November 2006

uname:SunOS 5.10 Generic_137111-08 sun4u  (T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 65536, AS infinity
load average:1.32 1.76 2.13

CPU:total 2 has_v8, has_v9, has_vis1, has_vis2, is_ultra3

Memory: 8k page, physical 10485760k(4473560k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (1.5.0_07-b03) for solaris-sparc, built on May  3 2006 01:15:53 by unknown with unknown Workshop:0x550
---------------------------------------------

You can see there are still about 4G free memory.  I double the problem was the  java process use too much memory for exceeding the  value of JVMARGS in startup script. 

#ps -ef|grep java
sgeadmin 27669 27662   0 16:09:06 pts/12 7:12 /usr/jdk/instances/jdk1.5.0/bin/sparcv9/java -Xmx1024M -server -Djava.library.p

I found the memory usage of the java process will grown up and I will run a script to monitor it . Just want to make sure if the  dbwriter will crash when the memory usage of java process reach 1024M. 
I will post the result later.

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=229509

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].

    [ Part 2, "hs_err_pid29336.log"  Application/OCTET-STREAM (Name: ]
    [ "hs_err_pid29336.log") 10 KB. ]
    [ Unable to print this part. ]



More information about the gridengine-users mailing list