[GE users] Queue dropped because it is full

Kirk Patton kpatton at transmeta.com
Sat Apr 24 03:20:33 BST 2004


    [ The following text is in the "unknown-8bit" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some characters may be displayed incorrectly. ]

Hello Andy,

We had another failure to schedule jobs.  We first noticed the issue
after a user submitted and then deleted about 4000 jobs.  After that,
not further scheduling took place.

I collected some output from several commands.

We may be able to repeate this issue.  I can ask the user to try.
Is there any other data I could collect that would be usefull in tracking this down?

Kirk


HOSTNAME             ARCH       NPROC  LOAD   MEMTOT   MEMUSE   SWAPTO   SWAPUS
-------------------------------------------------------------------------------
global               -              -     -        -        -        -        -
aslt1.transmeta.com  i686           2  1.13     2.0G   301.0M     1.9G   165.0M
aslt2.transmeta.com  i686           2  0.77     3.8G   822.1M     1.9G   181.9M
captain.transmeta.com i686           2  0.03  1006.6M   259.5M     2.0G    65.0M
chris-linux.transmeta.com i686           2  0.03  1006.6M   332.4M     2.0G   143.1M
dioxide.transmeta.com sun4u         12  0.12    12.0G     1.8G    17.9G      0.0
l0023001.transmeta.com i686           2  0.00  1006.6M    36.9M     2.0G    25.3M
l0023004.transmeta.com i686           2  0.00  1006.6M    42.2M     2.0G    25.2M
l0023013.transmeta.com i686           2  0.00  1006.6M    38.0M     2.0G    26.2M
l0023017.transmeta.com i686           2  0.00  1006.6M    46.1M     2.0G    27.9M
l0023018.transmeta.com i686           2  1.05  1006.6M   404.3M     2.0G   257.1M
l0023021.transmeta.com i686           2  0.00  1006.6M    40.1M     2.0G    28.7M
l0023037.transmeta.com i686           2  0.27  1006.6M    45.1M     2.0G    32.1M
l0023041.transmeta.com i686           2  0.02  1006.6M    43.3M     2.0G    19.8M
l0023046.transmeta.com i686           2  1.73  1006.6M   698.5M     2.0G   865.1M
op240-000.transmeta.com i686           2  0.00     3.9G   286.9M     7.9G      0.0
op240-001.transmeta.com i686           2  0.00     3.9G   315.3M     7.9G     4.0K
op240-002.transmeta.com i686           2  0.00     3.9G   308.3M     7.9G      0.0
op240-003.transmeta.com i686           2  1.03     3.9G     1.6G     7.9G    20.0K
op240-004.transmeta.com i686           2  0.00     3.9G   276.9M     7.9G     4.0K
op240-005.transmeta.com i686           2  0.00     3.9G   220.1M     7.9G      0.0
op240-006.transmeta.com i686           2  0.00     3.9G   333.0M     7.9G     8.0K
op240-007.transmeta.com i686           2  0.00     3.9G   372.8M     7.9G      0.0
op240-008.transmeta.com i686           2  0.00     3.9G   230.9M     7.9G      0.0
op240-009.transmeta.com i686           2  0.00     3.9G   228.7M     7.9G      0.0
op240-010.transmeta.com i686           2  0.00     3.9G   652.7M     7.9G   764.0K
op240-011.transmeta.com i686           2  0.00     3.9G   217.1M     7.9G      0.0
op240-012.transmeta.com i686           2  0.00     3.9G   211.8M     7.9G      0.0
op240-013.transmeta.com i686           2  0.00     3.9G   213.3M     7.9G      0.0
op240-014.transmeta.com i686           2  0.13     3.9G   214.3M     7.9G      0.0
op240-015.transmeta.com i686           2  0.00     3.9G   208.9M     7.9G      0.0
op240-016.transmeta.com i686           2  0.00     3.9G   209.5M     7.9G      0.0
op240-017.transmeta.com i686           2  0.00     3.9G   213.4M     7.9G      0.0
op240-018.transmeta.com i686           2  0.00     3.9G   205.5M     7.9G      0.0
op240-019.transmeta.com i686           2  0.01     3.9G   207.2M     7.9G      0.0
op240-020.transmeta.com i686           2  0.00     3.9G   211.5M     7.9G      0.0
op240-021.transmeta.com i686           2  0.00     3.9G   210.6M     7.9G      0.0
op240-022.transmeta.com i686           2  0.00     3.9G   272.5M     7.9G    20.0K
op240-023.transmeta.com i686           2  0.01     3.9G   203.5M     7.9G      0.0
op240-024.transmeta.com i686           2  0.01     3.9G   204.0M     7.9G      0.0
op240-025.transmeta.com i686           2  0.07     3.9G   202.2M     7.9G      0.0
op240-026.transmeta.com i686           2  0.00     3.9G   221.8M     7.9G      0.0
op240-027.transmeta.com i686           2  0.00     3.9G   209.8M     7.9G      0.0
op240-028.transmeta.com i686           2  0.01     3.9G   202.4M     7.9G      0.0
op240-029.transmeta.com i686           2  1.05     3.9G   806.8M     7.9G      0.0
opteron-test.transmeta.com i686           2  0.01     3.9G   705.7M     7.9G    18.8M
ruthenium.transmeta.com sun4u          2  0.26     2.0G   472.0M   512.0M      0.0
technetium.transmeta.com sun4u          2  0.27     1.0G   141.0M     1.2G      0.0


job_number:                 10542
submission_time:            Fri Apr 23 16:07:50 2004
owner:                      teesea3d
uid:                        1719
group:                      teesea3d
gid:                        1719
sge_o_home:                 /home/teesea3d
sge_o_path:                 /usr/bin:/bin
sge_o_shell:                /bin/bash
sge_o_workdir:              /export/home/tmp
sge_o_host:                 teese-1
account:                    sge
cwd:                        /export/home/tmp
path_aliases:               /tmp_mnt/ * * /
hard resource_list:         priority=common,os=Linux
mail_list:                  teesea3d at teese-1.transmeta.com
notify:                     FALSE
job_name:                   teesejob
restart:                    n
env_list:                   PWD=/export/home/tmp,USER=teesea3d,SGE_ROOT=/transmeta/sge,SHLVL=2,_=/transmeta/sge/bin/glinux/qrsh,SHELL=/bin/bash,HOME=/home/teesea3d,PATH=/usr/bin:/bin,QRSH_PORT=teese-1.transmeta.com:35029,QRSH_COMMAND=/project/teese/sbin/teese-slave.pl?--nslaves?9?--sim?sim?--timeout?64800?--sim-mode?translator-heavy?--gdbpath?../astro-gdb?--bundle-server?http://teese-1:12008//?--netport?teese-1/12003?--netport?teese-1/12004?--netport?teese-1/12005?--netport?teese-1/12006?--netport?teese-1/12007?--netport?teese-1/12000?--netport?teese-1/12001?--netport?teese-1/12002
project:                    teese_normal
context:                    SGE_INTERACTIVE=yes
scheduling info:            queue "trivial.aslt2" dropped because it is temporarily not available
                            queue "trivial.opteron-test" dropped because it is temporarily not available
                            queue "trivial.op240-015" dropped because it is temporarily not available
                            queue "trivial.op240-004" dropped because it is temporarily not available
                            queue "trivial.op240-028" dropped because it is temporarily not available
                            queue "trivial.op240-022" dropped because it is temporarily not available
                            queue "trivial.op240-024" dropped because it is temporarily not available
                            queue "trivial.op240-005" dropped because it is temporarily not available
                            queue "trivial.op240-014" dropped because it is temporarily not available
                            queue "trivial.op240-023" dropped because it is temporarily not available
                            queue "trivial.op240-010" dropped because it is temporarily not available
                            queue "trivial.op240-007" dropped because it is temporarily not available
                            queue "trivial.op240-006" dropped because it is temporarily not available
                            queue "trivial.op240-009" dropped because it is temporarily not available
                            queue "trivial.op240-003" dropped because it is temporarily not available
                            queue "trivial.op240-029" dropped because it is temporarily not available
                            queue "trivial.op240-017" dropped because it is temporarily not available
                            queue "trivial.op240-012" dropped because it is temporarily not available
                            queue "trivial.op240-001" dropped because it is temporarily not available
                            queue "trivial.l0023017" dropped because it is temporarily not available
                            queue "trivial.op240-018" dropped because it is temporarily not available
                            queue "trivial.op240-019" dropped because it is temporarily not available
                            queue "trivial.l0023018" dropped because it is temporarily not available
                            queue "trivial.l0023041" dropped because it is temporarily not available
                            queue "trivial.l0023013" dropped because it is temporarily not available
                            queue "trivial.l0023001" dropped because it is temporarily not available
                            queue "trivial.aslt1" dropped because it is temporarily not available
                            queue "trivial.op240-008" dropped because it is temporarily not available
                            queue "trivial.l0023021" dropped because it is temporarily not available
                            queue "trivial.l0023037" dropped because it is temporarily not available
                            queue "trivial.l0023046" dropped because it is temporarily not available
                            queue "trivial.op240-025" dropped because it is temporarily not available
                            queue "trivial.op240-000" dropped because it is temporarily not available
                            queue "trivial.op240-013" dropped because it is temporarily not available
                            queue "common.op240-001" dropped because it is full
                            queue "common.op240-014" dropped because it is full
                            queue "common.op240-009" dropped because it is full
                            queue "common.op240-019" dropped because it is full
                            queue "common.op240-018" dropped because it is full
                            queue "common.op240-005" dropped because it is full
                            queue "common.op240-024" dropped because it is full
                            queue "common.aslt2" dropped because it is full
                            queue "common.op240-029" dropped because it is full
                            queue "common.op240-012" dropped because it is full
                            queue "common.op240-022" dropped because it is full
                            queue "common.op240-015" dropped because it is full
                            queue "common.l0023018" dropped because it is full
                            queue "common.l0023041" dropped because it is full
                            queue "common.l0023017" dropped because it is full
                            queue "common.l0023013" dropped because it is full
                            queue "common.op240-006" dropped because it is full
                            queue "common.aslt1" dropped because it is full
                            queue "common.op240-028" dropped because it is full
                            queue "common.op240-007" dropped because it is full
                            queue "common.op240-023" dropped because it is full
                            queue "common.op240-004" dropped because it is full
                            queue "common.op240-003" dropped because it is full
                            queue "common.opteron-test" dropped because it is full
                            queue "common.op240-010" dropped because it is full
                            queue "common.op240-017" dropped because it is full
                            queue "common.l0023001" dropped because it is full
                            queue "common.op240-008" dropped because it is full
                            queue "common.l0023021" dropped because it is full
                            queue "common.l0023037" dropped because it is full
                            queue "common.l0023046" dropped because it is full
                            queue "common.op240-025" dropped because it is full
                            queue "common.op240-000" dropped because it is full
                            queue "common.op240-013" dropped because it is full


queuename            qtype used/tot. load_avg arch      states
----------------------------------------------------------------------------
critical.op240-000   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-001   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-002   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-003   BIP   0/2       1.02     i686      
----------------------------------------------------------------------------
critical.op240-004   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-005   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-006   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-007   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-008   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-009   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-010   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-011   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-012   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-013   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-014   BIP   0/2       0.11     i686      
----------------------------------------------------------------------------
critical.op240-015   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-016   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-017   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-018   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-019   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-020   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-021   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-022   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-023   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-024   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-025   BIP   0/2       0.23     i686      
----------------------------------------------------------------------------
critical.op240-026   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-027   BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.op240-028   BIP   0/2       0.01     i686      
----------------------------------------------------------------------------
critical.op240-029   BIP   0/2       1.04     i686      
----------------------------------------------------------------------------
critical.aslt1       BIP   0/2       1.11     i686      
----------------------------------------------------------------------------
critical.aslt2       BIP   0/2       0.76     i686      
----------------------------------------------------------------------------
critical.opteron-tes BIP   0/2       0.01     i686      
----------------------------------------------------------------------------
critical.l0023001    BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.l0023004    BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.l0023013    BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.l0023017    BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.l0023018    BIP   0/2       1.04     i686      
----------------------------------------------------------------------------
critical.l0023021    BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
critical.l0023037    BIP   0/2       0.23     i686      
----------------------------------------------------------------------------
critical.l0023041    BIP   0/2       0.02     i686      
----------------------------------------------------------------------------
critical.l0023046    BIP   0/2       1.76     i686      
----------------------------------------------------------------------------
common.op240-000     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-001     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-002     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-003     BIP   1/2       1.02     i686      
   6206     0 run_cmd_sc gurmeet      r     04/22/2004 18:02:44 MASTER         
----------------------------------------------------------------------------
common.op240-004     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-005     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-006     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-007     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-008     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-009     BIP   1/2       0.00     i686      
    540     0 QLOGIN     justin       r     04/14/2004 11:46:34 MASTER         
----------------------------------------------------------------------------
common.op240-010     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-011     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-012     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-013     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-014     BIP   0/2       0.11     i686      
----------------------------------------------------------------------------
common.op240-015     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-016     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-017     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-018     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-019     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-020     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-021     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-022     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-023     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-024     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-025     BIP   0/2       0.23     i686      
----------------------------------------------------------------------------
common.op240-026     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-027     BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.op240-028     BIP   0/2       0.01     i686      
----------------------------------------------------------------------------
common.op240-029     BIP   1/2       1.04     i686      
   7071     0 run_cmd_it gurmeet      r     04/23/2004 12:05:29 MASTER         
----------------------------------------------------------------------------
critical.ruthenium   BIP   0/2       0.26     sun4u     
----------------------------------------------------------------------------
critical.technetium  BIP   0/2       0.27     sun4u     
----------------------------------------------------------------------------
common.aslt1         BIP   0/2       1.11     i686      
----------------------------------------------------------------------------
common.aslt2         BIP   0/2       0.76     i686      
----------------------------------------------------------------------------
common.opteron-test  BIP   0/2       0.01     i686      
----------------------------------------------------------------------------
common.l0023001      BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.l0023004      BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.l0023013      BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.l0023017      BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.l0023018      BIP   1/2       1.04     i686      
   6992     0 da_tag_mas cwang        r     04/23/2004 10:51:57 MASTER         
----------------------------------------------------------------------------
common.l0023021      BIP   0/2       0.00     i686      
----------------------------------------------------------------------------
common.l0023037      BIP   0/2       0.23     i686      
----------------------------------------------------------------------------
common.l0023041      BIP   0/2       0.02     i686      
----------------------------------------------------------------------------
common.l0023046      BIP   0/2       1.76     i686      
----------------------------------------------------------------------------
trivial.op240-000    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-001    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-002    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-003    BICP  0/2       1.02     i686      
----------------------------------------------------------------------------
trivial.op240-004    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-005    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-006    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-007    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-008    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-009    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-010    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-011    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-012    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-013    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-014    BICP  0/2       0.11     i686      
----------------------------------------------------------------------------
trivial.op240-015    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-016    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-017    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-018    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-019    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-020    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-021    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-022    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-023    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-024    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-025    BICP  0/2       0.23     i686      
----------------------------------------------------------------------------
trivial.op240-026    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-027    BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.op240-028    BICP  0/2       0.01     i686      
----------------------------------------------------------------------------
trivial.op240-029    BICP  0/2       1.04     i686      
----------------------------------------------------------------------------
common.ruthenium     BIP   0/2       0.26     sun4u     
----------------------------------------------------------------------------
common.technetium    BIP   0/2       0.27     sun4u     
----------------------------------------------------------------------------
trivial.aslt1        BICP  0/2       1.11     i686      
----------------------------------------------------------------------------
trivial.aslt2        BICP  0/2       0.76     i686      
----------------------------------------------------------------------------
trivial.opteron-test BICP  0/2       0.01     i686      
----------------------------------------------------------------------------
trivial.l0023001     BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.l0023004     BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.l0023013     BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.l0023017     BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.l0023018     BICP  0/2       1.04     i686      
----------------------------------------------------------------------------
trivial.l0023021     BICP  0/2       0.00     i686      
----------------------------------------------------------------------------
trivial.l0023037     BICP  0/2       0.23     i686      
----------------------------------------------------------------------------
trivial.l0023041     BICP  0/2       0.02     i686      
----------------------------------------------------------------------------
trivial.l0023046     BICP  0/2       1.76     i686      
----------------------------------------------------------------------------
trivial.ruthenium    BICP  0/2       0.26     sun4u     
----------------------------------------------------------------------------
trivial.technetium   BICP  0/2       0.27     sun4u     

############################################################################
 - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS
############################################################################
   7764     0 teesejob   teesea3d     qw    04/23/2004 14:22:53                
   7771     0 teesejob   teesea3d     qw    04/23/2004 14:23:13                
   7777     0 teesejob   teesea3d     qw    04/23/2004 14:23:28                
   7788     0 teesejob   teesea3d     qw    04/23/2004 14:23:58                
   7797     0 teesejob   teesea3d     qw    04/23/2004 14:24:18                
   7800     0 teesejob   teesea3d     qw    04/23/2004 14:24:28                
   7801     0 teesejob   teesea3d     qw    04/23/2004 14:24:28                
   9694     0 hostname   kpatton      qw    04/23/2004 15:37:23                
  10101     0 teesejob   teesea3d     qw    04/23/2004 15:51:57                
  10194     0 run_cmd_fl gurmeet      qw    04/23/2004 15:55:23                
  10278     0 run_cmd_iu gurmeet      qw    04/23/2004 15:58:19                
  10540     0 teesejob   teesea3d     qw    04/23/2004 16:07:50                
  10541     0 teesejob   teesea3d     qw    04/23/2004 16:07:50                
  10542     0 teesejob   teesea3d     qw    04/23/2004 16:07:50                
  10543     0 teesejob   teesea3d     qw    04/23/2004 16:07:50                
  10544     0 teesejob   teesea3d     qw    04/23/2004 16:07:50                
  10545     0 teesejob   teesea3d     qw    04/23/2004 16:07:50                
  10546     0 teesejob   teesea3d     qw    04/23/2004 16:07:50                
  10547     0 teesejob   teesea3d     qw    04/23/2004 16:07:51                
  10642     0 run_cmd_pc gurmeet      qw    04/23/2004 16:11:11                
  11510     0 date       gsmith       qw    04/23/2004 16:42:31                
  11521     0 make       chris        qw    04/23/2004 17:46:56                
  11525     0 da_tag_mas cwang        qw    04/23/2004 18:12:03                
  11526     0 teesejob   teesec60d    qw    04/23/2004 18:14:08                
  11527     0 teesejob   teesec60d    qw    04/23/2004 18:14:08                
  11528     0 teesejob   teesec60d    qw    04/23/2004 18:14:08                
  11529     0 teesejob   teesec60d    qw    04/23/2004 18:14:08                
  11530     0 teesejob   teesec60d    qw    04/23/2004 18:14:08                
  11531     0 teesejob   teesec60d    qw    04/23/2004 18:14:08                
  11532     0 teesejob   teesec60d    qw    04/23/2004 18:14:08                
  11533     0 teesejob   teesec60d    qw    04/23/2004 18:14:08                
  11539     0 teesejob   teeseastro   qw    04/23/2004 18:32:24                
  11540     0 teesejob   teeseastro   qw    04/23/2004 18:32:57                
  11541     0 teesejob   teeseastro   qw    04/23/2004 18:33:02                
  11542     0 teesejob   teeseastro   qw    04/23/2004 18:33:41                
  11543     0 teesejob   teeseastro   qw    04/23/2004 18:33:46                
  11544     0 teesejob   teeseastro   qw    04/23/2004 18:33:58                
  11545     0 teesejob   teeseastro   qw    04/23/2004 18:34:06                


qstat -f -l -q common.op240-012
queuename            qtype used/tot. load_avg arch      states
----------------------------------------------------------------------------
common.op240-012     BIP   0/2       0.00     i686



On Tue, Apr 20, 2004 at 07:12:15AM -0700, Kirk Patton wrote:
> Hello Andy,
> 
> qstat -j reported the queues were full.  I do not have the output. I will be sure to save if should 
> it happen again.  Since the cluster was restarted successfully, I am not able to try submiting with the
> -now option.  The cluster had been up for over a month.
> 
> Are you interested in any of the log files?
> 
> 
> Kirk
> On Tue, Apr 20, 2004 at 03:49:12PM +0200, Andy Schwierskott wrote:
> > Kirk,
> > 
> >   - was does "qstat -f -l q=common.op240-012" show?
> >   - what does "qstat -j" show?
> >   - what happens of the job is submitted with the "-now n" option?
> > 
> > Andy
> > 
> > > Hello,
> > >
> > > I recently experienced jobs hanging with SGE 5.3.p5.
> > >
> > > Jobs submitted to the cluster were pending with the message
> > > queue "common.op240-012" dropped because it is full...
> > >
> > > There were only 4 jobs reported as submitted in the system.
> > > :>qstat
> > > job-ID  prior name       user         state submit/start at     queue      maste
> > > r  ja-task-ID
> > > --------------------------------------------------------------------------------
> > > -------------
> > >     540     0 QLOGIN     justin       r     04/14/2004 11:46:34 common.op2 MASTE
> > > R
> > >    3003     0 VOLDEMORT_ jayanto      qw    04/19/2004 17:11:02
> > >
> > >    3006     0 ls         gsmith       qw    04/19/2004 17:43:59
> > >
> > >    3011     0 who        kpatton      qw    04/20/2004 06:21:34
> > >
> > > I look for similiar issues in the mailing list and came accross a post by
> > > Ron Koester
> > > >Date: Wed, 07 May 2003 09:10:04 -0400
> > > >From: Ron Koester <koester at carbondesignsystems.com>
> > > >Content-Type: text/plain; charset=us-ascii
> > > >Subject: Re: [GE users] queue dropped because it is full
> > > >
> > > >
> > > >Thanks for everyone's ideas -
> > > >
> > > >This morning I resorted to stopping/restarting the daemons on
> > > >the SGE master -- and... -- the problem went away.
> > > >
> > > >I did do a 'qstat -f' prior to restarting the master daemons,
> > > >and the reported load for the two malfunctioning queues was
> > > >correct.
> > > >
> > > >One other clue I got, if anyone cares at this point, is that
> > > >when I did './rcsge stop' on the SGE master, the schedd wasn't
> > > >stopped.  Doing it again still didn't kill off the schedd.  So
> > > >I ended up killing the schedd with 'kill -9'.
> > > >
> > > >Ron
> > >
> > > This is the same exact behavior I noted with my cluster.  I shutdown the
> > > master node and had to kill -9 sge_sched as it was running at 98% of the cpu.
> > >
> > > Restarting the master brought the system back on line.
> > >
> > > The acting master at the time of the incident was a Opteron based linux box running
> > > 2.4.24 kernel.
> > >
> > > When the daemons came back up, there were 51 messages like the following
> > > removing reference to no longer existing job of user "gsmith".
> > >
> > > Does anyone know what could have caused SGE to stop processing jobs?
> > >
> > > Thanks,
> > > Kirk
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> > For additional commands, e-mail: users-help at gridengine.sunsource.net
> > 
> 
> -- 
> Kirk Patton
> Unix Administrator
> Transmeta Inc.
> Tel. 408 919-3055

-- 
Kirk Patton
Unix Administrator
Transmeta Inc.
Tel. 408 919-3055

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list