[GE users] Queue dropped because it is full

Kirk Patton kpatton at transmeta.com
Tue Apr 27 15:13:39 BST 2004


    [ The following text is in the "unknown-8bit" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some characters may be displayed incorrectly. ]

On Fri, Apr 23, 2004 at 07:42:32PM -0700, Ron Chen wrote:
> What did the qmaster log say?
> 
> (and it may help if you can check the contents of the
> other logs)
> 
>  -Ron

The problem is reproduceable.  sge_schedd is running at 99.0% of cpu currently.
I cleared the log yesterday before we tried to hammer the cluster.  Here is what 
got logged when the jobs started comming in.
Mon Apr 26 15:14:37 2004|qmaster|lsf-k8|E|enrolled, but leave_commd() call failed with status: NOT ENROLLED
Mon Apr 26 15:14:38 2004|qmaster|lsf-k8|E|enrolled, but leave_commd() call failed with status: NOT ENROLLED
Mon Apr 26 16:32:29 2004|qmaster|lsf-k8|W|job 13580.1 failed on host op240-020.transmeta.com  assumedly after job because: job 13580.1 died through signal HUP (1)
Mon Apr 26 16:49:06 2004|qmaster|lsf-k8|W|job 13585.1 failed on host op240-012.transmeta.com  assumedly after job because: job 13585.1 died through signal HUP (1)
Mon Apr 26 17:01:22 2004|qmaster|lsf-k8|W|job 13587.1 failed on host op240-027.transmeta.com  assumedly after job because: job 13587.1 died through signal HUP (1)
Mon Apr 26 17:52:13 2004|qmaster|lsf-k8|E|acknowledge timeout after 600 seconds for event client (schedd:1) on host "lsf-k8.transmeta.com"

qstat

job-ID  prior name       user         state submit/start at     queue      master  ja-task-ID 
---------------------------------------------------------------------------------------------
   6206     0 run_cmd_sc gurmeet      r     04/22/2004 18:02:44 common.op2 MASTER         
  13478     0 teesejob   teeseastro   dr    04/26/2004 10:44:57 common.op2 MASTER         
  13549     0 teesejob   teeseastro   r     04/26/2004 12:21:50 common.op2 MASTER         
    540     0 QLOGIN     justin       r     04/14/2004 11:46:34 common.op2 MASTER         
  13551     0 teesejob   teeseastro   r     04/26/2004 12:27:27 common.op2 MASTER         
  13547     0 teesejob   teeseastro   r     04/26/2004 12:20:19 common.op2 MASTER         
  13548     0 teesejob   teeseastro   r     04/26/2004 12:20:19 common.op2 MASTER         
  13427     0 teesejob   teeseastro   r     04/26/2004 10:11:03 common.op2 MASTER         
  13560     0 teesejob   teeseastro   r     04/26/2004 13:02:11 common.op2 MASTER         
  13970     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13971     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13972     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13973     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13974     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13975     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13976     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13977     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13978     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13979     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13980     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13981     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13982     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13983     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13984     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13985     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13986     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13987     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13988     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13989     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13990     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13991     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13992     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13993     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13994     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13995     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13996     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13997     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13998     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  13999     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14000     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14001     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14002     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14003     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14004     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14005     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14006     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14007     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14008     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14009     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14010     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14011     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14012     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14013     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14014     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14015     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14016     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14017     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14018     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14019     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14020     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14021     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14022     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14023     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14024     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14025     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14026     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14027     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14028     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14029     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14030     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14031     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14032     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14033     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14034     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14035     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14036     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14037     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14038     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14039     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14040     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14041     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14042     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14043     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14044     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14045     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14046     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14047     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14048     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14049     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14050     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14051     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14052     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14053     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14054     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14055     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14056     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14057     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14058     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14059     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14060     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14061     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14062     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14063     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14064     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14065     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14066     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14067     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14068     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14069     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14070     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14071     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14072     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14073     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14074     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14075     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14076     0 grid       gsmith       qw    04/26/2004 17:41:58                           
  14077     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14078     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14079     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14080     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14081     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14082     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14083     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14084     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14085     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14086     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14087     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14088     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14089     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14090     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14091     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14092     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14093     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14094     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14095     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14096     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14097     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14098     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14099     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14100     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14101     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14102     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14103     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14104     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14105     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14106     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14107     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14108     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14109     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14110     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14111     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14112     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14113     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14114     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14115     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14116     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14117     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14118     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14119     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14120     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14121     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14122     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14123     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14124     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14125     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14126     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14127     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14128     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14129     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14130     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14131     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14132     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14133     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14134     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14135     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14136     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14137     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14138     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14139     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14140     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14141     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14142     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14143     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14144     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14145     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14146     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14147     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14148     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14149     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14150     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14151     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14152     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14153     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14154     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14155     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14156     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14157     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14158     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14159     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14160     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14161     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14162     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14163     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14164     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14165     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14166     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14167     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14168     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14169     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14170     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14171     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14172     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14173     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14174     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14175     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14176     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14177     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14178     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14179     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14180     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14181     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14182     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14183     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14184     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14185     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14186     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14187     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14188     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14189     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14190     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14191     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14192     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14193     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14194     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14195     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14196     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14197     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14198     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14199     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14200     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14201     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14202     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14203     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14204     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14205     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14206     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14207     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14208     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14209     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14210     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14211     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14212     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14213     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14214     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14215     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14216     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14217     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14218     0 grid       gsmith       qw    04/26/2004 17:41:59                           
  14219     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14220     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14221     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14222     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14223     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14224     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14225     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14226     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14227     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14228     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14229     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14230     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14231     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14232     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14233     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14234     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14235     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14236     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14237     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14238     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14239     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14240     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14241     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14242     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14243     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14244     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14245     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14246     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14247     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14248     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14249     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14250     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14251     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14252     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14253     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14254     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14255     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14256     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14257     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14258     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14259     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14260     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14261     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14262     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14263     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14264     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14265     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14266     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14267     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14268     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14269     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14270     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14271     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14272     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14273     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14274     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14275     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14276     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14277     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14278     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14279     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14280     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14281     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14282     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14283     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14284     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14285     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14286     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14287     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14288     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14289     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14290     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14291     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14292     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14293     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14294     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14295     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14296     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14297     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14298     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14299     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14300     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14301     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14302     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14303     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14304     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14305     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14306     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14307     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14308     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14309     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14310     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14311     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14312     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14313     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14314     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14315     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14316     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14317     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14318     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14319     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14320     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14321     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14322     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14323     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14324     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14325     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14326     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14327     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14328     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14329     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14330     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14331     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14332     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14333     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14334     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14335     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14336     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14337     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14338     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14339     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14340     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14341     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14342     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14343     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14344     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14345     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14346     0 grid       gsmith       qw    04/26/2004 17:42:00                           
  14347     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14348     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14349     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14350     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14351     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14352     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14353     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14354     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14355     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14356     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14357     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14358     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14359     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14360     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14361     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14362     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14363     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14364     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14365     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14366     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14367     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14368     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14369     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14370     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14371     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14372     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14373     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14374     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14375     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14376     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14377     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14378     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14379     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14380     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14381     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14382     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14383     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14384     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14385     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14386     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14387     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14388     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14389     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14390     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14391     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14392     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14393     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14394     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14395     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14396     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14397     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14398     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14399     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14400     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14401     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14402     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14403     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14404     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14405     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14406     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14407     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14408     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14409     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14410     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14411     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14412     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14413     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14414     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14415     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14416     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14417     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14418     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14419     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14420     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14421     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14422     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14423     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14424     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14425     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14426     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14427     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14428     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14429     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14430     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14431     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14432     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14433     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14434     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14435     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14436     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14437     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14438     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14439     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14440     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14441     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14442     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14443     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14444     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14445     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14446     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14447     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14448     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14449     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14450     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14451     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14452     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14453     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14454     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14455     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14456     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14457     0 grid       gsmith       qw    04/26/2004 17:42:01                           
  14458     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14459     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14460     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14461     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14462     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14463     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14464     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14465     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14466     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14467     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14468     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14469     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14470     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14471     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14472     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14473     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14474     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14475     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14476     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14477     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14478     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14479     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14480     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14481     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14482     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14483     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14484     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14485     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14486     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14487     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14488     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14489     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14490     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14491     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14492     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14493     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14494     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14495     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14496     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14497     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14498     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14499     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14500     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14501     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14502     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14503     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14504     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14505     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14506     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14507     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14508     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14509     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14510     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14511     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14512     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14513     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14514     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14515     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14516     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14517     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14518     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14519     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14520     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14521     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14522     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14523     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14524     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14525     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14526     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14527     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14528     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14529     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14530     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14531     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14532     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14533     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14534     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14535     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14536     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14537     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14538     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14539     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14540     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14541     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14542     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14543     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14544     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14545     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14546     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14547     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14548     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14549     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14550     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14551     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14552     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14553     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14554     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14555     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14556     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14557     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14558     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14559     0 grid       gsmith       qw    04/26/2004 17:42:02                           
  14560     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14561     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14562     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14563     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14564     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14565     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14566     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14567     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14568     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14569     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14570     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14571     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14572     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14573     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14574     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14575     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14576     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14577     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14578     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14579     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14580     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14581     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14582     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14583     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14584     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14585     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14586     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14587     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14588     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14589     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14590     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14591     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14592     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14593     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14594     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14595     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14596     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14597     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14598     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14599     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14600     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14601     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14602     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14603     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14604     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14605     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14606     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14607     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14608     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14609     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14610     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14611     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14612     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14613     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14614     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14615     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14616     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14617     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14618     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14619     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14620     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14621     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14622     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14623     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14624     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14625     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14626     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14627     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14628     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14629     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14630     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14631     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14632     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14633     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14634     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14635     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14636     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14637     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14638     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14639     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14640     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14641     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14642     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14643     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14644     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14645     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14646     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14647     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14648     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14649     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14650     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14651     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14652     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14653     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14654     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14655     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14656     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14657     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14658     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14659     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14660     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14661     0 grid       gsmith       qw    04/26/2004 17:42:03                           
  14662     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14663     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14664     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14665     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14666     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14667     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14668     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14669     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14670     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14671     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14672     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14673     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14674     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14675     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14676     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14677     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14678     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14679     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14680     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14681     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14682     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14683     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14684     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14685     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14686     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14687     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14688     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14689     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14690     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14691     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14692     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14693     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14694     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14695     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14696     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14697     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14698     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14699     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14700     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14701     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14702     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14703     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14704     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14705     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14706     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14707     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14708     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14709     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14710     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14711     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14712     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14713     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14714     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14715     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14716     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14717     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14718     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14719     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14720     0 grid       gsmith       qw    04/26/2004 17:42:04                           
  14721     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14722     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14723     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14724     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14725     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14726     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14727     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14728     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14729     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14730     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14731     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14732     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14733     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14734     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14735     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14736     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14737     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14738     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14739     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14740     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14741     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14742     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14743     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14744     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14745     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14746     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14747     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14748     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14749     0 grid       gsmith       qw    04/26/2004 17:42:05                           
  14751     0 date       gsmith       qw    04/26/2004 18:08:35                           
  14754     0 teesejob   teesea3d     qw    04/26/2004 22:16:41                           
  14755     0 teesejob   teesea3d     qw    04/26/2004 22:16:41                           
  14756     0 teesejob   teesea3d     qw    04/26/2004 22:16:41                           
  14757     0 teesejob   teesea3d     qw    04/26/2004 22:16:41                           
  14758     0 teesejob   teesea3d     qw    04/26/2004 22:16:41                           
  14759     0 teesejob   teesea3d     qw    04/26/2004 22:16:41                           
  14760     0 teesejob   teesea3d     qw    04/26/2004 22:16:41                           

qstat -j 14760

job_number:                 14760
submission_time:            Mon Apr 26 22:16:41 2004
owner:                      teesea3d
uid:                        1719
group:                      teesea3d
gid:                        1719
sge_o_home:                 /home/teesea3d
sge_o_path:                 /usr/bin:/bin
sge_o_shell:                /bin/bash
sge_o_workdir:              /export/home/tmp
sge_o_host:                 teese-1
account:                    sge
cwd:                        /export/home/tmp
path_aliases:               /tmp_mnt/ * * /
hard resource_list:         priority=common,os=Linux
mail_list:                  teesea3d at teese-1.transmeta.com
notify:                     FALSE
job_name:                   teesejob
restart:                    n
env_list:                   PWD=/export/home/tmp,USER=teesea3d,SGE_ROOT=/transmeta/sge,SHLVL=2,_=/transmeta/sge/bin/glinux/qrsh,SHELL=/bin/bash,HOME=/home/teesea3d,PATH=/usr/bin:/bin,QRSH_PORT=teese-1.transmeta.com:45368,QRSH_COMMAND=/project/teese/sbin/teese-slave.pl?--nslaves?8?--sim?sim?--timeout?64800?--sim-mode?translator-heavy?--gdbpath?../astro-gdb?--bundle-server?http://teese-1:12008//?--netport?teese-1/12001?--netport?teese-1/12002?--netport?teese-1/12003?--netport?teese-1/12004?--netport?teese-1/12005?--netport?teese-1/12006?--netport?teese-1/12007?--netport?teese-1/12000
project:                    teese_normal
context:                    SGE_INTERACTIVE=yes
scheduling info:            (Collecting of scheduler job information is turned off)

I will turn scheduling info back on.  I turned it off to see if it was part of the problem.

qhost

HOSTNAME             ARCH       NPROC  LOAD   MEMTOT   MEMUSE   SWAPTO   SWAPUS
-------------------------------------------------------------------------------
global               -              -     -        -        -        -        -
aslt1.transmeta.com  i686           2  0.08     2.0G   708.6M     1.9G   163.3M
aslt2.transmeta.com  i686           2  0.87     3.8G   936.1M     1.9G   192.1M
captain.transmeta.com i686           2  0.21  1006.6M   303.7M     2.0G    30.9M
chris-linux.transmeta.com i686           2  0.01  1006.6M   328.5M     2.0G   179.6M
dioxide.transmeta.com sun4u         12  0.03    12.0G     1.6G    17.9G      0.0
l0023001.transmeta.com i686           2  0.09  1006.6M   171.6M     2.0G    25.6M
l0023004.transmeta.com i686           2  0.17  1006.6M   170.4M     2.0G    25.8M
l0023013.transmeta.com i686           2  0.00  1006.6M   175.2M     2.0G    26.3M
l0023017.transmeta.com i686           2  0.00  1006.6M   183.2M     2.0G    28.0M
l0023018.transmeta.com i686           2  0.02  1006.6M   177.1M     2.0G    27.6M
l0023021.transmeta.com i686           2  0.07  1006.6M   177.6M     2.0G    28.9M
l0023037.transmeta.com i686           2  0.00  1006.6M   175.6M     2.0G    28.2M
l0023041.transmeta.com i686           2  0.00  1006.6M   175.1M     2.0G    20.0M
l0023046.transmeta.com i686           2  1.65  1006.6M   563.4M     2.0G   514.6M
op240-000.transmeta.com i686           2  0.00     3.9G   292.3M     7.9G      0.0
op240-001.transmeta.com i686           2  0.00     3.9G   331.3M     7.9G    12.0K
op240-002.transmeta.com i686           2  0.02     3.9G   317.4M     7.9G      0.0
op240-003.transmeta.com i686           2  1.00     3.9G     2.1G     7.9G     3.9M
op240-004.transmeta.com i686           2  0.00     3.9G   286.0M     7.9G     4.0K
op240-005.transmeta.com i686           2  4.00     3.9G   233.3M     7.9G      0.0
op240-006.transmeta.com i686           2  0.00     3.9G   342.0M     7.9G     8.0K
op240-007.transmeta.com i686           2  0.01     3.9G   403.0M     7.9G      0.0
op240-008.transmeta.com i686           2  0.00     3.9G   234.0M     7.9G      0.0
op240-009.transmeta.com i686           2  1.01     3.9G   308.3M     7.9G      0.0
op240-010.transmeta.com i686           2  0.00     3.9G   412.4M     7.9G   764.0K
op240-011.transmeta.com i686           2  0.00     3.9G   231.8M     7.9G      0.0
op240-012.transmeta.com i686           2  0.00     3.9G   226.2M     7.9G      0.0
op240-013.transmeta.com i686           2  0.00     3.9G   226.6M     7.9G      0.0
op240-014.transmeta.com i686           2  0.02     3.9G   230.0M     7.9G      0.0
op240-015.transmeta.com i686           2  0.03     3.9G   226.5M     7.9G      0.0
op240-016.transmeta.com i686           2  0.02     3.9G   216.5M     7.9G      0.0
op240-017.transmeta.com i686           2  0.00     3.9G   222.7M     7.9G      0.0
op240-018.transmeta.com i686           2  0.00     3.9G   225.3M     7.9G      0.0
op240-019.transmeta.com i686           2  0.00     3.9G   225.7M     7.9G      0.0
op240-020.transmeta.com i686           2  0.00     3.9G   222.9M     7.9G      0.0
op240-021.transmeta.com i686           2  0.00     3.9G   226.3M     7.9G      0.0
op240-022.transmeta.com i686           2  0.00     3.9G   267.8M     7.9G    20.0K
op240-023.transmeta.com i686           2  0.00     3.9G   230.2M     7.9G      0.0
op240-024.transmeta.com i686           2  0.00     3.9G   220.2M     7.9G      0.0
op240-025.transmeta.com i686           2  0.00     3.9G   238.6M     7.9G      0.0
op240-026.transmeta.com i686           2  0.00     3.9G   242.1M     7.9G      0.0
op240-027.transmeta.com i686           2  0.00     3.9G   227.3M     7.9G      0.0
op240-028.transmeta.com i686           2  0.02     3.9G   215.6M     7.9G      0.0
op240-029.transmeta.com i686           2  0.00     3.9G   222.6M     7.9G      0.0
opteron-test.transmeta.com i686           2  0.00     3.9G   971.1M     7.9G    18.8M
ruthenium.transmeta.com sun4u          2  0.25     2.0G   474.0M   512.0M      0.0
technetium.transmeta.com sun4u          2  0.32     1.0G   143.0M     1.2G      0.0

qstat -F -q common.op240-025

queuename            qtype used/tot. load_avg arch      states
----------------------------------------------------------------------------
common.op240-025     BIP   0/2       0.01     i686      
	gc:license_a=5.000000
	gc:arcadia=4.000000
	gc:hspice=6.000000
	gc:vampire-drc=6.000000
	gc:vampire-lvs=6.000000
	gc:solid=190.000000
	gc:simplex_v_storm=2.000000
	gc:dynacore=9.000000
	hl:arch=i686
	hl:num_proc=2.000000
	hl:load_avg=0.010000
	hl:load_short=0.050000
	hl:load_medium=0.010000
	hl:load_long=0.000000
	hl:np_load_avg=0.005000
	hl:np_load_short=0.025000
	hl:np_load_medium=0.005000
	hl:np_load_long=0.000000
	hl:mem_free=3.65G
	hl:mem_total=3.88G
	hl:swap_free=7.87G
	hl:swap_total=7.87G
	hl:virtual_free=11.52G
	hl:virtual_total=11.75G
	hl:mem_used=239.03M
	hl:swap_used=0.000000
	hl:virtual_used=239.03M
	hv:swap_rsvd=0.000000
	hv:swap_rate=0.000000
	qc:slots=2.000000
	qf:s_vmem=infinity
	qf:h_vmem=infinity
	qf:s_fsize=infinity
	qf:h_fsize=infinity
	hl:cpu=0.600000
	hv:systype=0.000000
	hv:glibc=0.000000
	hl:cpuf=100.000000
	hl:relative_speed=5.000000
	hl:bitsize=32-bit
	hl:os=Linux
	hl:platform=i386
	hl:release=2.4.20-18.7smp
	hl:vendor-rel=7.3
	hl:cpuinfo=_AMD_Opteron(tm)_Processor_248_
	hc:exclusive=1.000000
	hc:running=50.000000
	qf:qname=common.op240-025
	qf:hostname=op240-025.transmeta.com
	qf:tmpdir=/tmp
	qv:calendar=NONE
	qf:seq_no=13000.000000
	qf:rerun=1.000000
	qf:s_rt=infinity
	qf:h_rt=infinity
	qf:s_cpu=infinity
	qf:h_cpu=infinity
	qf:s_data=infinity
	qf:h_data=infinity
	qf:s_stack=infinity
	qf:h_stack=infinity
	qf:s_core=infinity
	qf:h_core=infinity
	qf:s_rss=infinity
	qf:h_rss=infinity
	qf:min_cpu_interval=00:05:00
	qf:priority=common

qalter -w v 13983

Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.aslt1" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.aslt2" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023001" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023004" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023013" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023017" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023018" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023021" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023037" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023041" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023046" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-000" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-001" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-002" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-003" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-004" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-005" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-006" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-007" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-008" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-009" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-010" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-011" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-012" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-013" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-014" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-015" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-016" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-017" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-018" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-019" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-020" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-021" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-022" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-023" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-024" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-025" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-026" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-027" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-028" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-029" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.opteron-test" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.ruthenium" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.technetium" because it offers only hv:os=none
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.aslt1" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.aslt2" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023001" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023004" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023013" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023017" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023018" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023021" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023037" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023041" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023046" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-000" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-001" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-002" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-003" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-004" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-005" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-006" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-007" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-008" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-009" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-010" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-011" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-012" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-013" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-014" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-015" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-016" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-017" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-018" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-019" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-020" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-021" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-022" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-023" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-024" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-025" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-026" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-027" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-028" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-029" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.opteron-test" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.ruthenium" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.technetium" because it offers only qf:priority=critical
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.aslt1" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.aslt2" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023001" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023004" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023013" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023017" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023018" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023021" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023037" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023041" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023046" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-000" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-001" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-002" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-003" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-004" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-005" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-006" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-007" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-008" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-009" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-010" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-011" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-012" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-013" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-014" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-015" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-016" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-017" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-018" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-019" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-020" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-021" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-022" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-023" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-024" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-025" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-026" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-027" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-028" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-029" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.opteron-test" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.ruthenium" because it offers only qf:priority=trivial
Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.technetium" because it offers only qf:priority=trivial
verification: no suitable queues


> 
> --- Kirk Patton <kpatton at transmeta.com> wrote:
> > Hello Andy,
> > 
> > We had another failure to schedule jobs.  We first
> > noticed the issue
> > after a user submitted and then deleted about 4000
> > jobs.  After that,
> > not further scheduling took place.
> > 
> > I collected some output from several commands.
> > 
> > We may be able to repeate this issue.  I can ask the
> > user to try.
> > Is there any other data I could collect that would
> > be usefull in tracking this down?
> > 
> > Kirk
> > 
> > 
> > HOSTNAME             ARCH       NPROC  LOAD   MEMTOT
> >   MEMUSE   SWAPTO   SWAPUS
> >
> -------------------------------------------------------------------------------
> > global               -              -     -        -
> >        -        -        -
> > aslt1.transmeta.com  i686           2  1.13     2.0G
> >   301.0M     1.9G   165.0M
> > aslt2.transmeta.com  i686           2  0.77     3.8G
> >   822.1M     1.9G   181.9M
> > captain.transmeta.com i686           2  0.03 
> > 1006.6M   259.5M     2.0G    65.0M
> > chris-linux.transmeta.com i686           2  0.03 
> > 1006.6M   332.4M     2.0G   143.1M
> > dioxide.transmeta.com sun4u         12  0.12   
> > 12.0G     1.8G    17.9G      0.0
> > l0023001.transmeta.com i686           2  0.00 
> > 1006.6M    36.9M     2.0G    25.3M
> > l0023004.transmeta.com i686           2  0.00 
> > 1006.6M    42.2M     2.0G    25.2M
> > l0023013.transmeta.com i686           2  0.00 
> > 1006.6M    38.0M     2.0G    26.2M
> > l0023017.transmeta.com i686           2  0.00 
> > 1006.6M    46.1M     2.0G    27.9M
> > l0023018.transmeta.com i686           2  1.05 
> > 1006.6M   404.3M     2.0G   257.1M
> > l0023021.transmeta.com i686           2  0.00 
> > 1006.6M    40.1M     2.0G    28.7M
> > l0023037.transmeta.com i686           2  0.27 
> > 1006.6M    45.1M     2.0G    32.1M
> > l0023041.transmeta.com i686           2  0.02 
> > 1006.6M    43.3M     2.0G    19.8M
> > l0023046.transmeta.com i686           2  1.73 
> > 1006.6M   698.5M     2.0G   865.1M
> > op240-000.transmeta.com i686           2  0.00    
> > 3.9G   286.9M     7.9G      0.0
> > op240-001.transmeta.com i686           2  0.00    
> > 3.9G   315.3M     7.9G     4.0K
> > op240-002.transmeta.com i686           2  0.00    
> > 3.9G   308.3M     7.9G      0.0
> > op240-003.transmeta.com i686           2  1.03    
> > 3.9G     1.6G     7.9G    20.0K
> > op240-004.transmeta.com i686           2  0.00    
> > 3.9G   276.9M     7.9G     4.0K
> > op240-005.transmeta.com i686           2  0.00    
> > 3.9G   220.1M     7.9G      0.0
> > op240-006.transmeta.com i686           2  0.00    
> > 3.9G   333.0M     7.9G     8.0K
> > op240-007.transmeta.com i686           2  0.00    
> > 3.9G   372.8M     7.9G      0.0
> > op240-008.transmeta.com i686           2  0.00    
> > 3.9G   230.9M     7.9G      0.0
> > op240-009.transmeta.com i686           2  0.00    
> > 3.9G   228.7M     7.9G      0.0
> > op240-010.transmeta.com i686           2  0.00    
> > 3.9G   652.7M     7.9G   764.0K
> > op240-011.transmeta.com i686           2  0.00    
> > 3.9G   217.1M     7.9G      0.0
> > op240-012.transmeta.com i686           2  0.00    
> > 3.9G   211.8M     7.9G      0.0
> > op240-013.transmeta.com i686           2  0.00    
> > 3.9G   213.3M     7.9G      0.0
> > op240-014.transmeta.com i686           2  0.13    
> > 3.9G   214.3M     7.9G      0.0
> > op240-015.transmeta.com i686           2  0.00    
> > 3.9G   208.9M     7.9G      0.0
> > op240-016.transmeta.com i686           2  0.00    
> > 3.9G   209.5M     7.9G      0.0
> > op240-017.transmeta.com i686           2  0.00    
> > 3.9G   213.4M     7.9G      0.0
> > op240-018.transmeta.com i686           2  0.00    
> > 3.9G   205.5M     7.9G      0.0
> > op240-019.transmeta.com i686           2  0.01    
> > 3.9G   207.2M     7.9G      0.0
> > op240-020.transmeta.com i686           2  0.00    
> > 3.9G   211.5M     7.9G      0.0
> > op240-021.transmeta.com i686           2  0.00    
> > 3.9G   210.6M     7.9G      0.0
> > op240-022.transmeta.com i686           2  0.00    
> > 3.9G   272.5M     7.9G    20.0K
> > op240-023.transmeta.com i686           2  0.01    
> > 3.9G   203.5M     7.9G      0.0
> > op240-024.transmeta.com i686           2  0.01    
> > 3.9G   204.0M     7.9G      0.0
> > op240-025.transmeta.com i686           2  0.07    
> > 3.9G   202.2M     7.9G      0.0
> > op240-026.transmeta.com i686           2  0.00    
> > 3.9G   221.8M     7.9G      0.0
> > op240-027.transmeta.com i686           2  0.00    
> > 3.9G   209.8M     7.9G      0.0
> > op240-028.transmeta.com i686           2  0.01    
> > 3.9G   202.4M     7.9G      0.0
> > op240-029.transmeta.com i686           2  1.05    
> > 3.9G   806.8M     7.9G      0.0
> > opteron-test.transmeta.com i686           2  0.01   
> >  3.9G   705.7M     7.9G    18.8M
> > ruthenium.transmeta.com sun4u          2  0.26    
> > 2.0G   472.0M   512.0M      0.0
> > technetium.transmeta.com sun4u          2  0.27    
> > 1.0G   141.0M     1.2G      0.0
> > 
> > 
> > job_number:                 10542
> > submission_time:            Fri Apr 23 16:07:50 2004
> > owner:                      teesea3d
> > uid:                        1719
> > group:                      teesea3d
> > gid:                        1719
> > sge_o_home:                 /home/teesea3d
> > sge_o_path:                 /usr/bin:/bin
> > sge_o_shell:                /bin/bash
> > sge_o_workdir:              /export/home/tmp
> > sge_o_host:                 teese-1
> > account:                    sge
> > cwd:                        /export/home/tmp
> > path_aliases:               /tmp_mnt/ * * /
> > hard resource_list:         priority=common,os=Linux
> > mail_list:                 
> > teesea3d at teese-1.transmeta.com
> > notify:                     FALSE
> > job_name:                   teesejob
> > restart:                    n
> > env_list:                  
> >
> PWD=/export/home/tmp,USER=teesea3d,SGE_ROOT=/transmeta/sge,SHLVL=2,_=/transmeta/sge/bin/glinux/qrsh,SHELL=/bin/bash,HOME=/home/teesea3d,PATH=/usr/bin:/bin,QRSH_PORT=teese-1.transmeta.com:35029,QRSH_COMMAND=/project/teese/sbin/teese-slave.pl?--nslaves?9?--sim?sim?--timeout?64800?--sim-mode?translator-heavy?--gdbpath?../astro-gdb?--bundle-server?http://teese-1:12008///?--netport?teese-1/12003?--netport?teese-1/12004?--netport?teese-1/12005?--netport?teese-1/12006?--netport?teese-1/12007?--netport?teese-1/12000?--netport?teese-1/12001?--netport?teese-1/12002
> > project:                    teese_normal
> > context:                    SGE_INTERACTIVE=yes
> > scheduling info:            queue "trivial.aslt2"
> > dropped because it is temporarily not available
> >                             queue
> > "trivial.opteron-test" dropped because it is
> > temporarily not available
> >                             queue
> > "trivial.op240-015" dropped because it is
> > temporarily not available
> >                             queue
> > "trivial.op240-004" dropped because it is
> > temporarily not available
> >                             queue
> > "trivial.op240-028" dropped because it is
> > temporarily not available
> >                             queue
> > "trivial.op240-022" dropped because it is
> > temporarily not available
> >                             queue
> > "trivial.op240-024" dropped because it is
> > temporarily not available
> >                             queue
> > "trivial.op240-005" dropped because it is
> > temporarily not available
> >                             queue
> > "trivial.op240-014" dropped because it is
> > temporarily not available
> >                             queue
> > "trivial.op240-023" dropped because it is
> > temporarily not available
> >                             queue
> > "trivial.op240-010" dropped because it is
> > temporarily not available
> >                             queue
> > "trivial.op240-007" dropped because it is
> > temporarily not available
> >                             queue
> > "trivial.op240-006" dropped because it is
> > temporarily not available
> >                             queue
> > "trivial.op240-009" dropped because it is
> > temporarily not available
> >                             queue
> > "trivial.op240-003" dropped because it is
> > temporarily not available
> >                             queue
> > "trivial.op240-029" dropped because it is
> > temporarily not available
> >                             queue
> > "trivial.op240-017" dropped because it is
> > temporarily not available
> >                             queue
> > "trivial.op240-012" dropped because it is
> > temporarily not available
> >                             queue
> > "trivial.op240-001" dropped because it is
> > temporarily not available
> >                             queue "trivial.l0023017"
> > dropped because it is temporarily not available
> > 
> === message truncated ===
> 
> 
> 
> 	
> 		
> __________________________________
> Do you Yahoo!?
> Yahoo! Photos: High-quality 4x6 digital prints for 25?
> http://photos.yahoo.com/ph/print_splash
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail: users-help at gridengine.sunsource.net
> 

-- 
Kirk Patton
Unix Administrator
Transmeta Inc.
Tel. 408 919-3055

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list