[GE users] Queue dropped because it is full

Andreas Haas Andreas.Haas at Sun.COM
Tue Apr 27 18:21:52 BST 2004


    [ The following text is in the "ISO-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Hi Kirk,

I searched the changes from p5 to p6 whether they match your symptom ...
without success. So we need to figure out what's wrong, but we should
not do this via users@ as other users might not be interested in this.

At first I'd like to get qmaster and scheduler messages that are complete
insofar that they contain the "starting up" message of the run where you had
that failure.

Can you please send me these files?

Thanks a lot,
Andreas

On Tue, 27 Apr 2004, Kirk Patton wrote:

> On Fri, Apr 23, 2004 at 07:42:32PM -0700, Ron Chen wrote:
> > What did the qmaster log say?
> >
> > (and it may help if you can check the contents of the
> > other logs)
> >
> >  -Ron
>
> The problem is reproduceable.  sge_schedd is running at 99.0% of cpu currently.
> I cleared the log yesterday before we tried to hammer the cluster.  Here is what
> got logged when the jobs started comming in.
> Mon Apr 26 15:14:37 2004|qmaster|lsf-k8|E|enrolled, but leave_commd() call failed with status: NOT ENROLLED
> Mon Apr 26 15:14:38 2004|qmaster|lsf-k8|E|enrolled, but leave_commd() call failed with status: NOT ENROLLED
> Mon Apr 26 16:32:29 2004|qmaster|lsf-k8|W|job 13580.1 failed on host op240-020.transmeta.com  assumedly after job because: job 13580.1 died through signal HUP (1)
> Mon Apr 26 16:49:06 2004|qmaster|lsf-k8|W|job 13585.1 failed on host op240-012.transmeta.com  assumedly after job because: job 13585.1 died through signal HUP (1)
> Mon Apr 26 17:01:22 2004|qmaster|lsf-k8|W|job 13587.1 failed on host op240-027.transmeta.com  assumedly after job because: job 13587.1 died through signal HUP (1)
> Mon Apr 26 17:52:13 2004|qmaster|lsf-k8|E|acknowledge timeout after 600 seconds for event client (schedd:1) on host "lsf-k8.transmeta.com"
>
> qstat
>
> job-ID  prior name       user         state submit/start at     queue      master  ja-task-ID
> ---------------------------------------------------------------------------------------------
>    6206     0 run_cmd_sc gurmeet      r     04/22/2004 18:02:44 common.op2 MASTER
>   13478     0 teesejob   teeseastro   dr    04/26/2004 10:44:57 common.op2 MASTER
>   13549     0 teesejob   teeseastro   r     04/26/2004 12:21:50 common.op2 MASTER
>     540     0 QLOGIN     justin       r     04/14/2004 11:46:34 common.op2 MASTER
>   13551     0 teesejob   teeseastro   r     04/26/2004 12:27:27 common.op2 MASTER
>   13547     0 teesejob   teeseastro   r     04/26/2004 12:20:19 common.op2 MASTER
>   13548     0 teesejob   teeseastro   r     04/26/2004 12:20:19 common.op2 MASTER
>   13427     0 teesejob   teeseastro   r     04/26/2004 10:11:03 common.op2 MASTER
>   13560     0 teesejob   teeseastro   r     04/26/2004 13:02:11 common.op2 MASTER
>   13970     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13971     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13972     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13973     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13974     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13975     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13976     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13977     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13978     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13979     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13980     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13981     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13982     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13983     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13984     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13985     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13986     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13987     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13988     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13989     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13990     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13991     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13992     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13993     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13994     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13995     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13996     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13997     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13998     0 grid       gsmith       qw    04/26/2004 17:41:58
>   13999     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14000     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14001     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14002     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14003     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14004     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14005     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14006     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14007     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14008     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14009     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14010     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14011     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14012     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14013     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14014     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14015     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14016     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14017     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14018     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14019     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14020     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14021     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14022     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14023     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14024     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14025     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14026     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14027     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14028     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14029     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14030     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14031     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14032     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14033     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14034     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14035     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14036     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14037     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14038     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14039     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14040     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14041     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14042     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14043     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14044     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14045     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14046     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14047     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14048     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14049     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14050     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14051     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14052     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14053     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14054     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14055     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14056     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14057     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14058     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14059     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14060     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14061     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14062     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14063     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14064     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14065     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14066     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14067     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14068     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14069     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14070     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14071     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14072     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14073     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14074     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14075     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14076     0 grid       gsmith       qw    04/26/2004 17:41:58
>   14077     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14078     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14079     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14080     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14081     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14082     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14083     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14084     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14085     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14086     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14087     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14088     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14089     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14090     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14091     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14092     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14093     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14094     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14095     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14096     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14097     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14098     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14099     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14100     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14101     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14102     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14103     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14104     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14105     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14106     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14107     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14108     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14109     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14110     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14111     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14112     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14113     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14114     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14115     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14116     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14117     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14118     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14119     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14120     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14121     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14122     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14123     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14124     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14125     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14126     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14127     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14128     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14129     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14130     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14131     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14132     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14133     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14134     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14135     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14136     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14137     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14138     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14139     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14140     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14141     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14142     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14143     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14144     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14145     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14146     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14147     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14148     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14149     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14150     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14151     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14152     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14153     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14154     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14155     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14156     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14157     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14158     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14159     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14160     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14161     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14162     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14163     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14164     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14165     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14166     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14167     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14168     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14169     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14170     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14171     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14172     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14173     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14174     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14175     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14176     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14177     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14178     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14179     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14180     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14181     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14182     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14183     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14184     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14185     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14186     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14187     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14188     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14189     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14190     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14191     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14192     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14193     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14194     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14195     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14196     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14197     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14198     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14199     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14200     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14201     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14202     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14203     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14204     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14205     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14206     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14207     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14208     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14209     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14210     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14211     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14212     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14213     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14214     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14215     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14216     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14217     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14218     0 grid       gsmith       qw    04/26/2004 17:41:59
>   14219     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14220     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14221     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14222     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14223     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14224     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14225     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14226     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14227     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14228     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14229     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14230     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14231     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14232     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14233     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14234     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14235     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14236     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14237     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14238     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14239     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14240     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14241     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14242     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14243     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14244     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14245     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14246     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14247     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14248     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14249     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14250     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14251     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14252     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14253     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14254     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14255     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14256     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14257     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14258     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14259     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14260     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14261     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14262     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14263     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14264     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14265     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14266     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14267     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14268     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14269     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14270     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14271     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14272     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14273     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14274     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14275     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14276     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14277     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14278     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14279     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14280     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14281     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14282     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14283     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14284     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14285     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14286     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14287     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14288     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14289     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14290     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14291     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14292     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14293     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14294     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14295     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14296     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14297     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14298     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14299     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14300     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14301     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14302     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14303     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14304     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14305     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14306     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14307     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14308     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14309     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14310     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14311     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14312     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14313     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14314     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14315     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14316     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14317     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14318     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14319     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14320     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14321     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14322     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14323     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14324     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14325     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14326     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14327     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14328     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14329     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14330     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14331     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14332     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14333     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14334     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14335     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14336     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14337     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14338     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14339     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14340     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14341     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14342     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14343     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14344     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14345     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14346     0 grid       gsmith       qw    04/26/2004 17:42:00
>   14347     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14348     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14349     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14350     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14351     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14352     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14353     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14354     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14355     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14356     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14357     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14358     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14359     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14360     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14361     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14362     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14363     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14364     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14365     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14366     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14367     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14368     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14369     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14370     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14371     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14372     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14373     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14374     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14375     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14376     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14377     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14378     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14379     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14380     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14381     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14382     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14383     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14384     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14385     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14386     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14387     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14388     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14389     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14390     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14391     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14392     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14393     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14394     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14395     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14396     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14397     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14398     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14399     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14400     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14401     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14402     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14403     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14404     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14405     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14406     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14407     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14408     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14409     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14410     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14411     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14412     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14413     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14414     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14415     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14416     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14417     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14418     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14419     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14420     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14421     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14422     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14423     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14424     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14425     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14426     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14427     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14428     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14429     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14430     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14431     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14432     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14433     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14434     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14435     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14436     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14437     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14438     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14439     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14440     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14441     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14442     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14443     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14444     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14445     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14446     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14447     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14448     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14449     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14450     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14451     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14452     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14453     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14454     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14455     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14456     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14457     0 grid       gsmith       qw    04/26/2004 17:42:01
>   14458     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14459     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14460     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14461     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14462     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14463     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14464     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14465     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14466     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14467     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14468     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14469     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14470     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14471     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14472     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14473     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14474     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14475     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14476     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14477     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14478     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14479     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14480     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14481     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14482     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14483     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14484     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14485     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14486     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14487     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14488     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14489     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14490     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14491     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14492     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14493     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14494     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14495     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14496     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14497     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14498     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14499     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14500     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14501     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14502     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14503     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14504     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14505     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14506     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14507     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14508     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14509     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14510     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14511     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14512     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14513     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14514     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14515     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14516     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14517     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14518     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14519     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14520     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14521     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14522     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14523     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14524     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14525     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14526     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14527     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14528     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14529     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14530     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14531     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14532     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14533     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14534     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14535     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14536     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14537     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14538     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14539     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14540     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14541     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14542     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14543     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14544     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14545     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14546     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14547     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14548     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14549     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14550     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14551     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14552     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14553     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14554     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14555     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14556     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14557     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14558     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14559     0 grid       gsmith       qw    04/26/2004 17:42:02
>   14560     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14561     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14562     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14563     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14564     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14565     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14566     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14567     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14568     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14569     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14570     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14571     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14572     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14573     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14574     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14575     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14576     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14577     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14578     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14579     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14580     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14581     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14582     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14583     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14584     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14585     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14586     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14587     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14588     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14589     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14590     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14591     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14592     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14593     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14594     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14595     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14596     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14597     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14598     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14599     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14600     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14601     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14602     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14603     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14604     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14605     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14606     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14607     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14608     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14609     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14610     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14611     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14612     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14613     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14614     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14615     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14616     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14617     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14618     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14619     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14620     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14621     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14622     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14623     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14624     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14625     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14626     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14627     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14628     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14629     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14630     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14631     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14632     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14633     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14634     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14635     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14636     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14637     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14638     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14639     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14640     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14641     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14642     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14643     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14644     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14645     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14646     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14647     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14648     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14649     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14650     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14651     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14652     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14653     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14654     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14655     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14656     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14657     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14658     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14659     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14660     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14661     0 grid       gsmith       qw    04/26/2004 17:42:03
>   14662     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14663     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14664     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14665     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14666     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14667     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14668     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14669     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14670     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14671     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14672     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14673     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14674     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14675     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14676     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14677     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14678     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14679     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14680     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14681     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14682     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14683     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14684     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14685     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14686     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14687     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14688     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14689     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14690     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14691     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14692     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14693     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14694     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14695     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14696     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14697     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14698     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14699     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14700     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14701     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14702     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14703     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14704     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14705     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14706     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14707     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14708     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14709     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14710     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14711     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14712     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14713     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14714     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14715     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14716     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14717     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14718     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14719     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14720     0 grid       gsmith       qw    04/26/2004 17:42:04
>   14721     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14722     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14723     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14724     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14725     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14726     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14727     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14728     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14729     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14730     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14731     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14732     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14733     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14734     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14735     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14736     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14737     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14738     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14739     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14740     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14741     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14742     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14743     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14744     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14745     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14746     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14747     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14748     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14749     0 grid       gsmith       qw    04/26/2004 17:42:05
>   14751     0 date       gsmith       qw    04/26/2004 18:08:35
>   14754     0 teesejob   teesea3d     qw    04/26/2004 22:16:41
>   14755     0 teesejob   teesea3d     qw    04/26/2004 22:16:41
>   14756     0 teesejob   teesea3d     qw    04/26/2004 22:16:41
>   14757     0 teesejob   teesea3d     qw    04/26/2004 22:16:41
>   14758     0 teesejob   teesea3d     qw    04/26/2004 22:16:41
>   14759     0 teesejob   teesea3d     qw    04/26/2004 22:16:41
>   14760     0 teesejob   teesea3d     qw    04/26/2004 22:16:41
>
> qstat -j 14760
>
> job_number:                 14760
> submission_time:            Mon Apr 26 22:16:41 2004
> owner:                      teesea3d
> uid:                        1719
> group:                      teesea3d
> gid:                        1719
> sge_o_home:                 /home/teesea3d
> sge_o_path:                 /usr/bin:/bin
> sge_o_shell:                /bin/bash
> sge_o_workdir:              /export/home/tmp
> sge_o_host:                 teese-1
> account:                    sge
> cwd:                        /export/home/tmp
> path_aliases:               /tmp_mnt/ * * /
> hard resource_list:         priority=common,os=Linux
> mail_list:                  teesea3d at teese-1.transmeta.com
> notify:                     FALSE
> job_name:                   teesejob
> restart:                    n
> env_list:                   PWD=/export/home/tmp,USER=teesea3d,SGE_ROOT=/transmeta/sge,SHLVL=2,_=/transmeta/sge/bin/glinux/qrsh,SHELL=/bin/bash,HOME=/home/teesea3d,PATH=/usr/bin:/bin,QRSH_PORT=teese-1.transmeta.com:45368,QRSH_COMMAND=/project/teese/sbin/teese-slave.pl?--nslaves?8?--sim?sim?--timeout?64800?--sim-mode?translator-heavy?--gdbpath?../astro-gdb?--bundle-server?http://teese-1:12008//?--netport?teese-1/12001?--netport?teese-1/12002?--netport?teese-1/12003?--netport?teese-1/12004?--netport?teese-1/12005?--netport?teese-1/12006?--netport?teese-1/12007?--netport?teese-1/12000
> project:                    teese_normal
> context:                    SGE_INTERACTIVE=yes
> scheduling info:            (Collecting of scheduler job information is turned off)
>
> I will turn scheduling info back on.  I turned it off to see if it was part of the problem.
>
> qhost
>
> HOSTNAME             ARCH       NPROC  LOAD   MEMTOT   MEMUSE   SWAPTO   SWAPUS
> -------------------------------------------------------------------------------
> global               -              -     -        -        -        -        -
> aslt1.transmeta.com  i686           2  0.08     2.0G   708.6M     1.9G   163.3M
> aslt2.transmeta.com  i686           2  0.87     3.8G   936.1M     1.9G   192.1M
> captain.transmeta.com i686           2  0.21  1006.6M   303.7M     2.0G    30.9M
> chris-linux.transmeta.com i686           2  0.01  1006.6M   328.5M     2.0G   179.6M
> dioxide.transmeta.com sun4u         12  0.03    12.0G     1.6G    17.9G      0.0
> l0023001.transmeta.com i686           2  0.09  1006.6M   171.6M     2.0G    25.6M
> l0023004.transmeta.com i686           2  0.17  1006.6M   170.4M     2.0G    25.8M
> l0023013.transmeta.com i686           2  0.00  1006.6M   175.2M     2.0G    26.3M
> l0023017.transmeta.com i686           2  0.00  1006.6M   183.2M     2.0G    28.0M
> l0023018.transmeta.com i686           2  0.02  1006.6M   177.1M     2.0G    27.6M
> l0023021.transmeta.com i686           2  0.07  1006.6M   177.6M     2.0G    28.9M
> l0023037.transmeta.com i686           2  0.00  1006.6M   175.6M     2.0G    28.2M
> l0023041.transmeta.com i686           2  0.00  1006.6M   175.1M     2.0G    20.0M
> l0023046.transmeta.com i686           2  1.65  1006.6M   563.4M     2.0G   514.6M
> op240-000.transmeta.com i686           2  0.00     3.9G   292.3M     7.9G      0.0
> op240-001.transmeta.com i686           2  0.00     3.9G   331.3M     7.9G    12.0K
> op240-002.transmeta.com i686           2  0.02     3.9G   317.4M     7.9G      0.0
> op240-003.transmeta.com i686           2  1.00     3.9G     2.1G     7.9G     3.9M
> op240-004.transmeta.com i686           2  0.00     3.9G   286.0M     7.9G     4.0K
> op240-005.transmeta.com i686           2  4.00     3.9G   233.3M     7.9G      0.0
> op240-006.transmeta.com i686           2  0.00     3.9G   342.0M     7.9G     8.0K
> op240-007.transmeta.com i686           2  0.01     3.9G   403.0M     7.9G      0.0
> op240-008.transmeta.com i686           2  0.00     3.9G   234.0M     7.9G      0.0
> op240-009.transmeta.com i686           2  1.01     3.9G   308.3M     7.9G      0.0
> op240-010.transmeta.com i686           2  0.00     3.9G   412.4M     7.9G   764.0K
> op240-011.transmeta.com i686           2  0.00     3.9G   231.8M     7.9G      0.0
> op240-012.transmeta.com i686           2  0.00     3.9G   226.2M     7.9G      0.0
> op240-013.transmeta.com i686           2  0.00     3.9G   226.6M     7.9G      0.0
> op240-014.transmeta.com i686           2  0.02     3.9G   230.0M     7.9G      0.0
> op240-015.transmeta.com i686           2  0.03     3.9G   226.5M     7.9G      0.0
> op240-016.transmeta.com i686           2  0.02     3.9G   216.5M     7.9G      0.0
> op240-017.transmeta.com i686           2  0.00     3.9G   222.7M     7.9G      0.0
> op240-018.transmeta.com i686           2  0.00     3.9G   225.3M     7.9G      0.0
> op240-019.transmeta.com i686           2  0.00     3.9G   225.7M     7.9G      0.0
> op240-020.transmeta.com i686           2  0.00     3.9G   222.9M     7.9G      0.0
> op240-021.transmeta.com i686           2  0.00     3.9G   226.3M     7.9G      0.0
> op240-022.transmeta.com i686           2  0.00     3.9G   267.8M     7.9G    20.0K
> op240-023.transmeta.com i686           2  0.00     3.9G   230.2M     7.9G      0.0
> op240-024.transmeta.com i686           2  0.00     3.9G   220.2M     7.9G      0.0
> op240-025.transmeta.com i686           2  0.00     3.9G   238.6M     7.9G      0.0
> op240-026.transmeta.com i686           2  0.00     3.9G   242.1M     7.9G      0.0
> op240-027.transmeta.com i686           2  0.00     3.9G   227.3M     7.9G      0.0
> op240-028.transmeta.com i686           2  0.02     3.9G   215.6M     7.9G      0.0
> op240-029.transmeta.com i686           2  0.00     3.9G   222.6M     7.9G      0.0
> opteron-test.transmeta.com i686           2  0.00     3.9G   971.1M     7.9G    18.8M
> ruthenium.transmeta.com sun4u          2  0.25     2.0G   474.0M   512.0M      0.0
> technetium.transmeta.com sun4u          2  0.32     1.0G   143.0M     1.2G      0.0
>
> qstat -F -q common.op240-025
>
> queuename            qtype used/tot. load_avg arch      states
> ----------------------------------------------------------------------------
> common.op240-025     BIP   0/2       0.01     i686
> 	gc:license_a=5.000000
> 	gc:arcadia=4.000000
> 	gc:hspice=6.000000
> 	gc:vampire-drc=6.000000
> 	gc:vampire-lvs=6.000000
> 	gc:solid=190.000000
> 	gc:simplex_v_storm=2.000000
> 	gc:dynacore=9.000000
> 	hl:arch=i686
> 	hl:num_proc=2.000000
> 	hl:load_avg=0.010000
> 	hl:load_short=0.050000
> 	hl:load_medium=0.010000
> 	hl:load_long=0.000000
> 	hl:np_load_avg=0.005000
> 	hl:np_load_short=0.025000
> 	hl:np_load_medium=0.005000
> 	hl:np_load_long=0.000000
> 	hl:mem_free=3.65G
> 	hl:mem_total=3.88G
> 	hl:swap_free=7.87G
> 	hl:swap_total=7.87G
> 	hl:virtual_free=11.52G
> 	hl:virtual_total=11.75G
> 	hl:mem_used=239.03M
> 	hl:swap_used=0.000000
> 	hl:virtual_used=239.03M
> 	hv:swap_rsvd=0.000000
> 	hv:swap_rate=0.000000
> 	qc:slots=2.000000
> 	qf:s_vmem=infinity
> 	qf:h_vmem=infinity
> 	qf:s_fsize=infinity
> 	qf:h_fsize=infinity
> 	hl:cpu=0.600000
> 	hv:systype=0.000000
> 	hv:glibc=0.000000
> 	hl:cpuf=100.000000
> 	hl:relative_speed=5.000000
> 	hl:bitsize=32-bit
> 	hl:os=Linux
> 	hl:platform=i386
> 	hl:release=2.4.20-18.7smp
> 	hl:vendor-rel=7.3
> 	hl:cpuinfo=_AMD_Opteron(tm)_Processor_248_
> 	hc:exclusive=1.000000
> 	hc:running=50.000000
> 	qf:qname=common.op240-025
> 	qf:hostname=op240-025.transmeta.com
> 	qf:tmpdir=/tmp
> 	qv:calendar=NONE
> 	qf:seq_no=13000.000000
> 	qf:rerun=1.000000
> 	qf:s_rt=infinity
> 	qf:h_rt=infinity
> 	qf:s_cpu=infinity
> 	qf:h_cpu=infinity
> 	qf:s_data=infinity
> 	qf:h_data=infinity
> 	qf:s_stack=infinity
> 	qf:h_stack=infinity
> 	qf:s_core=infinity
> 	qf:h_core=infinity
> 	qf:s_rss=infinity
> 	qf:h_rss=infinity
> 	qf:min_cpu_interval=00:05:00
> 	qf:priority=common
>
> qalter -w v 13983
>
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.aslt1" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.aslt2" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023001" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023004" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023013" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023017" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023018" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023021" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023037" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023041" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.l0023046" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-000" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-001" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-002" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-003" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-004" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-005" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-006" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-007" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-008" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-009" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-010" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-011" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-012" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-013" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-014" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-015" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-016" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-017" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-018" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-019" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-020" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-021" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-022" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-023" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-024" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-025" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-026" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-027" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-028" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.op240-029" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.opteron-test" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.ruthenium" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "common.technetium" because it offers only hv:os=none
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.aslt1" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.aslt2" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023001" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023004" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023013" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023017" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023018" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023021" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023037" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023041" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.l0023046" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-000" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-001" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-002" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-003" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-004" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-005" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-006" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-007" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-008" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-009" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-010" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-011" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-012" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-013" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-014" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-015" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-016" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-017" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-018" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-019" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-020" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-021" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-022" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-023" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-024" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-025" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-026" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-027" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-028" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.op240-029" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.opteron-test" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.ruthenium" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "critical.technetium" because it offers only qf:priority=critical
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.aslt1" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.aslt2" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023001" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023004" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023013" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023017" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023018" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023021" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023037" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023041" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.l0023046" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-000" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-001" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-002" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-003" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-004" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-005" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-006" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-007" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-008" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-009" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-010" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-011" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-012" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-013" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-014" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-015" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-016" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-017" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-018" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-019" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-020" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-021" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-022" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-023" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-024" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-025" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-026" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-027" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-028" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.op240-029" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.opteron-test" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.ruthenium" because it offers only qf:priority=trivial
> Job 13983 (-l priority=common,os=Linux) cannot run in queue "trivial.technetium" because it offers only qf:priority=trivial
> verification: no suitable queues
>
>
> >
> > --- Kirk Patton <kpatton at transmeta.com> wrote:
> > > Hello Andy,
> > >
> > > We had another failure to schedule jobs.  We first
> > > noticed the issue
> > > after a user submitted and then deleted about 4000
> > > jobs.  After that,
> > > not further scheduling took place.
> > >
> > > I collected some output from several commands.
> > >
> > > We may be able to repeate this issue.  I can ask the
> > > user to try.
> > > Is there any other data I could collect that would
> > > be usefull in tracking this down?
> > >
> > > Kirk
> > >
> > >
> > > HOSTNAME             ARCH       NPROC  LOAD   MEMTOT
> > >   MEMUSE   SWAPTO   SWAPUS
> > >
> > -------------------------------------------------------------------------------
> > > global               -              -     -        -
> > >        -        -        -
> > > aslt1.transmeta.com  i686           2  1.13     2.0G
> > >   301.0M     1.9G   165.0M
> > > aslt2.transmeta.com  i686           2  0.77     3.8G
> > >   822.1M     1.9G   181.9M
> > > captain.transmeta.com i686           2  0.03
> > > 1006.6M   259.5M     2.0G    65.0M
> > > chris-linux.transmeta.com i686           2  0.03
> > > 1006.6M   332.4M     2.0G   143.1M
> > > dioxide.transmeta.com sun4u         12  0.12
> > > 12.0G     1.8G    17.9G      0.0
> > > l0023001.transmeta.com i686           2  0.00
> > > 1006.6M    36.9M     2.0G    25.3M
> > > l0023004.transmeta.com i686           2  0.00
> > > 1006.6M    42.2M     2.0G    25.2M
> > > l0023013.transmeta.com i686           2  0.00
> > > 1006.6M    38.0M     2.0G    26.2M
> > > l0023017.transmeta.com i686           2  0.00
> > > 1006.6M    46.1M     2.0G    27.9M
> > > l0023018.transmeta.com i686           2  1.05
> > > 1006.6M   404.3M     2.0G   257.1M
> > > l0023021.transmeta.com i686           2  0.00
> > > 1006.6M    40.1M     2.0G    28.7M
> > > l0023037.transmeta.com i686           2  0.27
> > > 1006.6M    45.1M     2.0G    32.1M
> > > l0023041.transmeta.com i686           2  0.02
> > > 1006.6M    43.3M     2.0G    19.8M
> > > l0023046.transmeta.com i686           2  1.73
> > > 1006.6M   698.5M     2.0G   865.1M
> > > op240-000.transmeta.com i686           2  0.00
> > > 3.9G   286.9M     7.9G      0.0
> > > op240-001.transmeta.com i686           2  0.00
> > > 3.9G   315.3M     7.9G     4.0K
> > > op240-002.transmeta.com i686           2  0.00
> > > 3.9G   308.3M     7.9G      0.0
> > > op240-003.transmeta.com i686           2  1.03
> > > 3.9G     1.6G     7.9G    20.0K
> > > op240-004.transmeta.com i686           2  0.00
> > > 3.9G   276.9M     7.9G     4.0K
> > > op240-005.transmeta.com i686           2  0.00
> > > 3.9G   220.1M     7.9G      0.0
> > > op240-006.transmeta.com i686           2  0.00
> > > 3.9G   333.0M     7.9G     8.0K
> > > op240-007.transmeta.com i686           2  0.00
> > > 3.9G   372.8M     7.9G      0.0
> > > op240-008.transmeta.com i686           2  0.00
> > > 3.9G   230.9M     7.9G      0.0
> > > op240-009.transmeta.com i686           2  0.00
> > > 3.9G   228.7M     7.9G      0.0
> > > op240-010.transmeta.com i686           2  0.00
> > > 3.9G   652.7M     7.9G   764.0K
> > > op240-011.transmeta.com i686           2  0.00
> > > 3.9G   217.1M     7.9G      0.0
> > > op240-012.transmeta.com i686           2  0.00
> > > 3.9G   211.8M     7.9G      0.0
> > > op240-013.transmeta.com i686           2  0.00
> > > 3.9G   213.3M     7.9G      0.0
> > > op240-014.transmeta.com i686           2  0.13
> > > 3.9G   214.3M     7.9G      0.0
> > > op240-015.transmeta.com i686           2  0.00
> > > 3.9G   208.9M     7.9G      0.0
> > > op240-016.transmeta.com i686           2  0.00
> > > 3.9G   209.5M     7.9G      0.0
> > > op240-017.transmeta.com i686           2  0.00
> > > 3.9G   213.4M     7.9G      0.0
> > > op240-018.transmeta.com i686           2  0.00
> > > 3.9G   205.5M     7.9G      0.0
> > > op240-019.transmeta.com i686           2  0.01
> > > 3.9G   207.2M     7.9G      0.0
> > > op240-020.transmeta.com i686           2  0.00
> > > 3.9G   211.5M     7.9G      0.0
> > > op240-021.transmeta.com i686           2  0.00
> > > 3.9G   210.6M     7.9G      0.0
> > > op240-022.transmeta.com i686           2  0.00
> > > 3.9G   272.5M     7.9G    20.0K
> > > op240-023.transmeta.com i686           2  0.01
> > > 3.9G   203.5M     7.9G      0.0
> > > op240-024.transmeta.com i686           2  0.01
> > > 3.9G   204.0M     7.9G      0.0
> > > op240-025.transmeta.com i686           2  0.07
> > > 3.9G   202.2M     7.9G      0.0
> > > op240-026.transmeta.com i686           2  0.00
> > > 3.9G   221.8M     7.9G      0.0
> > > op240-027.transmeta.com i686           2  0.00
> > > 3.9G   209.8M     7.9G      0.0
> > > op240-028.transmeta.com i686           2  0.01
> > > 3.9G   202.4M     7.9G      0.0
> > > op240-029.transmeta.com i686           2  1.05
> > > 3.9G   806.8M     7.9G      0.0
> > > opteron-test.transmeta.com i686           2  0.01
> > >  3.9G   705.7M     7.9G    18.8M
> > > ruthenium.transmeta.com sun4u          2  0.26
> > > 2.0G   472.0M   512.0M      0.0
> > > technetium.transmeta.com sun4u          2  0.27
> > > 1.0G   141.0M     1.2G      0.0
> > >
> > >
> > > job_number:                 10542
> > > submission_time:            Fri Apr 23 16:07:50 2004
> > > owner:                      teesea3d
> > > uid:                        1719
> > > group:                      teesea3d
> > > gid:                        1719
> > > sge_o_home:                 /home/teesea3d
> > > sge_o_path:                 /usr/bin:/bin
> > > sge_o_shell:                /bin/bash
> > > sge_o_workdir:              /export/home/tmp
> > > sge_o_host:                 teese-1
> > > account:                    sge
> > > cwd:                        /export/home/tmp
> > > path_aliases:               /tmp_mnt/ * * /
> > > hard resource_list:         priority=common,os=Linux
> > > mail_list:
> > > teesea3d at teese-1.transmeta.com
> > > notify:                     FALSE
> > > job_name:                   teesejob
> > > restart:                    n
> > > env_list:
> > >
> > PWD=/export/home/tmp,USER=teesea3d,SGE_ROOT=/transmeta/sge,SHLVL=2,_=/transmeta/sge/bin/glinux/qrsh,SHELL=/bin/bash,HOME=/home/teesea3d,PATH=/usr/bin:/bin,QRSH_PORT=teese-1.transmeta.com:35029,QRSH_COMMAND=/project/teese/sbin/teese-slave.pl?--nslaves?9?--sim?sim?--timeout?64800?--sim-mode?translator-heavy?--gdbpath?../astro-gdb?--bundle-server?http://teese-1:12008///?--netport?teese-1/12003?--netport?teese-1/12004?--netport?teese-1/12005?--netport?teese-1/12006?--netport?teese-1/12007?--netport?teese-1/12000?--netport?teese-1/12001?--netport?teese-1/12002
> > > project:                    teese_normal
> > > context:                    SGE_INTERACTIVE=yes
> > > scheduling info:            queue "trivial.aslt2"
> > > dropped because it is temporarily not available
> > >                             queue
> > > "trivial.opteron-test" dropped because it is
> > > temporarily not available
> > >                             queue
> > > "trivial.op240-015" dropped because it is
> > > temporarily not available
> > >                             queue
> > > "trivial.op240-004" dropped because it is
> > > temporarily not available
> > >                             queue
> > > "trivial.op240-028" dropped because it is
> > > temporarily not available
> > >                             queue
> > > "trivial.op240-022" dropped because it is
> > > temporarily not available
> > >                             queue
> > > "trivial.op240-024" dropped because it is
> > > temporarily not available
> > >                             queue
> > > "trivial.op240-005" dropped because it is
> > > temporarily not available
> > >                             queue
> > > "trivial.op240-014" dropped because it is
> > > temporarily not available
> > >                             queue
> > > "trivial.op240-023" dropped because it is
> > > temporarily not available
> > >                             queue
> > > "trivial.op240-010" dropped because it is
> > > temporarily not available
> > >                             queue
> > > "trivial.op240-007" dropped because it is
> > > temporarily not available
> > >                             queue
> > > "trivial.op240-006" dropped because it is
> > > temporarily not available
> > >                             queue
> > > "trivial.op240-009" dropped because it is
> > > temporarily not available
> > >                             queue
> > > "trivial.op240-003" dropped because it is
> > > temporarily not available
> > >                             queue
> > > "trivial.op240-029" dropped because it is
> > > temporarily not available
> > >                             queue
> > > "trivial.op240-017" dropped because it is
> > > temporarily not available
> > >                             queue
> > > "trivial.op240-012" dropped because it is
> > > temporarily not available
> > >                             queue
> > > "trivial.op240-001" dropped because it is
> > > temporarily not available
> > >                             queue "trivial.l0023017"
> > > dropped because it is temporarily not available
> > >
> > === message truncated ===
> >
> >
> >
> >
> >
> > __________________________________
> > Do you Yahoo!?
> > Yahoo! Photos: High-quality 4x6 digital prints for 25?
> > http://photos.yahoo.com/ph/print_splash
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> > For additional commands, e-mail: users-help at gridengine.sunsource.net
> >
>
> --
> Kirk Patton
> Unix Administrator
> Transmeta Inc.
> Tel. 408 919-3055
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail: users-help at gridengine.sunsource.net
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list