[GE users] Preemption vs dedicating nodes by group?

Jim Marconnet jmarconnet at knology.net
Wed May 4 17:22:15 BST 2005


    [ The following text is in the "iso-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Sorry if this is a duplicate. My emails recently from work to the group have
been ignored, so I just subscribed using my home email address.

Justus:

Thanks. This helps some. Sorry I'm so dense, but I'm still confused on what
specifically triggers the secondary que instance node to suspend (in this
case kill, due to the added checkpointing settings) its job so the primary
que has that node available immediately. Que subordination keeps additional
jobs from being assigned to a node from a subordinated que, but AFAIK que
subordination does not affect already running jobs. Checkpointing can be set
up to kill (and resubmit) instead of suspending a job, but I'm struggling to
understand what happens that specifically triggers this particular job
suspension, other than possibly oversubscribing the node and using a
load-limit to suspend that job.

You said you did not have to use a load sensor, I assume you mean the load
sensor and host complex resource described in
http://gridengine.sunsource.net/howto/idle.html Since this is what you
apparently did not need to do, then I'm confused as to what you do need to
do to triger it when group2 submits a job using their primary que. Again,
sorry I'm so dense.

To make it easier for you (or anyone else) to help me understand this, just
feel free to fill in the blank:
"The job suspension/kill trigger is _______________________________________
(in 10 words or less - more if you need them!)
which I set up by:
_________________________________________________________________"

Also, do you have an email, a memo, a readme.txt file, a webpage, or
whatever you use now or you will use to educate/instruct your users so they
know how it all works and what they must do and what they must not do to
make this scheme work well for everyone that you could share? Having their
job unexpectedly killed and restarted from the beginning after several days,
weeks, or even months run time could be an unpleasant surprise! And
unexpected, unexplained file corruption would be an even worse! Perhaps you
just name your secondary queue "run_jobs_here_at_your_own_peril.q".

If they figure out that if they simply and conveniently leave off the -ckpt
flag in their qsub command, it could be dangerous to the overall scheme. Who
wants their jobs to be (voluntarily) killed and restarted if they can
conveniently "leave off" that mysterious alphabet-soup flag and thus be
immune from automatic kill/restart?

Thanks again for all the help and especially the patience shared here,
Jim Marconnet

> -----Original Message-----
> From: Justus Loerke [mailto:loerke at molgen.mpg.de]
> Sent: Friday, April 29, 2005 5:42 AM
> To: users at gridengine.sunsource.net
> Subject: Re: [GE users] Preemption vs dedicating nodes by group?
>
>
> Hi Jim,
>
> yes, I have configured all secondary queues open to all users. My first
idea
> was to reconfigure queue access depending on a load sensor (as suggested
by
> Reuti, I believe, in a previous thread) monitoring the jobs submitted by
the
> different groups. This setup, however, saves me the work of having to
> reconfigure queue access all the time.
>
> I have done only some preliminary testing so far, since I have running
jobs
> on my machines and didn't want to disrupt any of these. Suspension and
> restarting of the jobs seems to work fine, but I will have to do some work
> on the file corruption problem on the scripting level first. I am not only
> worried about this, in our case it's a serious problem. We are running
> iterating molecular volume refinement, split into some 100s of single
jobs,
> that read and write to the same files more or less over the whole time the
> jobs is running. If this kind of job is killed and restarted, it will find
a
> file (it has produced ifself before being
> killed) that has a mix of old and new (from the previously started and
> killed job) data and this will in effect kill not only this one job, but
the
> whole iterative process depending on this data (or even worse:
> disrupt the data and introduce errors into our calculations). It's not
that
> hard to work around (using temporary data files), but I haven't had time
to
> do it yet, so far I have only done some testing on the basic queue setup.
>
> I took the idea for this setup from the 'HOWTO configure idle workstations
> for queue work and howto to migrate jobs from workstations'
> on the GE page. The only difference is that I'm really using checkpointing
> for migrating jobs only, the cpu time already spent on this job is wasted
> (although I'm working on user-level checkpointing as well). In my
> understanding, the checkpointing is activated by the -ckpt flag in qsub
and
> triggered by the suspend command, and the checkpointing option 'restart on
> suspend' then leads to migration. Jobs submitted without the ckpt flag
will
> run on primary queue, secondary queues if any primary queues are empty,
but
> will not kicked off the secondary queue if a higher priority job comes
> along; it will only be suspended, as with normal subordinated queues. In
> some cases this may not be a problem, but in my case, I used migration to
> work around the worst case, in which some group1 calculations are idle,
> waiting for one job suspended for weeks on a secondary queue because
group2
> is using all primary queues.
>
> Hope that helps,
> Justus
>
> --
> -------------------------------------------
>
> Dipl. Phys. Justus Loerke
> - UltraStrukturNetzwerk -
> Max Planck Institute for Molecular Genetics Ihnestr. 63-73
> D-14195 Berlin



-- 
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.308 / Virus Database: 266.11.3 - Release Date: 5/3/2005


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list