[GE users] Altering array task range does not work
iadzhubey at rics.bwh.harvard.edu
Tue Mar 23 19:07:26 GMT 2010
On Tuesday 23 March 2010 01:41:21 pm reuti wrote:
> Am 23.03.2010 um 17:31 schrieb iadzhubey:
> > So you are saying '-t' option (task range) cannot be modified with qalter
> > command?
> yes, this is my conclusion.
Right, and the workaround I am using now is to submit each array with maximum
allowed number of tasks initially (e.g., 75,000) and then qdel extraneous
tasks later when we know how many we actually have. To avoid race conditions
arrays jobs are always initially submitted either with -h option or as jobs
dependent on the previous pipeline stage job, so they are held in the queue
and only restarted when their task range has been successfully adjusted.
I do not know if allocating very large array sizes initially and cutting them
down later causes additional overhead for the qmaster/scheduler but I haven't
noticed any performance degradation yet. We will need to move to array sizes
of up to several million tasks in a few weeks time, at which point this could
become a problem. We'll see.
> > The qalter manpage seems to disagree so at least this should be fixed
> > in the documentation and the (rather odd) syntax deserves documenting
> > too.
> Please file an issue component "man".
I will, thank you for confirming this.
> Couldn't you submit in stages? I mean, the first master job will determine
> the size and submit the array task? The drawback is of course, that array
> job will again be the last in the queue.
The approach I describe above uses this idea, and you can even have your array
job(s) at any position inside a pipe as long as its task range is adjusted
properly by the script run at the previous stage. I had to add all of my
execution nodes to the submit nodes list however, since qdel command refuses
to run on non-execute nodes (but why? qdel does not "execute" anything so
this is counterintuitive).
> You can file another issue for it as an enhancement of an already available
> feature. The restriction would be, that the range can't be lowered than the
> highest already executed (or at least started) task. E.g. you submit 1-100
> and the tasks 1-50 were already started or finished it can't be lowered
> than 50 would be feasible I think.
Perhaps I should file an RFE for this. However, I think exceptions resulting
from specifying already completed (or any otherwise non-existant) task IDs
should be handled by gridengine and not by a user. At least qdel has no
problem dealing with non-existing or vanished task IDs specified by a user at
the command line (even if it will complain a bit).
The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.
To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].
More information about the gridengine-users