[GE users] Altering array task range does not work

reuti reuti at staff.uni-marburg.de
Tue Mar 23 17:41:21 GMT 2010


Hi,

Am 23.03.2010 um 17:31 schrieb iadzhubey:

> So you are saying '-t' option (task range) cannot be modified with qalter 
> command?

yes, this is my conclusion.


> The qalter manpage seems to disagree so at least this should be fixed 
> in the documentation and the (rather odd) syntax deserves documenting too.

Please file an issue component "man".


> It's a pity if this can't be done. As I said, I found a reasonably simple 
> workaround that works for my purpose but it's a rather awkward one. The reason 
> I need this dynamic adjustment of the task range is that I wrote a program 
> that automatically generates and submits "pipelines" of jobs based on simple 
> template description. Pipelines are common in many applications: output of one 
> program is fed into the next one and so on, just like Unix "pipes". With my 
> script, a user can implement his/her new analysis pipeline in less than 5 
> minutes with no programming involved. However, the input of a program in the 
> middle of a pipeline is not defined until the previous one that generates it 
> has completed. If this is an array job we simply don't know how many tasks it 
> will comprise until we have an output file from the previous stage of a pipe 
> available. Hence, the size of this array job should be determined dynamically, 
> right before it is actually *executed*. It can't be specified correctly at the 
> point when the job is *submitted*.

Couldn't you submit in stages? I mean, the first master job will determine the size and submit the array task? The drawback is of course, that array job will again be the last in the queue.

You can file another issue for it as an enhancement of an already available feature. The restriction would be, that the range can't be lowered than the highest already executed (or at least started) task. E.g. you submit 1-100 and the tasks 1-50 were already started or finished it can't be lowered than 50 would be feasible I think.

-- Reuti


> I would suggest this as a feature request for the next gridengine version.
> 
> --Ivan
> 
> On Tuesday 23 March 2010 04:29:38 am reuti wrote:
>> Am 23.03.2010 um 08:56 schrieb dougalb:
>>> Can you provide the full command you are using?
>>> 
>>> $ qalter 4562 -t 1000-2000 -h s
>> 
>> Great, thx for clarifying this. But I fear it's not the answer the
>> original poster is expecting to change the complete range of the
>> alread submitted jobs as long as they are in "qw" state.
>> 
>> ==
>> 
>> Interesting syntax - not easy to find. It also doesn't correspond with
>> the manpage. It should be mentioned there:
>> 
>> The -t option with `qalter` is only valid for the -h {u|s|...} option,
>> the order of these two options doesn't matter and a wc_job_range_list
>> can be disregarded. In fact it will be used independently. I mean:
>> 
>> $ qalter -h u 917 -t 15-25 918
>> 
>> is valid to change two jobs. The funny thing is:
>> 
>> $ qalter -A sdffd 917 -t 15-25 -h n
>> 
>> is allowed and changes the hold (subrange) and account (for all tasks
>> of course), while:
>> 
>> $ qalter -A sdffd 917 -t 15-25
>> 
>> throws an error. No -t allowed w/o -h, hence it's really an option to -
>> h.
>> 
>> -- Reuti
>> 
>>> That should work.
>>> 
>>> On Mon, Mar 22, 2010 at 10:37 PM, iadzhubey
>>> 
>>> <iadzhubey at rics.bwh.harvard.edu> wrote:
>>>> Any suggestions at all? Anyone, please?
>>>> 
>>>> --Ivan
>>>> 
>>>> On Monday 22 March 2010 10:48:53 am iadzhubey wrote:
>>>>> Hi Reuti,
>>>>> 
>>>>> On Monday 22 March 2010 08:42:44 am reuti wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> Am 20.03.2010 um 08:57 schrieb iadzhubey:
>>>>>>> I am trying to adjust the task range of an already submitted and
>>>>>>> pending array job but cannot figure out how to do this (and if
>>>>>>> it's
>>>>>>> even supported). Qalter manpage lists '-t' as supported option
>>>>>>> but when
>>>>>>> specified with qalter command it is recognised as an (incomplete)
>>>>>>> wc_job_range_list specification instead, which results in a syntax
>>>>>>> error message.
>>>>>> 
>>>>>> you mean the error message like:
>>>>>> 
>>>>>> found lonely '-t 15-20' option (The -t option needs a leading job
>>>>>> name)'?
>>>>> 
>>>>> Yep, this is the error message. I've also tried every other
>>>>> combination of
>>>>> command-line switches to qalter I can think of but to no avail.
>>>>> 
>>>>> Any hope this may actually work? For now, I am using a crude
>>>>> workaround by
>>>>> submitting array jobs with the fixed maximum range (e.g., -t
>>>>> 1-75000) and
>>>>> then letting the first dispatched task of each array qdel all
>>>>> extraneous
>>>>> tasks at runtime. This works but is not an elegant solution.
>>>>> 
>>>>> Best,
>>>>> Ivan
>>>> 
>>>> The information in this e-mail is intended only for the person to
>>>> whom it is
>>>> addressed. If you believe this e-mail was sent to you in error and
>>>> the e-mail
>>>> contains patient information, please contact the Partners
>>>> Compliance HelpLine at
>>>> http://www.partners.org/complianceline . If the e-mail was sent to
>>>> you in error
>>>> but does not contain patient information, please contact the sender
>>>> and properly
>>>> dispose of the e-mail.
>>>> 
>>>> ------------------------------------------------------
>>>> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessage
>>>> Id=250613
>>>> 
>>>> To unsubscribe from this discussion, e-mail:
>>>> [users-unsubscribe at gridengine.sunsource.net ].
>>> 
>>> ------------------------------------------------------
>>> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageI
>>> d=250711
>>> 
>>> To unsubscribe from this discussion, e-mail:
>>> [users-unsubscribe at gridengine.sunsource.net ].
>> 
>> ------------------------------------------------------
>> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=
>> 250715
>> 
>> To unsubscribe from this discussion, e-mail:
>> [users-unsubscribe at gridengine.sunsource.net].
> 
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=250866
> 
> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=250882

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list