[GE users] Allow users to limit concurrent array job task execution

heywood heywood at cshl.edu
Thu Nov 19 17:07:45 GMT 2009


Hi,

Thanks. Yes, I could do that as the admin, but the interesting thing about
-tc is that it allows *users" to throttle job arrays. So why not allow
*users* to throttle a set of hundreds of jobs they submit?

We have cluster-savy users here who would like the ability to throttle both
job arrays (so -tc is welecome) and also large sets of jobs.

Todd


On 11/19/09 5:47 AM, "reuti" <reuti at staff.uni-marburg.de> wrote:

> Am 18.11.2009 um 15:51 schrieb heywood:
> 
>> Is it possible to limit the number of running *jobs*? Not tasks of
>> a job
>> array, but letting the user submit a lot of jobs but specify a maximum
>> number that can run at once?
> 
> First let me add to the original issue, that there is also the global
> parameter "max_aj_instances" in `qconf -monf`which will control the
> number of concurrent array job tasks respectively defines an upper
> limit for the -tc (although no error will be output when you specify
> a higher value there).
> 
> ==
> 
> In your case there is a parameter in the scheduler configuration
> `qconf -msconf` called "maxujobs" which also defines it at a global
> level the number of each user's *running* jobs.
> 
> Starting with SGE 6.2u3 you can define different limits per user: we
> need a custom complex e.g. called "jobs" which must be set to "type
> INT", "requestable YES"*, "consumable JOB" and "default 1".
> 
> Second thing is to define a global arbitrary high value for this
> resource (it also serves as the total number of jobs you allow in the
> cluster) in the global exechost configuration `qconf -me global`.
> 
> Now we can setup an RQS, which contains limits for individual users
> or ACLs:
> 
> limit users foo to jobs=10
> limit users {@bar} to jobs=5
> 
> -- Reuti
> 
> *) Maybe used to bypass the limit when one requests jobs=0, but there
> is a bug which will prevent the jobs from being scheduled when it's
> set to "NO" right now.
> 
> 
>> Todd
>> 
>> 
>> On 11/18/09 4:38 AM, "aja" <alena.plestilova at sun.com> wrote:
>> 
>>> Hi,
>>> 
>>> this feature is not documented in 6.2u4. But it's already added to
>>> 6.2u5.
>>> Here is the man qsub input for this feature:
>>> 
>>>      -tc max_running_tasks
>>>           -allow users to limit concurrent array job task  execu-
>>>           tion.  Parameter  max_running_tasks  specifies  maximum
>>>           number of simultaneously running tasks.  For example we
>>>           have  running  SGE  with 10 free slots. We call qsub -t
>>>           1-100 -tc 2  jobscript.  Then  only  2  tasks  will  be
>>>           scheduled to run even when 8 slots are free.
>>> 
>>> Regards,
>>> aja
>>> 
>>> 
>>> On 11/17/09 19:06, svibhore wrote:
>>>> Thanks Reuti, this is great!
>>>> 
>>>> 
>>>> On Tue, Nov 17, 2009 at 9:12 PM, reuti <reuti at staff.uni-marburg.de
>>>> <mailto:reuti at staff.uni-marburg.de>> wrote:
>>>> 
>>>>     Am 17.11.2009 um 15:40 schrieb svibhore:
>>>> 
>>>>> Hello,
>>>>> 
>>>>> Release notes for 6.2u4 has a entry for job throttle.
>>>>> 
>>>>> 
>>>>> Bugs fixed in SGE 6.2u4 since release 6.2u3
>>>>> -------------------------------------------
>>>>> 
>>>>> 2542 6684887 allow users to limit concurrent array job task
>>>>     execution
>>>>> 
>>>>> I do not see any documentation related to this feature, can anyone
>>>>> point me in right direction.
>>>> 
>>>>     In issue 2542 the syntax is mentioned:
>>>> 
>>>>     $ qsub -t 1-20 -tc 5 test.sh
>>>> 
>>>>     to limit it to e.g. 5 and AFAICS it's working - it's just
>>>> missing
>>>>     from the man page. Can you please file a documentation issue?
>>>> 
>>>>     -- Reuti
>>>> 
>>>> 
>>>>> 
>>>>> Regards,
>>>>> Shantanu
>>>> 
>>>>     ------------------------------------------------------
>>>> 
>>>> http://gridengine.sunsource.net/ds/viewMessage.do?
>>>> dsForumId=38&dsMessageId=22
>>>> 7467
>>>> 
>>>> <http://gridengine.sunsource.net/ds/viewMessage.do?
>>>> dsForumId=38&dsMessageId=2
>>>> 27467>
>>>> 
>>>>     To unsubscribe from this discussion, e-mail:
>>>>     [users-unsubscribe at gridengine.sunsource.net
>>>>     <mailto:users-unsubscribe at gridengine.sunsource.net>].
>>>> 
>>>> 
>>> 
>> 
>> ------------------------------------------------------
>> http://gridengine.sunsource.net/ds/viewMessage.do?
>> dsForumId=38&dsMessageId=227689
>> 
>> To unsubscribe from this discussion, e-mail: [users-
>> unsubscribe at gridengine.sunsource.net].
> 
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=227
> 889
> 
> To unsubscribe from this discussion, e-mail:
> [users-unsubscribe at gridengine.sunsource.net].

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=228017

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list