[GE users] Querying available slots with a resource string

Daniel Templeton Dan.Templeton at Sun.COM
Thu Nov 8 23:52:17 GMT 2007


    [ The following text is in the "ISO-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Brad,

Did I mention that you can submit a slot range with a PE?

Is there a reason you're separating batch from interactive via projects 
instead of by queue type?  Even if you do it by projects, can you 
additionally do it by queue type?  If so, you can probably awk out the 
queues that don't support batch jobs, i.e. queues whose type doesn't 
include "B".

Daniel

Brad Falch wrote:
> Daniel,
>
> I figured out how to do it by using qstat instead:
>
> qselect -l os_version=WS4.0,amd64=1 | xargs qstat -F -q | awk -F= ' /^.*qc:slots/ { sum+=$2 } END { print sum } '
> However, I now have another problem.  We use projects to call out some hosts for interactive and some for batch sessions.  These jobs will go to batch sessions, so I need a way to distinguish between them.  Any ideas there?  (Looks like -P is not available to qselect or qstat, only the submission and admin commands.)
>
> Thanks,
>
> Brad
>
> ----- Original Message ----
> From: Daniel Templeton <Dan.Templeton at Sun.COM>
> To: users at gridengine.sunsource.net
> Sent: Thursday, November 8, 2007 4:38:21 PM
> Subject: Re: [GE users] Querying available slots with a resource string
>
> Brad,
>
> Duh.  Reuti pointed that out to me, but I hadn't had enough coffee yet 
> to comprehend what he meant.  Yes.  Ignore my previous "helpful"
>  advice.
>
> qstat -f -l resource=value will show you the queues (including 
> free/total slots) with those resources, as well as all jobs running in 
> those queues and all pending jobs.  You'll have to parse out and sum up
>  
> the free slot count.  The -xml switch might make the output easier for 
> you to parse.
>
> I can't come up with a simpler solution off the top of my head.
>
> Daniel
>
> Brad Falch wrote:
>   
>> Daniel,
>>
>> Looks like the one-line script only shows the full number of slots
>>     
>  that fit that description, not the slots available for running.  Is that
>  correct?  I wanted to see how many are open.
>   
>> Thanks,
>>
>> Brad
>>
>> ----- Original Message ----
>> From: Daniel Templeton <Dan.Templeton at Sun.COM>
>> To: users at gridengine.sunsource.net
>> Sent: Thursday, November 8, 2007 3:28:41 PM
>> Subject: Re: [GE users] Querying available slots with a resource
>>     
>  string
>   
>> Brad,
>>
>> Reuti sent me a cleaner way to write the script:
>>
>> qselect -l resource=value | xargs qconf -sq | awk ' /^slots/ {
>>     
>  sum+=$2
>   
>>  } 
>> END { print sum } '
>>
>> Daniel
>>
>> Brad Falch wrote:
>>   
>>     
>>> Daniel,
>>>
>>> Thanks for explaining and the shell script.   I'll give it a shot.
>>>
>>> -Brad
>>>
>>> ----- Original Message ----
>>> From: Daniel Templeton <Dan.Templeton at Sun.COM>
>>> To: users at gridengine.sunsource.net
>>> Sent: Thursday, November 8, 2007 1:28:08 PM
>>> Subject: Re: [GE users] Querying available slots with a resource
>>>     
>>>       
>>  string
>>   
>>     
>>> Brad,
>>>
>>> There are two answers.  The first is to use the PE functionality
>>>       
>  that
>   
>>>     
>>>       
>>  
>>   
>>     
>>> SGE provides.  I assume you're talking about a parallel job since
>>>  you've 
>>> got a single job that's using multiple CPUs simultaneously.  (I know
>>>  you 
>>> know some of this already, but bear with me.)  When you request a PE
>>>     
>>>       
>>  at
>>   
>>     
>>>  
>>> job submission time, you can request a number of slots (-pe name
>>>     
>>>       
>>  slots,
>>   
>>     
>>>  
>>> e.g. -pe make 4).  The job will be accepted, but it will not be 
>>> scheduled until the requested number of slots is available.  To
>>>     
>>>       
>>  prevent
>>   
>>     
>>>  
>>> the job from being accepted, you can use the "-now y" switch to
>>>     
>>>       
>>  request
>>   
>>     
>>>  
>>> that if the job cannot be immediately scheduled, that the job
>>>  submission 
>>> be rejected.  You also have the ability to request a slot range
>>>     
>>>       
>>  instead
>>   
>>     
>>>  
>>> of a fixed number (-pe name min-max, e.g. -pe make 4-16).  When 
>>> requesting a slot range, the scheduler will give you the largest
>>>     
>>>       
>>  number
>>   
>>     
>>>  
>>> of slots in that range that is possible.  By submitting a slot range
>>>       
>  
>   
>>> with -now, you let the scheduler assign you as many slots as it can,
>>>  and 
>>> if it can't meet your minimum requirements, the submission will be
>>>  rejected.
>>>
>>> The second answer is qselect and some shell scripting, something
>>>     
>>>       
>>  like:
>>   
>>     
>>> #!/bin/sh
>>> sum=0
>>> for q in `qselect -l $1=$2`
>>> do
>>>    tmp=`qconf -sq $q | grep slots | awk '{ print $2 }'`
>>>    sum=`expr $sum + $tmp`
>>> done
>>> echo $sum
>>>
>>> Daniel
>>>
>>> Brad Falch wrote:
>>>   
>>>     
>>>       
>>>> Hello, All,
>>>>
>>>> Please forgive this newbie question.  From the user perspective, is
>>>>     
>>>>       
>>>>         
>>>  there a way to report the number of slots available on a Grid,
>>>       
>  given
>   
>>>     
>>>       
>>  a
>>   
>>     
>>>  particular resource string?  I'm hoping there is something built
>>>     
>>>       
>>  into
>>   
>>     
>>>  SGE, but any suggestions for scripts are welcome.
>>>   
>>>     
>>>       
>>>> I run distributed processing jobs on a cluster that is
>>>>       
>>>>         
>>  heterogeneous,
>>   
>>     
>>>>     
>>>>       
>>>>         
>>>  and I only want to launch the job if a certain number of slots (all
>>>  requested) are available.  I would also like to use this command
>>>  interactively to see what is available and decide how many slots to
>>>     
>>>       
>>  use.  (I
>>   
>>     
>>>  also don't want to wait for the job to start.  I do understand
>>>       
>  there
>   
>>>     
>>>       
>>  is a
>>   
>>     
>>>  way to submit so it only starts once all slots are available.)
>>>   
>>>     
>>>       
>>>> Thanks,
>>>>
>>>> Brad
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> __________________________________________________
>>>> Do You Yahoo!?
>>>> Tired of spam?  Yahoo! Mail has the best spam protection around 
>>>> http://mail.yahoo.com 
>>>>
>>>>
>>>>       
>>>>         
>   ---------------------------------------------------------------------
>   
>>   
>>     
>>>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>>> For additional commands, e-mail:
>>>>         
>  users-help at gridengine.sunsource.net
>   
>>>>   
>>>>     
>>>>       
>>>>         
>  ---------------------------------------------------------------------
>   
>>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>>
>>>
>>>
>>>
>>>
>>> __________________________________________________
>>> Do You Yahoo!?
>>> Tired of spam?  Yahoo! Mail has the best spam protection around 
>>> http://mail.yahoo.com 
>>>
>>>
>>>       
>  ---------------------------------------------------------------------
>   
>>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>>
>>>   
>>>     
>>>       
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>
>>
>>
>>
>>
>> __________________________________________________
>> Do You Yahoo!?
>> Tired of spam?  Yahoo! Mail has the best spam protection around 
>> http://mail.yahoo.com 
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>
>>   
>>     
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail: users-help at gridengine.sunsource.net
>
>
>
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around 
> http://mail.yahoo.com 
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail: users-help at gridengine.sunsource.net
>
>   

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list