AW: [GE users] regex in complexes

Shannon V. Davidson svdavidson at swbell.net
Fri Sep 3 16:42:03 BST 2004


    [ The following text is in the "ISO-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Mark,

I originally ran my tests by putting the license definitions in the 
complex_values of the host, rather than as load values.  This works as 
we expect:

$ qconf -se pogo -se appro | egrep "^host|^complex"
hostname              pogo.hpc-mo.com
complex_values        license="fire7"
hostname              appro.hpc-mo.com
complex_values        license=""

$ qstat -F license
queuename                      qtype used/tot. load_avg arch          states
----------------------------------------------------------------------------
all.q at appro.hpc-mo.com         BIP   0/2       1.01     lx24-amd64
        hf:license=""
----------------------------------------------------------------------------
all.q at pogo.hpc-mo.com          BIP   0/2       0.05     lx24-x86
        hf:license="fire7"

$ qselect -l lic="*fire7*"
all.q at appro.hpc-mo.com

$ qselect -l lic="*junk*"
all.q at appro.hpc-mo.com
all.q at pogo.hpc-mo.com

However, as you've discovered, when the license values are stored in 
load values, it doesn't work.

$ qstat -F license
queuename                      qtype used/tot. load_avg arch          states
----------------------------------------------------------------------------
all.q at appro.hpc-mo.com         BIP   0/2       1.00     lx24-amd64
        hl:license=lic2+lic3+lic4
----------------------------------------------------------------------------
all.q at pogo.hpc-mo.com          BIP   0/2       0.06     lx24-x86
        hl:license=lic1+lic2

$ qselect -l lic="*fire7*"
no queues remaining after selection

$ qselect -l lic="*junk*"
no queues remaining after selection

The problem is the qselect command itself.  According to the man page, 
it doesn't consider load values.

       -l resource[=value],...
              Defines  the  resources to be granted by the queues which 
should
              be included in the queue list output. Matching is  
performed  on
              queues  based  on  non-mutable resource availability 
information
              only.  That means load values are always ignored except 
the  so-
              called static load values (i.e. "arch", "num_proc", 
"mem_total",
              "swap_total" and "virtual_total") ones. Also consumbale 
utiliza-
              tion is ignored.

There are a couple of related issues in Issuezilla that deal 
specifically with this problem:

Issue 771 made qstat/qselect to not consider load values:

http://gridengine.sunsource.net/issues/show_bug.cgi?id=771

Issue 772 is an enhancement request to add a -full switch to 
qstat/qselect to allow the user to decide if load values should be 
considered.

http://gridengine.sunsource.net/issues/show_bug.cgi?id=772

I would like to see issue 772 implememented soon.  I have several 
customers (still using SGE 5.3) with customized scripts which rely on 
this feature.

BTW - specifying the license resource on qsub still works as you expect.

Cheers,
Shannon


Olesen, Mark wrote:

>>I took a look at the complex code and ran a few tests and it looks to me
>>like your license complex idea will work if you use:
>>
>>    qselect -l license="*lic3*"
>>
>>to return the list of queues where the license string does not include
>>the string lic3, and:
>>
>>    qsub -l license="*lic3*" ...
>>
>>to submit a job to any queue which is currently not using lic3.
>>
>>Of course, there is some lag between when the job gets started and when
>>the load sensor reports the license, but I assume you already have a
>>mechanism to handle that.
>>
>>AFAICT, both == and != work for RESTRING complex types.  The code is
>>doing a shell wildcard type match between the load value string and the
>>pattern argument.
>>    
>>
>
>Shannon,
>
>I don't seem to be as fortunate with getting things to work.
>
>My config:
>
>$ qconf -sc | grep lic
>license lic        RESTRING    !=    YES         NO         NONE  0 
>
>
>The loads are getting reported:
>$ qstat -F | grep lic
>        hl:license=0
>        hl:license=0
>        hl:license=0
>        hl:license=0
>        hl:license=0
>        hl:license=0
>        hl:license=0
>        hl:license=0
>        hl:license=0
>        hl:license=fire7
>        hl:license=0
>        hl:license=fire7
>        hl:license=0
>
>But the selection seems to be a bit wonky:
>
>$ qselect -l license="*junk*"
>no queues remaining after selection
>
>$ qselect -l license="*fire*"
>no queues remaining after selection
>
>
>Any ideas?
>
>/mark
>
>
>Dr. Mark Olesen
>Thermofluid Dynamics Analyst
>ArvinMeritor Light Vehicle Systems
>Zeuna Staerker GmbH & Co. KG
>Biberbachstr. 9
>D-86154 Augsburg, GERMANY
>tel: +49 (821) 4103 - 862
>fax: +49 (821) 4103 - 7862
>Mark.Olesen at ArvinMeritor.com
>
>
>  
>
>>Cheers,
>>Shannon
>>
>>
>>Olesen, Mark wrote:
>>
>>    
>>
>>>Despite the explicit information from complex(5) that the only valid
>>>operator for string type attributes is "==", I attempted to use the
>>>following complex, in the vague hope that the documentation was too
>>>conservative:
>>>
>>>#name               shortcut   type        relop requestable consumable
>>>default  urgency
>>>#------------------------------------------------------------------------
>>>      
>>>
>>---
>>    
>>
>>>-------------
>>>license             lic        RESTRING    !=    YES         NO
>>>      
>>>
>>NONE
>>    
>>
>>>0
>>>
>>>
>>>As expected, this failed miserably.
>>>
>>>I would like to have the functionality of a "!=" operator.
>>>Is there an obscure workaround to give me what I want?
>>>What would it take to have the "!=" operator added for string operations?
>>>(Andy?).
>>>
>>>My load sensor will report something like the following:
>>>
>>>	machine:license:lic1+lic2+lic4
>>>
>>>Then I could do something like 'qselect -l license=lic3' to find queues
>>>where the license lic3 is *not* currently in use.
>>>
>>>IMO, a very useful and flexible hack would be to disallow the '!'
>>>      
>>>
>>character
>>>from complex names and use it to reverse the request logic,
>>    
>>
>>>eg,
>>>	qselect -l arch=lx24-x86	# select linux architecture
>>>	qselect -l arch!=lx24-x86	# select non-linux architecture(s)
>>>
>>>
>>>Regards,
>>>/mark
>>>
>>>Dr. Mark Olesen
>>>Thermofluid Dynamics Analyst
>>>ArvinMeritor Light Vehicle Systems
>>>Zeuna Staerker GmbH & Co. KG
>>>Biberbachstr. 9
>>>D-86154 Augsburg, GERMANY
>>>tel: +49 (821) 4103 - 862
>>>fax: +49 (821) 4103 - 7862
>>>Mark.Olesen at ArvinMeritor.com
>>>?
>>>
>>>---------------------------------------------------------------------
>>>To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>>For additional commands, e-mail: users-help at gridengine.sunsource.net
>>>
>>>
>>>
>>>
>>>      
>>>
>>--
>>___________________________________________
>>
>>Shannon V. Davidson <svdavidson at swbell.net>
>>Senior Software Engineer           Raytheon
>>636-479-7465 office        443-383-0331 fax
>>___________________________________________
>>
>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>For additional commands, e-mail: users-help at gridengine.sunsource.net
>>    
>>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>For additional commands, e-mail: users-help at gridengine.sunsource.net
>
>
>  
>




-- 
___________________________________________

Shannon V. Davidson <svdavidson at swbell.net>
Senior Software Engineer           Raytheon
636-479-7465 office        443-383-0331 fax
___________________________________________





More information about the gridengine-users mailing list