[GE users] SGE large memory jobs

mhanby mhanby at uab.edu
Fri Jul 17 22:30:03 BST 2009


    [ The following text is in the "utf-8" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some characters may be displayed incorrectly. ]

I'm confused now :-) Is the mem_free consumed per slot, per compute node or per job?

Say the following job gets submitted to 4 compute node (each node has 8 cores and 24GB)

qsub -pe openmpi 32 -l mem_free=2G

Is this job consuming 64GB of RAM (2GB per slot), 8GB (2GB per compute node) or 2GB (2GB requested for the entire job, which would be 512MB per compute node)


-----Original Message-----
From: m0zes [mailto:adam.tygart at gmail.com] 
Sent: Friday, July 17, 2009 1:51 PM
To: users at gridengine.sunsource.net
Subject: Re: [GE users] SGE large memory jobs

One thing to keep in mind, if you do this, it will make that memory
request to every host utilized by any job. So one job that uses
mem_free=2G, and two hosts, would request 2G of memory from both of
them, effectively requesting 4G which is more than necessary. This
could have some serious implications if your cluster is even
moderately memory hungry or have many jobs that utilize more than one
host.

I hope that what I wrote was a comprehensible to you as it seems to me.

--
Adam

On Thu, Jul 16, 2009 at 15:10, matbradford<matthew.bradford at eds.com> wrote:
> If you put mem_free=<value> into the default sge_requests file, I think
> that should force it to be present in every submission request.
> Possibly.
>
> Cheers,
>
> Mat
>
>>-----Original Message-----
>>From: mbay2002 [mailto:jeff at haferman.com]
>>Sent: 16 July 2009 18:44
>>To: users at gridengine.sunsource.net
>>Subject: Re: [GE users] SGE large memory jobs
>>
>>Thanks for the response.  My biggest concern now is that by making
>>mem_free a consumable, we need to make sure that all jobs request RAM.
>>
>>I'll play around with this, I'm curious to see what happens if RAM is
>>not requested by the job submission once mem_free is made consumable...
>>
>>
>>
>>adary wrote:
>>> Yes, you just need to change the NO to YES to make it consumable.
>>>
>>> For your other question, you need to add mem_free to each host, but
>>you can actually automate it a bit:
>>>
>>> for host in `qhost | sed '1,3d' | awk '{ print $1 }'`; do qconf
> -mattr
>>exechost complex_values mem_free=8G $host; done
>>>
>>> (if all your hosts are 8G hosts ofcourse)
>>>
>>> My recommendation is to always define hosts to 5% less ram than they
>>actually have, and to handle RAM requests accordingly, since some RAM
> is
>>always used by the OS itself
>>>
>>> As for consequences, once you make your mem_free consumable and your
>>jobs start requesting RAM, you need to make sure that all jobs that are
>>sent also request RAM.
>>>
>>> In my case we created a general wrapper for qsub that will always
>>request ram based on certain defaults, and each user can override it
>>with the need of the application.
>>>
>>
>>------------------------------------------------------
>>http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessag
> e
>>Id=207582
>>
>>To unsubscribe from this discussion, e-mail: [users-
>>unsubscribe at gridengine.sunsource.net].
>
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=207608
>
> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].
>

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=207844

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=207860

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list