[GE users] Dynamically changing the slots value in the queue
bharanitn at yahoo.com
Wed Nov 24 11:13:57 GMT 2010
[ The following text is in the "utf-8" character set. ]
[ Your display is set for the "ISO-8859-10" character set. ]
[ Some characters may be displayed incorrectly. ]
--- On Wed, 24/11/10, reuti <reuti at staff.uni-marburg.de> wrote:
From: reuti <reuti at staff.uni-marburg.de>
Subject: Re: [GE users] Dynamically changing the slots value in the queue
To: users at gridengine.sunsource.net
Date: Wednesday, 24 November, 2010, 2:15 PM
Am 24.11.2010 um 06:03 schrieb llikethat:
> --- On Tue, 23/11/10, reuti <reuti at staff.uni-marburg.de</mc/compose?to=reuti at staff.uni-marburg.de>> wrote:
> From: reuti <reuti at staff.uni-marburg.de</mc/compose?to=reuti at staff.uni-marburg.de>>
> Subject: Re: [GE users] Dynamically changing the slots value in the queue
> To: users at gridengine.sunsource.net</mc/compose?to=users at gridengine.sunsource.net>
> Date: Tuesday, 23 November, 2010, 5:23 PM
> Am 23.11.2010 um 12:31 schrieb llikethat:
> > When i'm running a certain job, the cpu usage is 100% the slots value for this node's queue is 1 by default. Now if i'm running a job which uses only 20% of the cpu is possible to dynamically change the slots value to 5 so that the job uses the maximum available resources?
> I would suggest to investigate, why this job isn't running at 100%.
> 20% which i mentioned was just a rough number, but the actual is like this, certain rendering process does not use all the cores in the machine like if i have 4 cores it uses only 1 core leaving the rest 3 cores free. The idea is if i can identify this usage at runtime then i can start multiple threads depending on the resource availability.
Then the best is to define the slots for a queue equal to the installed cores and submit a serial job as such when they use only one core, other jobs which use more than one core have to be submitted as parallel ones. Each of the parallel jobs should use exactly the granted amount of cores. Whether they use forks or threads.
For OpenMP you can set e.g. in your jobscript: export OMP_NUM_THREADS=$NSLOTS
Thank you for the reply. But we are not working on parallel jobs at all. These are 3d animation renders, so we are using array jobs for submission to the grid.
As you said the better option would be to create different queues with different slots value and submit accordingly. But before starting a job in SGE we will not know if the job is using 1 core or all the 4 cores.
How do i automate the following,
For instance, if i have 2 queues (a) slot4.q (b) slot1.q
if i submit a job to slot1.q and realise that the job is not utilizing all the core for computing, then what would be the better way to migrate the job to slot4.q or vice versa
If i can get an approach to this then i can create multiple queues with different slot combination and migrate them automatically after knowing the cpu usage.
To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net</mc/compose?to=users-unsubscribe at gridengine.sunsource.net>].
More information about the gridengine-users