[GE users] Dynamically changing the slots value in the queue

llikethat bharanitn at yahoo.com
Wed Nov 24 12:17:30 GMT 2010


    [ The following text is in the "iso-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Why is it so hard to read your replies - they are always quoted as being part of the original message. It's coming from Yahoo AFAICS - is it their feature?

I think it is a Yahoo feature, not sure. but now im not specifying the quote mentioned in the threads.

Why? What is the documentation of your application saying. In the worst case some can be forced by using the mentioned variable to run in serial. If it's only using threads from time to time you are out of luck anyway. In this case it would be best to force all to run serial but run four of them at a time. In total you lose less computing time.

I'm sorry i didn't understand this. If i'm using all the slots to run the job how will i lose on the computing time.

No, migration between queues is not implemented.

NB: You could setup and abuse the checkpointing environemnt for this, but I don't think it would be a good solution.

I tried reading about checkpointing - but the application we use does not support checkpointing. I was also not sure how to implement or make it work for a 3d animation package.

Thanks,

--- On Wed, 24/11/10, reuti <reuti at staff.uni-marburg.de> wrote:

From: reuti <reuti at staff.uni-marburg.de>
Subject: Re: [GE users] Dynamically changing the slots value in the queue
To: users at gridengine.sunsource.net
Date: Wednesday, 24 November, 2010, 5:15 PM

Am 24.11.2010 um 12:13 schrieb llikethat:

> --- On Wed, 24/11/10, reuti <reuti at staff.uni-marburg.de</mc/compose?to=reuti at staff.uni-marburg.de>> wrote:
>
> From: reuti <reuti at staff.uni-marburg.de</mc/compose?to=reuti at staff.uni-marburg.de>>
> Subject: Re: [GE users] Dynamically changing the slots value in the queue
> To: users at gridengine.sunsource.net</mc/compose?to=users at gridengine.sunsource.net>
> Date: Wednesday, 24 November, 2010, 2:15 PM
>
> Am 24.11.2010 um 06:03 schrieb llikethat:
>
> > --- On Tue, 23/11/10, reuti <reuti at staff.uni-marburg.de</mc/compose?to=reuti at staff.uni-marburg.de>> wrote:
> >
> > From: reuti <reuti at staff.uni-marburg.de</mc/compose?to=reuti at staff.uni-marburg.de>>
> > Subject: Re: [GE users] Dynamically changing the slots value in the queue
> > To: users at gridengine.sunsource.net</mc/compose?to=users at gridengine.sunsource.net>
> > Date: Tuesday, 23 November, 2010, 5:23 PM
> >
> > Hi,
> >
> > Am 23.11.2010 um 12:31 schrieb llikethat:
> >
> > > When i'm running a certain job, the cpu usage is 100% the slots value for this node's queue is 1 by default. Now if i'm running a job which uses only 20% of the cpu is possible to dynamically change the slots value to 5 so that the job uses the maximum available resources?
> >
> > I would suggest to investigate, why this job isn't running at 100%.
> >
> > 20% which i mentioned was just a rough number, but the actual is like this, certain rendering process does not use all the cores in the machine like if i have 4 cores it uses only 1 core leaving the rest 3 cores free. The idea is if i can identify this usage at runtime then i can start multiple threads depending on the resource availability.
>
> Then the best is to define the slots for a queue equal to the installed cores and submit a serial job as such when they use only one core, other jobs which use more than one core have to be submitted as parallel ones. Each of the parallel jobs should use exactly the granted amount of cores. Whether they use forks or threads.
>
> For OpenMP you can set e.g. in your jobscript: export OMP_NUM_THREADS=$NSLOTS
>
> -- Reuti
>
> Thank you for the reply. But we are not working on parallel jobs at all. These are 3d animation renders, so we are using array jobs for submission to the grid.

Why is it so hard to read your replies - they are always quoted as being part of the original message. It's coming from Yahoo AFAICS - is it their feature?


> As you said the better option would be to create different queues with different slots value and submit accordingly. But before starting a job in SGE we will not know if the job is using 1 core or all the 4 cores.

Why? What is the documentation of your application saying. In the worst case some can be forced by using the mentioned variable to run in serial. If it's only using threads from time to time you are out of luck anyway. In this case it would be best to force all to run serial but run four of them at a time. In total you lose less computing time.


> How do i automate the following,
>
> For instance, if i have 2 queues (a) slot4.q (b) slot1.q
>
> if i submit a job to slot1.q and realise that the job is not utilizing all the core for computing, then what would be the better way to migrate the job to slot4.q or vice versa

No, migration between queues is not implemented.

NB: You could setup and abuse the checkpointing environemnt for this, but I don't think it would be a good solution.


> If i can get an approach to this then i can create multiple queues with different slot combination and migrate them automatically after knowing the cpu usage.
>
>
>
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=298261
>
> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net</mc/compose?to=users-unsubscribe at gridengine.sunsource.net>].
>

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=298296

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net</mc/compose?to=users-unsubscribe at gridengine.sunsource.net>].





More information about the gridengine-users mailing list