[GE users] Does oversubscribing nodes actually hurt anything?

Anthony J. Ciani aciani1 at uic.edu
Sun Feb 20 18:09:59 GMT 2005

On Thu, 17 Feb 2005, Reuti wrote:
> If we neglect the limit of the memory/IO bandwidth, it should be the 
> same time in total running 5 jobs one after the other on a single CPU, 
> or let them running all at once.
On old Pentiums, Xeons, and Athlons, you should only let one task per 
CPU.  Some of the newer, higher-end Xeons (depending on MB chipset) and 
Opterons have much higher memory and I/O bandwidth, and can handle several 
tasks per CPU with only about a 5% penalty.  Xeons start having some minor 
difficulty if the tasks start competing for I/O.  Also, depending on the 
amounts of cross-node communication, they may start stepping on eachother.

> Another sidenote: do you have an UPS?
At the least, a surge supressor is a definite, but they can fail, and they 
don't tell you when they have.  If you don't want a UPS (i.e. your compute 
nodes are diskless), then you should really have a power conditioner, 
which can handle several nodes at a time.

> For us the HT on Dual-Xeons seems only to allow 3 jobs at once on a machine,
> more will slow down the other jobs. But this depends on the used software.
> Maybe you can have 4.
The HT processors can perform as well as 1.5 processors, depending on the 
functional overlap of the various programs with the functional 
capabilities of the processor.  So a dual-Xeon(HT) should, at best, 
perform like a three processor system.

> But you can also set the slots to 4 in the exec host definition. This is then
> the upper limit for all slots in total on a machine, regardless from which
> queues they are coming. If you do this, you can remove the load threshold on
> the queues.
I prefer to set the load limit scaling (in host config) so that a load of 
1.0 means 100% on all processors.  Then I set the load adjustment so that 
a new task adds 45%.  Of course, this only works correctly when all nodes 
have the same number of processors.  But if the load average is low, then 
another task can be started.

> Quoting "Marconnet, James E Mr /Computer Sciences Corporation"
> <james.marconnet at smdc.army.mil>:
>> We have some Penguin nodes and some RLX nodes. The Penguin nodes are
>> approximately 2GHz dual-processor, hyperthreaded. The RLX nodes are 800 MHz
>> single-processor.
>> We're using SGE6.0u1, setting slots to 4 for the Penguin nodes and to 1 for
>> the RLX nodes, since most jobs are CPU-bound.
>> Usually we end up with one job running on each RLX node and usually 4 on
>> each Penguin node. But sometimes when people submit jobs from different
>> ques, there are 2 or even 3 jobs running on a RLX node, and sometimes as
>> many as 8 jobs running on a Penguin node. All jobs try to get as much CPU
>> as
>> they can. And the Load Factors go way above 1.0. Nodes operating in alarm
>> state "a" are common.
>> As long as there is enough memory, disk space, network capacity, etc. etc.
>> does this oversubscribing actually hurt anything, such as overheating the
>> nodes, or causing file corruption? It seems to me that there is only 100%
>> of
>> each CPU available, no matter what you do. And that if one node can run X
>> jobs at a time, then each of them from the same manufacturer should be able
>> to do it just about as well as the others.
>> The question asked, I've seen some of our nodes that seem to be able to
>> take
>> oversubscribing, and others that seem to actually suddenly hang or to do
>> something else unexpected like give repeated bizarre Java errors when more
>> jobs are running at once there than we planned. It all depends.
>> I've tried setting the load limit to 0.75. But then the Penguin nodes
>> sometimes only get 3 jobs running at a time, seemingly wasting throughput.
>> And still sometimes they get 4-5 jobs.
>> Thanks!
>> Jim

               Anthony Ciani (aciani1 at uic.edu)
            Computational Condensed Matter Physics
    Department of Physics, University of Illinois, Chicago

To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net

More information about the gridengine-users mailing list