[GE users] Default CPU core binding
reuti at staff.uni-marburg.de
Tue Oct 12 17:15:14 BST 2010
Am 04.10.2010 um 13:53 schrieb jewellc:
>>> Is it possible to set the new core binding feature to automagically use CPU affinity to achieve the same effect? Can I set up my Grid Engine installation to automatically assign a process to only the cores that have been requested in the qsub (or qlogin) command?
>> this is the reason, why I prefer using the non-threaded versions of ATLAS or ACML. If they are not prepared to do it, you don't have to set it up in the queueing system (unless this is exactly the parallelization you want to use for your application of course).
> I'm trying to make my system easy for inexperienced users by providing the parallel BLAS libraries. Users simply choose how many CPUs they want to use, and the system allocates their job to an SMP parallel environment.
>> I don't know what the "sge_cpuset" you used did in particular, but you can check here what the new feature supplies:
> the sge_cpuset suite essentially consists of job prolog, starter, and epilog scripts. The prolog sets up a "cpuset" on the machine, which essentially attaches the sge_shepherd and all sub-processes to a fixed set of CPUs. Thus, even if the user's job spawns multiple threads, they are contained inside the cpuset and do not interfere with other jobs running on the same host. I think this is a really useful feature, since it essentially provides a 'virtual' environment without the startup and runtime overheads of a full-blown virtual machine. I would encourage the SGE Developers to perhaps explore this avenue a little further....
> As a philosophical point, modern HPC seems to require a mix of multithreading and message passing, so addressing the issue of how to prevent jobs interfering with each other on SMP (or NUMA) machines seems really important. I wonder whether I could ask other readers of this list to share their experiences of controlling other multithreaded applications within the multi-core scheduled environment?
I posted it just on the Open MPI list, and I think this also concerns your question:
With the default binding_instance set to "set" (the default) the shepherd should bind the processes to cores already. With other types of binding_instance these selected cores must be forward to the application via an environment variable or in the hostfile.
As this is only a hint to SGE and not a hard request, the user must plan a little bit the allocation beforehand. Especially if you oversubscribe a machine it won't work. When I look at /proc/*/status it's mentioned there as it happened. And it's also noted in "config" file of each job's .../active_jobs/... file. E.g. a top shows:
9926 ms04 39 19 3756 292 228 R 25 0.0 0:19.31 ever
9927 ms04 39 19 3756 292 228 R 25 0.0 0:19.31 ever
9925 ms04 39 19 3756 288 228 R 25 0.0 0:19.30 ever
9928 ms04 39 19 3756 292 228 R 25 0.0 0:19.30 ever
for 4 forks of an endless loop in one and the same jobscript when submitted with `qsub -binding linear:1 demo.sh`. Well, the funny thing is that with this kernel version I still get a load of 4, despite the fact that all 4 forks are bound to one core. Should it really be four?
> Dr Chris Jewell
> Department of Statistics
> University of Warwick
> CV4 7AL
> Tel: +44 (0)24 7615 0778
> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].
To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].
More information about the gridengine-users