[GE users] Gridengine and Abaqus HP-MPI parallel computing integration

Jeroen M. Kleijer jeroen.m.kleijer at philips.com
Thu Jun 22 14:30:34 BST 2006

    [ The following text is in the "UTF-8" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some characters may be displayed incorrectly. ]


It's been over a year since I've looked into it.
I had it working but since most of our jobs didn't benefit from adding 
cpus we left it at that.
There are a few pointers:
- submission goes through the abaqus_v6.env file but I think you already 
figured that one out yourself
  here's a snippet from the one we use:

        mp_mode = MPI
        mp_file_system = (DETECT,DETECT)
        mp_file_system = (SHARED,LOCAL)
        mp_mpi_implementation = MPICH
        mp_mpirun_path = 
        ##mp_mpirun_options = " -ssi rpi usysv -ger -O "
        ##mp_mpirun_options = " -ssi rpi usysv -ger -O "
        ##mp_rsh_command = 'rsh -n -l %U %H "export 
        mp_rsh_command = 'rsh -n -l %U %H %C'
        run_mode = BATCH
        from driverQueues import *
        class SGE_Queue(LSFQueue):
            def __repr__(self):
                return 'Submits to SGE %s queue '
            def submit(self, options, env):
                job = options['job']
                verbose = options.get('verbose', 0)
                queue = self.name
                cpus = options.get('cpus','1')
                ##qsub = '/home/sge/bin/lx24-amd64/qsub -pe mpich %s -V -v 
LAMHOME -v PATH -l arch=lx24-amd64 -cwd -q %s -N %s -o %s.log -b y 
"~/.caddata/abaqus/abaqus python %s.com"' % \
                qsub = '/home/sge/bin/lx24-amd64/qsub -V -v LAMHOME -v 
PATH -l arch=lx24-amd64 -cwd -q %s -N %s -o %s.log -b y  
"~/.caddata/abaqus/abaqus python %s.com"' % \
                   (queue, job, job, job)
        ##               (cpus, queue, job, job, job)
                return self.spawn(qsub, env, verbose)
        ##queues['batch.q']   = SGE_Queue(name='batch.q')
        queues['default']   = SGE_Queue(name='batch.q')
        queues['immediate.q']  = SGE_Queue(name='immediate.q')
        queues['hold.q']  = HoldQueue()
- as you can see we switched back to single cpu  because of several issues 
we ran in to
- when you submit the job (an .inp file) abaqus check whether there's a 
file called abaqus_v6.env (in either your job or your homedirectory) and 
checks whether you've got a variable called mp_host_list with the exact 
same number of cpus you requested. (so if you've requested 3 cpus, it 
should hold something like [['machine1',1],['machine3',2]] Whether these 
machines actually exist doesn't matter, the line should exist!
- the problem you then get is getting the machines file, created by SGE to 
be used by Abaqus which is a different matter.
- also bear in mind that the syntax for the machines file differs per 
implementation.... mpich wants it another way than HP-MPI 

I hope this gives you some idea on where to look but it's been a while 
since I looked into it because of the major headache it can turn into.

Met vriendelijke groeten / Kind regards

Jeroen Kleijer
Unix Systeembeheer
Philips Applied Technologies

"Gautier, Valentin" <Valentin.Gautier at eads.net> 
2006-06-22 03:01 PM
Please respond to
users at gridengine.sunsource.net

<users at gridengine.sunsource.net>

[GE users] Gridengine and Abaqus HP-MPI parallel computing integration

Hello everybody,
We?ve got a 6 nodes bi-opteron linux cluster used for abaqus computing, 
I?ve got  no problem to get abaqus work with  SGE  in threads 
multi-processing mode, but I have no idea of how to make it work in mpi 
mode with hp-mpi ( delivered with abaqus 6.6-1).
Does anyone have some sort of solution to configure the Gridengine mpi 
queue ?
Is there any howto for that ?
Thanks a lot.

More information about the gridengine-users mailing list