[GE users] [OT] How to know if it *really* uses InfiniBand ?
dev_hyd2001 at yahoo.com
Sun May 30 11:06:59 BST 2010
One way to check that out is doing lsof | grep infniband or lsof | grep verbs if you are using verbs for the Infiniband communication. This must be done on one of the nodes involved in your MPI job.
Another possible way is to look at the counters in the IBA switch logs, to see if they are getting incremented when a job is running.
Hope it helps!
--- On Fri, 5/21/10, igardais <igardais at yahoo.fr> wrote:
From: igardais <igardais at yahoo.fr>
Subject: [GE users] [OT] How to know if it *really* uses InfiniBand ?
To: users at gridengine.sunsource.net
Date: Friday, May 21, 2010, 3:56 PM
Our new toy is almost ready to use : we are using SGE over an ethernet connection and IB-RDMA for the MPI jobs.
We're using IntelMPI.
We set I_MPI_DEVICE=rdma and everything else to run over InfiniBand.
I_MPI_DEBUG=50 shows that the job selected "ofa-v2-ib0" for RDMA transfers, which is good.
But, when doing a 'netstat -tanpu', all the computing processes show they are connected using the ethernet IP address.
Is it OK or should I expect the processes to use the InfiniBand IP to confirm that they *really* uses InfiniBand for their chat ?
Excuse my newby questions but this look strange to me.
More information about the gridengine-users