[GE users] [OT] How to know if it *really* uses InfiniBand ?

icaci hristo at mc.phys.uni-sofia.bg
Mon May 31 10:55:57 BST 2010


Hi,

MPI jobs first need to communicate various InfiniBand properties (like port GUIDs) before they could talk to each other. Also I/O redirection and other service information runs on top of TCP/IP.

Hope that helps,

Hristo

On 21.05.2010, at 16:56, igardais wrote:

Hi list,

Our new toy is almost ready to use : we are using SGE over an ethernet connection and IB-RDMA for the MPI jobs.
We're using IntelMPI.
We set I_MPI_DEVICE=rdma and everything else to run over InfiniBand.
I_MPI_DEBUG=50 shows that the job selected "ofa-v2-ib0" for RDMA transfers, which is good.

But, when doing a 'netstat -tanpu', all the computing processes show they are connected using the ethernet IP address.

Is it OK or should I expect the processes to use the InfiniBand IP to confirm that they *really* uses InfiniBand for their chat ?

Excuse my newby questions but this look strange to me.
Thanks,
Ionel



--
Dr Hristo Iliev
Monte Carlo research group
Faculty of Physics, University of Sofia
5 James Bourchier blvd, 1164 Sofia, Bulgaria
http://cluster.phys.uni-sofia.bg/hristo/




More information about the gridengine-users mailing list