How to Select a Version of MPI

There are two types of MPI, the message passing library, installed on the Discovery cluster.

  1. MPICH (MPICH2 and MPICH3) is used for message passing between compute nodes with an ethernet connection.
  2. MVAPICH (MVAPICH2) is used for message passing between compute nodes with an Infiniband connection.

Issue the command “features -a” on discovery to see which nodes have infiniband connection and to see the feature names ( ex. ib1, ib2) associated with them. If your MPI program has a lot of inter-node communication, it will run faster on the nodes with Infiniband connection.

The following module commands will list the different versions that are available:

  • module avail mpich – list the versions of MPICH using an ethernet connection
  • module avail mvapich – list the versions of MVAPICH using an InfiniBand connection

Select the version that you prefer and issue the module load modulefile command to load it.

  • For example, the command module load mpich3 will install the default version of MPICH3.

MPI code built for an ethernet connection will only use the 1 Gb network on the IB nodes instead of the faster Infiniband network device. MPI code built for the InfiniBand network connection will not work on the nodes with just the 1 Gb network connection.

See the Multi-Core Job Examples page for how to submit parallel jobs to the batch queue for information on how to specify the ethernet or InfiniBand connections.

How to Select a Compiler To Compile Your MPI Program

The name of the compiler used to build the MPI library is included in the name of the module.

  • For example mpich3/3.0.4-intel13.0 was built with the Intel v13.0 compilers.

Use the same compiler to compile your MPI program as was used to build the MPI library.

How to Compile and Link Your MPI Program

MPI has predefined scripts that can be used for compiling and linking programs. They automatically call the correct include file and libraries and use the same compiler that was used to build the MPI library.

They are mpicc for C programs, mpiCC for C++ programs, mpif77 for Fortran 77, and mpif90 for Fortran 90 programs.

  • Here is an example of how to compile a single C source code file using mpicc using level 3 optimization: mpicc -o mpi_example -O3 mpi_example.c

How to Run An MPI Program Interactively

When you are debugging your MPI code, it is easiest to run the program interactively. You can run your MPI jobs on the interactive nodes x01-x03. Use rsh to log in to any of these nodes. Since each of the three nodes has 16 processors, you can run on up to 48 processors.

You must specify a machine file that lists the nodes to run on. Create a file that contains one line for each processor that your MPI job will run on.

  • For example, if you log in to node x01 and want to run an 8-processor job using 4 processors on x01 and 4 processors on x02, here is what the machine file will look like:
    x01
    x01
    x01
    x01
    x02
    x02
    x02
    x02

The command to run an MPICH program interactively is mpirun. The format of the command for this example is:

 mpirun -np 8 -f machinefile mpi_example [input1 input2 ...]

See the mpirun man page for additional settings.

If you are running an MVAPICH program that uses the Infiniband, the command to run interactively is:

mpirun_rsh -rsh -np 8 -hostfile machinefile mpi_example [input1 input2 ...]