ARGO-NEW: The Message Passing Interface (MPI)

MPI (Message Passing Interface)* is a library specification the foundation of which is a group of functions that can be used, either in Fortran or C, to achieve parallelism. An MPI function permits one process to talk to another by the transmission of data (messages).

There is one implementation of MPI on the cluster:

• Vanilla MPI (version 2 as well as version 1) called MPICH2 and MPICH

MPICH version 2

Home location: /usr/common/mpich2-1.0.1

You will make two changes: one to the PATH variable; the other, the LD_LIBRARY_PATH. The former is mandatory. The latter is optional depending on the compiler you use.

If you login shell is C, then the changes are made to the .cshrc file. If you use bash, then the changes are to the .bash_profile file. To see which shell is your default, enter:

echo $SHELL

If the output is /bin/bash, you are using the bash shell; if /bin/csh, then the C shell.

For a bash shell user, append the following in .bash_profile:

export MPICH2_HOME=/usr/common/mpich2-1.0.1
export PATH=$MPICH2_HOME/bin:$PATH

export LD_LIBRARY_PATH=/usr/common/mpich2-1.0.1/lib:$LD_LIBRARY_PATH

For a C shell user, append the following in your .cshrc:

setenv MPICH2_HOME /usr/common/mpich2-1.0.1
setenv PATH /usr/common/mpich2-1.0.1/bin:$PATH

setenv LD_LIBRARY_PATH /usr/common/mpich2-1.0.1/lib:$LD_LIBRARY_PATH

Note: If you do not use the GNU compilers (instead, you use the supported Portland Group compilers), then the change to the LD_LIBRARY_PATH is unncessary because you will specify the appropriate library path with the -L option in the compile statement. See example below.

MPICH version 1

Home: /usr/common/pgi/linux86/6.0

Version 1 is included with the Portland Group compilers. Several enviromental variables must be defined or changed. The user must manually set them in the login file (.bash_profile for bash shell and .cshrc for C shell). To see which shell you use, enter the following command:

echo $SHELL

If the output is /bin/bash, you are using the bash shell; if /bin/csh, then the C shell.

If you are a bash shell user, append the following in your .bash_profile file:

export PGI=/usr/common/pgi
export PATH=$PATH:$PGI/linux86/6.0/bin

and then, in the same file, alter your LD_LIBRARY_PATH variable as follows:

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/common/pgi/linux86/6.0/lib:/usr/common/pgi/linux86/6.0/liblf

If you are a C shell user, append the following in your .cshrc file:

setenv PGI "/usr/common/pgi"
setenv PATH $PATH":$PGI/linux86/6.0/bin"

and then, in the same file, alter your LD_LIBRARY_PATH variable as follows:

setenv LD_LIBRARY_PATH $LD_LIBRARY_PATH":$PGI/linux86/6.0/lib:$PGI/linux86/6.0/liblf"

The GNU compilers as well as those from Portland Group are compatible with both versions of MPI. For more information about compilers, see the Available Software page.

Below are two sample scripts (both named my_script): one for bash shell users; the other, C shell users:

#! /bin/bash
export MPICH2_HOME=/usr/common/mpich2-1.0.1
export PATH=$MPICH2_HOME/bin:$PATH
mpiexec -n 3 $MPICH2_HOME/examples/cpi

#! /bin/csh
setenv MPICH2_HOME /usr/common/mpich2-1.0.1
setenv PATH=($MPICH2_HOME/bin $path)
mpiexec -n 3 $MPICH2_HOME/examples/cpi


To execute the appropriate example, enter:

qsub my_script

The expected output is something like

Process 0 of 3 is on argo
Process 1 of 3 is on argo
Process 2 of 3 is on argo
pi is approximately 3.1415926544231323, Error is 0.0000000008333392 wall clock time = 0.007743

If you have been using MPICH version 1, then the mpirun command is how you started your MPI program. While MPICH2 provides an mpirun command for backward compatability, mpiexec is the recommended execution command.

Compiling and Linking the Parallel Hello World Program using MPICH2

The following example illustrates compiling using the Portland Group compilers:

pgcc -o mpihello hello1.c -D REENTRANT -L/usr/common/mpich2-1.0.1/lib -I/usr/common/mpich2-1.0.1/include/ -lmpich -Wl -t

The -L/usr/common/mpich2-1.0.1/lib is required when using Portland Group compilers; without it, the compiler will use MPICH version 1 libraries and not version 2.
The -I/usr/common/mpich2-1.0.1/include/ will specify the location of the header files to the Portland Group compilers. The optional -Wl -t will display the libraries files used (warning: the list is long):

/usr/bin/ld: mode elf_i386
/usr/lib/crt1.o
/usr/lib/crti.o
/usr/lib/gcc-lib/i386-redhat-linux/2.96/crtbegin.o
/tmp/pgccbaaaanksab.o
(/usr/common/mpich2-1.0.1/lib/libmpich.a)comm_rank.o
(/usr/common/mpich2-1.0.1/lib/libmpich.a)comm_size.o
(/usr/common/mpich2-1.0.1/lib/libmpich.a)commutil.o
(/usr/common/mpich2-1.0.1/lib/libmpich.a)errutil.o
(/usr/common/mpich2-1.0.1/lib/libmpich.a)grouputil.o
(/usr/common/mpich2-1.0.1/lib/libmpich.a)init.o
(/usr/common/mpich2-1.0.1/lib/libmpich.a)initthread.o
...

The following scripts are another way to compile and link your program:

Script	Language
mpicc	C
mpicxx	C++
mpif77	Fortran 77
mpi90	Fortran 90

Each script invokes the appropriate GNU compiler..

Compiling and Linking the Parallel Hello World Program using MPICH

pgcc -o mpihello hello1.c -D REENTRANT -lmpich

Link flags

Vanilla MPI:

For C:

-lmpich

For Fortran:

-lfmpich -lmpich

Use the qsub command to submit an MPI job to torque.

Vanilla MPI requires a machine file which will contain the names of the nodes to be used to run a job. There is no required name for the machine file though it is recommended that a useful name be selected. It is also recommended that file be placed in a home directory though it is not required. The format of the machine file is one node per line with the names left-justified. For example:

• argo4-4
• argo4-3
• argo4-2
• argo4-1

the above machine file is called node_file and is located in the home directory. Obviously, your node file may have more or less nodes than is shown in the example.

There are two scripts that are used in the running of a example vanilla MPI job. Here is the key information:

• the name of the example user program is hello,
• the first script is called x1 (obviously, that name may be changed),
• the second script is called x.

Script #1 (x1):

#!/bin/csh
#Name of script: x1
set nodes = `perl -e 'while (<STDIN>){chop;$a.="$_+"}chop($a);print $a;' <$HOME/node_file`
qsub -l nodes=$nodes ./x

Script #2 (x):

#! /bin/csh
#Name of script: x
setenv PGI /usr/common/pgi
set path=($PGI/linux86/5.2/bin $path)
mpirun -np 4 hello

To excute the MPI program hello, enter the following:

./x1

The x1 script

• reads the user machine file,
• formats the nodes for PBS, and
• invokes the x script.

The x script

• establishes the vanilla MPI environment, and
• runs the user program hello.

Warnings and gotchas:

• Notice the correspondence between the -np 4 operand to mpirun and the number of nodes in the user node file.
• Do not use the mpirun -nolocal option.
• If your MPI program is called something other than hello, change the name of the executable program on the mpirun line in the x script.
• If you fail to use the x1 script and invoke the x script directly or if you skip both scripts and invoke mpirun, then the PBS accounting records will not accurately reflect the use the multiple nodes. Those accounting records are used to create summary statistics which are reviewed to determine if additional compute nodes are needed. If the system appears under-utilized, then there is no justification to purchase more nodes.

Examples

General MPICH2 informarion can be found at:

http://www-unix.mcs.anl.gov/mpi/mpich2/

http://www-unix.mcs.anl.gov/mpi/mpich2/developer.htm

Message Passing Interface Forum:

http://www.mpi-forum.org/

MPI-2: Extensions to the Message-Passing Interface:

http://www.mpi-forum.org/docs/mpi-20-html/mpi2-report.html