ARGO: More on Running Parallel MPI Jobs

January 22, 2003

There are more examples of MPI programs in Argo Beowulf Cluster: MPI Commands and Examples.

Executables issuing ScaMPI calls CANNOT be started directly from a shell prompt. Instead, ScaMPI programs can be started:

• from a shell prompt using the MPI monitor program mpimon,
• by using the wrapper script for mpimon called mpirun, or
• from the Scali Desktop GUI.

It is absolutely necessary to run an ScaMPI program on a homogenous filesystem, i.e. on a filesystem providing the same path and program name on all nodes. There are five filesystems that are all accessible to all nodes: /home, /scratch, and the three parallel filesystems (pvfs-scratch1, pvfs-scratch2, and pvfs-scratch3).

The Examples below run the hello.c program we compiled and linked in the previous section. Examples 2 and 4 are run in the PBS batch system; they are the preferred way to run programs on argo.

There are more examples of MPI programs in Argo Beowulf Cluster: MPI Commands and Examples.

• The control and startup of an ScaMPI application is monitored by mpimon
  
  
• By default, this skips PBS*, though the second example belows show how to run mpimon in batch. Example 2 is the preferred way to use mpimon.
  
  
• The basic syntax is:

  mpimon [mpimon-options] userprogramname program-options \
  -- hostname [count] [[hostname][<count>]...]    

  In the above, the variables and options contained in square brackets ([]) are optional. The backslash (\) at the end of the first line allows you to continue the command on a second line.
  The "--" is required.
  
  
• Example 1: scasub mpimon hello -- argo4-4 2 argo4-3 2
  
  
  • The hostname/count combination
    • You may specify the number of processes (the count value) that will run on a particular host.
    • Thus the example: mpimon hello -- argo4-4 2 argo4-3 2
      instructs argo to run two of the hello processes on the compute cluster node named argo4-4 and run two of the processes on the node named argo4-3.
      
      
  • Each process is executed on a different virtual processor on the nodes.
    
    
  • To run in the batch system, PBS (all jobs on argo must be run via batch), preface the mpimon command with scasub.

   

• Example 2: scasub mpimon hello -- argo4-4 1 argo4-3 1 argo4-2 1 argo4-1 1
  • Each of the four processes is executed on a different nodes - one process on argo4-4, one process on argo4-3, one process on argo4-2, and one process on argo4-1.

For additional information, type mpimon -help | more or click here.

• Mpirun is a wrapper script for mpimon, giving MPICH style startup for ScaMPI applications.
  
  
• Instead of the mpimon syntax which uses the hostname/count combination, mpirun uses only the total number of processes.
  
  
• Does a call to scaconftool (part of Scali) to get a list of operational nodes and evenly divides the processes among those nodes.
  
  
• By default, this skips PBS*, though the second example belows show how to run mpirun in batch. Example 4 is the preferred way to use mpirun.
  
  
• The basic syntax is:

  mpirun mpirun-options mpimon-options userprogramname program-options

• Example 3: scasub mpirun -np 4 hello
  

  The np -4 means that the hello program has a total of four processes (also called instances). However, those four instances will run on a single node (not the best way to do it because each of the nodes has only one processor. As a result, each instance is competing against the others for the attention of the processor).
  

• Example 4: scasub mpirun -nodes argo4-4,argo4-3,argo4-2,argo4-1 mpirun -np 4 hello
  
  Run each of the four instances of the hello program on the specified nodes: one instance on argo4-4, one on argo4-3, one on argo4-2, and one on argo4-1.

The ScaMPI users guide is located in the argo directory /opt/scali/doc/ScaMPI. It's also available online line at Scali. (Or go to the Scali home page, http://www.scali.com, point to the Download menu at the top of the page, and select Documentation -> Manuals.)

There are ten possible permutations for program execution. Of the ten, two are for serial program execution; eight, for parallel execution.

The following diagram outlines all ten program execution methods; descriptions of the methods follow the diagram.

  Master node - argo                                MPI System - Master/Compute nodes (D)
+--------------------------+                       +------------------------------------+
|                          |                       |                                    |
|                          |                       |                                    |
|                          |                       |                                    |
|                          |                       |                                    |
|                          |                       | +---------------------------+      |
|                          |                       | |                           |      |
|                          |                       | |                           |      |
|          +---------------------------------------|-+------> +-----------+      |      |
|          |               |   PBS Batch System (C)|     +--> | mpimon (E)| <-------+   |
|  +-------------------+   |      +------------+   |     |    +-----------+ <-+  |  |   |
|  | Scali desktop (A) |   |      |            |   |     |                    |  |  |   |
|  +-------------------+ ------>  |  batch (C) |---------+                    |  |  |   |
|                          |      |  (scasub)  |   |     |                    |  |  |   |
|  +-------------------+ ------>  |            |   |     |          +---->----+  |  |   |
|  | Command line  (B) |   |      +------------+   |     |          |            |  |   |
|  +-------------------+   |    (preferred route)  |     |    +-----------+      |  |   |
|          |               |                       |     +--> | mpirun (F)| <----+  |   |
|          +-----------------------------------+--- --------> +-----------+         |   |
|                          |                   |   |                                |   |
|                          |                   +---|------------>-------------------+   |
|                          |                       |                                    |
|                          |                       |                                    |
+--------------------------+                       +------------------------------------+
           

• A - Scali Desktop
• B - Linux command line
• C - The Batch System
• D - The MPI System
• E - mpimon
• F - mpirun

The ten are listed below of which the first two are for serial jobs:

Preferred: A --> C The Scali desktop --> PBS Batch system

Serial job submission. From the Scali desktop (accessible from the scadesktop command), a job is submitted to the batch system via the scasub command.

B --> D Linux command line --> MPI system

Same as the above except the job submission is initiated by entering the scasub command at the Linux prompt.

The remaining eight are for parallel jobs; please chose the routes below that include C , using the scasub command to submit the MPI job though the batch system:

A --> E

From the Scali desktop, a job is submitted to the MPI system via the mpimon command.

B --> E

Same as the previous one except the mpimon command is entered at the Linux prompt.

A --> F --> E

From the Scali desktop, a job is submitted to the MPI system via the mpirun command. The mpirun command converts the format of the submission from mpirun syntax to mpimon syntax and forwards the reformatted job to mpimon.

B --> F --> E

Same as the previous one except the mpirun command is entered at the Linux prompt.

A --> C --> E

From the Scali desktop, a job is submitted to the MPI system through the batch system. The mpimon command is encased in the batch scasub command.

B --> C --> E

Same as the previous one except the scasub command is entered at the Linux prompt.

A --> C --> F --> E

From the Scali desktop, a job is submitted to the MPI system through the batch system. The mpirun command is encased in the batch scasub command. The mpirun command converts the format of the submission from mpirun syntax to mpimon syntax and forwards the reformatted job to mpimon.

B --> C --> F --> E

Same as the previous one except the scasub command is entered at the Linux prompt.