ARGO: General Information

Jargon alert: There are a lot of terms in the argo pages that might be new to you; most are defined in the argo glossary. Links followed with an asterisk such as"the MPI* library" are to the term's definition in the glossary.

Most of what is presented here is not needed for you to use the cluster. Some folks are interested in the system details and that is why it is made available. But, for most users, it is unnecessary and may be skipped.

• Overview-Beowulf clustering
• The ACCC Beowulf
• Master Node
• Compute Nodes
• Home Server
• Scratch Server
• Network

A cluster is a group of PCs which are interconnected by one or more networks and are used for serial or parallel program execution. Clustering has been around for quite some time. In 1994 Donald Becker, a NASA researcher, invented a way to connect a group of inexpensive off-the-shelf PCs with special software to create a single system that could be scaled up to deliver supercomputer performance. The name assigned to that cluster: Beowulf. Now, the term Beowulf cluster generally refers to any cluster that loosely follows the Becker model.

Parallel programming on a Beowulf-type cluster is accomplished by dividing a computation into parts and making use of multiple processes. Sometimes a single processor can be used for all the processes. Most complex problems, however, involve processes executing on separate processors. Processes coordinate their activities by explicitly sending and receiving messages. The most commonly used method of programming distributed-memory MIMD* systems is message passing. The MPI* library specification is the method for message passing.

The UIC ACCC Beowulf cluster is called Argo; Internet domain name: argo.cc.uic.edu.

Cluster Overview

• Purpose
  • Make a group of computers perform as a single machine.
  • Clustering has been going on for quite a while. In 1994, Donald Becker, a NASA researcher invented a way to connect a group of inexpensive off-the-shelf PCs with special software to create a single system that could be scaled up to deliver supercomputer performance. The name he chose for this cluster was Beowulf.
  • Beowulf clusters can have hundreds or thousands of computers. Argo has 57
  
  
• Advantages
  • High performance computing at a fraction of the cost
  • Good scalability (add, subtract, modify nodes)
  • Have a heterogeneous environment.


• Parallel Programming -- Parallel computation on a Beowulf-type cluster is accomplished by dividing a computation into parts and making use of multiple processes.
  • Sometimes a single processor can be used for all the processes.
  • Most complex problems involve processes executing on separate processors.


• Processes coordinate their activities by sending an receiving messages (called message passing). MPI (Message Passing Interface) is a library specification for message passing.

Hardware

• 57 PCs
• A private, internal network shared only among the argo machines

Details

• Composition: one master machines, fifty-six compute machines, and two file servers, one providing home space; the other, scratch space. It is customary to refer to each compute machine as a node, so argo has one master node (also called the front-end or the head node) and 56 compute nodes. In some of the computing literature, compute nodes are sometimes referred to as slave nodes. At UIC, they are called compute nodes.
  
  
• Name of cluster: argo.
  
  
• The argo master is your doorway to the cluster:
  • clients log in there (it's the only way they can get into argo),
  • create/edit/compile programs there, and
  • submit them for execution.


• The 56 argo compute nodes:
  • Are heterogenous. Some of the machines have AMD processors; others, Intel processors
  • Have one or two internal disks. Compute node boot from an internal disk, and
  • Clients do not log into them.
  
  
• There are fourteen groupings, called zones, of compute nodes. Each of the first four zones corresponds to one horizontal SCI ring and has a numerical designation (zone 4, zone 3, zone 2, zone 1). Within each zone, there are four compute nodes with each having a numerical designation (4, 3, 2, 1). The hostname of a compute node includes the zone and node number. Syntax: argoZONE-NODE. Example: argo3-1 is zone three, node one. Fifty-six compute nodes = fourteen zones multiplied by four computers (nodes) per zone:
  
  

  Zones	Compute Nodes
  zone14	argo14-4	argo14-3	argo14-2	argo14-1
  zone13	argo13-4	argo13-3	argo13-2	argo13-1
  zone12	argo12-4	argo12-3	argo12-2	argo12-1
  zone11	argo11-4	argo11-3	argo11-2	argo11-1
  zone10	argo10-4	argo10-3	argo10-2	argo10-1
  zone9	argo9-4	argo9-3	argo9-2	argo9-1
  zone8	argo8-4	argo8-3	argo8-2	argo8-1
  zone7	argo7-4	argo7-3	argo7-2	argo7-1
  zone6	argo6-4	argo6-3	argo6-2	argo6-1
  zone5	argo5-4	argo5-3	argo5-2	argo5-1
  zone 4	argo4-4	argo4-3	argo4-2	argo4-1
  zone 3	argo3-4	argo3-3	argo3-2	argo3-1
  zone 2	argo2-4	argo2-3	argo2-2	argo2-1
  zone 1	argo1-4	argo1-3	argo1-2	argo1-1



• The argo file server: User home space (directories) on the master node are physically located on a separate machine whose sole purpose is to act as a file server. The file server is fiber-attached to the master node and NFS-mounted* to all compute nodes. That means that a file created in your home space is available to all compute nodes. When you login to argo, you are placed in your home directory. If you jump to another location, you can always get home by executing the cd command:
  
  
  cd
  
  
• The argo scratch server: User scratch space on the master node is, as is the case for home directories, located on a separate machine. The partition is NFS-mounted to the master and to all compute nodes. Each user has his own directory in the scratch area. Name of your space: /scratch/netid. For example, user jsmith can jump to his scratch directory by doing the following:
  
  
  cd /scratch/jsmith
  
  
• There are two types of filesystems on argo (most users don't need to know this nor do they care):
  • local partitions with a Linux filesystem
  • remote partitions -- ones on other machines -- available as an Network FileSystem

  A ext3 filesystem* is one constructed on disk that is physically attached to the machine. Some portion of or all of the local disk is used to contruct a partition which becomes the basis for the filesystem.

  An NFS filesystem* is local to one machine, but made to appear local to other machines by accessing it over a network.

Tigger vs. argo

Though tigger and argo are MIMD* systems, tigger is a shared memory multiprocessor systems whereas argo is a distributed memory multicomputer.

Tigger

Tigger is a shared memory multiprocessor system (a conventional computer) with multiple processors of the same type (symmetric) connected to multiple memory modules. Like your PC or laptop - just bigger. Access to physical memory and processors is shared among processes. Syncronization among processes is accomplished by reading from and writing to shared memory which is controlled by a locking mechanism (spinlocks and semaphores). The path that connects processors to physical memory is internal and transparent to users. The data along with the executable code is contained in memory which is shared among the processes. By using threads, portions of a single program may run on different processors.

Argo

A distributed memory multicomputer system is a group of complete computers that are connected to one another. Each of the constituent machines is a self contained unit (has its own processors and memory) capable of acting independently of the other machines in the group. The most common method of programming distributed-memory systems is message passing which permits processes to share data and coordinate their activities.

Argo has 56 compute nodes and apprximately 160GB of memory.

      Computer        Computer        Computer        Computer         Computer
      (Argo1-1)       (Argo1-2)       (Argo1-3)       (Argo1-4)  ...  (Argo4-4)
   +------------+  +------------+  +------------+  +------------+   +------------+
   | +--------+ |  | +--------+ |  | +--------+ |  | +--------+ |   | +--------+ |
   | |Processor |  | |Processor |  | |Processor |  | |Processor |   | |Processor |
   | +--------+ |  | +--------+ |  | +--------+ |  | +--------+ |   | +--------+ |
   |    |       |  |    |       |  |    |       |  |    |       |   |    |       |
   | +--------+ |  | +--------+ |  | +--------+ |  | +--------+ |   | +--------+ |
   | |Local   | |  | |Local   | |  | |Local   | |  | |Local   | |   | |Local   | |
   | |memory  | |  | |memory  | |  | |memory  | |  | |memory  | |   | |memory  | |
   | +--------+ |  | +--------+ |  | +--------+ |  | +--------+ |   | +--------+ |
   +------------+  +------------+  +------------+  +------------+   +------------+
        |                |              |               |                | 
        |                |              |               |                |

   +-----------------------------------------------------------------------------+
   | Private ethernet network                                                    |
   +-----------------------------------------------------------------------------+

        |                |              |               |                |
        |                |              |               |                |
     (Argo5-1)       (Argo5-2)       (Argo5-3)       (Argo5-4)  ...  (Argo8-4)
   +------------+  +------------+  +------------+  +------------+   +------------+
   | +--------+ |  | +--------+ |  | +--------+ |  | +--------+ |   | +--------+ |
   | |Processor |  | |Processor |  | |Processor |  | |Processor |   | |Processor |
   | +--------+ |  | +--------+ |  | +--------+ |  | +--------+ |   | +--------+ |
   |    |       |  |    |       |  |    |       |  |    |       |   |    |       |
   | +--------+ |  | +--------+ |  | +--------+ |  | +--------+ |   | +--------+ |
   | |Local   | |  | |Local   | |  | |Local   | |  | |Local   | |   | |Local   | |
   | |memory  | |  | |memory  | |  | |memory  | |  | |memory  | |   | |memory  | |
   | +--------+ |  | +--------+ |  | +--------+ |  | +--------+ |   | +--------+ |
   +------------+  +------------+  +------------+  +------------+   +------------+

Overview

Clients log into the master (also called the front-end), create/edit/compile programs, and submit them for execution on one or more of the compute nodes. Clients do not run their programs on the master. Repeat: user programs MUST NOT be run on the master . Instead, they are submitted to the compute nodes for execution.

Hardware

• 2U rack mounted system case (The depth of the system case is measured in U units. 1U = 1.75 inches.)
• Dual AMD Opteron processors
• 4GB of memory
• Dual NIC* (Network Interface Card, Intel Gigabit: eth0/eth1) with jumbo frames on the private network.
  • eth0: 172.16.0.35
  • eth1: 128.248.121.64
• one internal 72GB SCSI disks (sda)

Operating System

RedHat Linux 4.2 (Kernel level: 2.6.9-89.0.16.ELsmp)

Master Node Filesystems

There are two types of filesystems on argo:

• local partitions with a Linux ext3 filesystem
• remote partitions -- ones on other machines -- available as an Network FileSystem.

Filesystems on the master node	Size	Available to compute nodes?	Access Method	Explanation
boot	120MB	No	local	local partition - assessible only to master
home/homesXX	1.2TB	Yes	NFS	home area on argo's home server. Click here for more information
opt	10GB	No	local	local partition - assessible only to master
root	2GB	No	local	local partition - assessible only to master
scratch	1.2TB	Yes	NFS	scratch area on argo's scratch server
tmp	10GB	No	local	temporary storage - assessible only to master
usr	10GB	No	local	local partition - assessible only to master
usr/common	20GB	Yes	local	Common area for software
var	10GB	No	local	local partition - assessible only to master

Overview

The compute nodes have ONE AND ONLY ONE purpose: run user programs. Clients do not have login access to the compute nodes. From the master node, clients submit programs to torque which executes them on one or more compute nodes. While clients are prohibited from logging into compute nodes, they may issue remote shell commands from the master to retrieve information. For example:

rsh argo1-1 uptime
    12:23:54 up 83 days, 1:03, 0 users, load average: 0.00, 0.00, 0.00

Hardware

56 Nodes	Size	#/Type of processor	Memory	NIC	Disk (GB)
16: argo4-4 - argo1-1	1U	Dual AMD Opterons	4GB	Dual-port GigE	One 72 SCSI
32: argo12-4 - argo5-1	1U	Two 3.0GHz Xeon	2.0GB	Dual-port GigE	One 36 SCSI
8: argo14-4 - argo13-1	1U	Two Dual-Core AMD Opterons	4.0GB	Dual-port GigE	Single SCSI disk

Operating System

RedHat Linux 4.2 (Kernel level: 2.6.9-22.ELsmp)

Overview

User home directories are physically located on a separate machine whose sole purpose is to act as a file server. There are four disk partitions of 1.2GB each on the server. Total available space: approximately 4.8TB. The partitions are fiber-attached to the master and NFS-mounted to all compute nodes:

Partition	Mount point on the argo file server	Filesystem - master and compute
/dev/sdb1	/mnt/array_a	/home/homes50
/dev/sdc1	/mnt/array_b	/home/homes51
/dev/sdd1	/mnt/array_c	/home/homes52
/dev/sde1	/mnt/array_d	/home/homes53

When your account was created, the system created a directory (your home space) in ONE of the four home filesystems. To see which one, type:

echo $HOME

$ whoami
jsmith
$ echo $HOME
/home/homes50/jsmith

When your account was created, the system also made a directory for you in the scratch filesystem. The location is:

/scratch/netid

/scratch/jsmith

Overview

• To share small or large amounts of data,
• To synchronize operations.

Ethernet

Each node, master and compute, is connected to an ethernet network and the NIC within a compute node is gigabit (with jumbo frames). The ethernet NIC may be used for interprocess communication using vanilla MPI as well as for out-of-band-management (basically anything - NIS, NFS, etc - except interprocess communication):

+-----------+           Master            Switch          Switch           Switch           Switch
| Outside   |        +----------+      +---------+      +---------+      +---------+      +---------+
|  World    |  <--> eth1      eth0 <-->| | | | | | <--> | | | | | | <--> | | | | | | <--> | | | | | |         
+-----------+        +----------+      +---------+      +---------+      +---------+      +---------+
                                            |                |                |                |  
                                            |                |                |                |
                                            V                V                V                V
                                       +---------+      +---------+      +---------+      +---------+ 
                                       | argo4-4 |      | argo8-4 |      | argo12-4|      | argo14-4|
                                       +---------+      +---------+      +---------+      +---------+ 
                                       | argo4-3 |      | argo8-3 |      | argo12-3|      | argo14-3|
                                       +---------+      +---------+      +---------+      +---------+
                                       |   ...   |      |   ...   |      |   ...   |      |   ...   |
                                       +---------+      +---------+      +---------+      +---------+
                                       | argo1-1 |      | argo5-1 |      | argo9-1 |      | argo14-1|    
                                       +---------+      +---------+      +---------+      +---------+

Master:

128.248.121.64 (eth1)

Master:

172.16.0.35 (eth0)

Compute:

172.16.zone.node (eth0)