Argo-new Beowulf Cluster: General Information

	ACADEMIC COMPUTING and COMMUNICATIONS CENTER

High Performance Computing: The ACCC Cluster Argo-new
Contents	Overview	Getting Started	Available Software
Running Jobs	MPI	Glossary	More Info

ARGO-NEW: General Information

Jargon alert: There are a lot of terms in the argo-new pages that might be new to you; most are defined in the argo-new glossary. Links followed with an asterisk such as"the MPI* library" are to the term's definition in the glossary.

Overview-Beowulf clustering

A cluster is a group of PCs which are interconnected by one or more networks and are used for serial or parallel program execution. Clustering has been around for quite some time. In 1994 Donald Becker, a NASA researcher, invented a way to connect a group of inexpensive off-the-shelf PCs with special software to create a single system that could be scaled up to deliver supercomputer performance. The name assigned to that cluster: Beowulf. Now, the term Beowulf cluster generally refers to any cluster that loosely follows the Becker model.

Parallel programming on a Beowulf-type cluster is accomplished by dividing a computation into parts and making use of multiple processes. Sometimes a single processor can be used for all the processes. Most complex problems, however, involve processes executing on separate processors. Processes coordinate their activities by explicitly sending and receiving messages. The most commonly used method of programming distributed-memory MIMD* systems is message passing. The MPI* library specification is the method for message passing.

The UIC ACCC Beowulf cluster is called Argo-new; Internet domain name: argo-new.cc.uic.edu.

Cluster Overview

Purpose
- Make a group of computers perform as a single machine.
- Clustering has been going on for quite a while. In 1994, Donald Becker, a NASA researcher invented a way to connect a group of inexpensive off-the-shelf PCs with special software to create a single system that could be scaled up to deliver supercomputer performance. The name he chose for this cluster was Beowulf.
- Beowulf clusters can have hundreds or thousands of computers.
Advantages
- High performance computing at a fraction of the cost
- Good scalability (add, subtract, modify nodes)
- Have a heterogeneous environment.
Parallel Programming -- Parallel computation on a Beowulf-type cluster is accomplished by dividing a computation into parts and making use of multiple processes.
- Sometimes a single processor can be used for all the processes.
- Most complex problems involve processes executing on separate processors. In our case, it will be separate virtual processors. Each node has one physical processor but will have multiple virtual processors.
Processes coordinate their activities by sending an receiving messages (called message passing). MPI (Message Passing Interface) is a library specification for message passing.

The ACCC Beowulf

Hardware

68 PCs
One internal network (NICs*, cards, hubs, switches)
Console concentrator
One console/keyboard/mouse

Details

Composition: two master machines (a primary and a backup), sixty-four compute machines, and two file servers, one serving home directories; the other, scratch. It is customary to refer to each compute machine as a node, so argo-new has one master node (also called the front-end or the head node) and 64 compute nodes. In some literature, compute nodes are sometimes referred to as slave nodes. Herein, they are called compute nodes.
Name of cluster: argo-new.
The argo-new master node is the entry point to the cluster. Clients log into the master, create/edit/compile programs there, and submit them for execution on the compute nodes. Clients do not log into compute nodes.
The 64 argo-new compute nodes:
- Are heterogenous -- several different types of machines.
- Have internal disks -- each compute node has an internal disk and boots from it instead of from a server over a network.

There are sixteen groupings, called zones, of compute nodes in the cluster. Each of the first four zones corresponds to one horizontal SCI ring and each has a numerical designation (zone 4, zone 3, zone 2, zone 1). Within each zone, there are four compute nodes with each having a numerical designation (4, 3, 2, 1). The hostname of each compute node includes the zone and node number. Syntax: argoZONE-NODE. Example: argo3-1 is zone three, node one. Sixty-four compute nodes = sixteen zones X four computers (nodes) per zone:

Zones	Compute Nodes
zone16	argo16-4	argo16-3	argo16-2	argo16-1
zone15	argo15-4	argo15-3	argo15-2	argo15-1
zone14	argo14-4	argo14-3	argo14-2	argo14-1
zone13	argo13-4	argo13-3	argo13-2	argo13-1
zone12	argo12-4	argo12-3	argo12-2	argo12-1
zone11	argo11-4	argo11-3	argo11-2	argo11-1
zone10	argo10-4	argo10-3	argo10-2	argo10-1
zone9	argo9-4	argo9-3	argo9-2	argo9-1
zone8	argo8-4	argo8-3	argo8-2	argo8-1
zone7	argo7-4	argo7-3	argo7-2	argo7-1
zone6	argo6-4	argo6-3	argo6-2	argo6-1
zone5	argo5-4	argo5-3	argo5-2	argo5-1
zone 4	argo4-4	argo4-3	argo4-2	argo4-1
zone 3	argo3-4	argo3-3	argo3-2	argo3-1
zone 2	argo2-4	argo2-3	argo2-2	argo2-1
zone 1	argo1-4	argo1-3	argo1-2	argo1-1

The argo-new file server: User home directories on the master node are physically located on a separate machine whose sole purpose is to act as a file server.The 4 disk partitions on the file server are NFS-mounted* to the master node and to all compute nodes.
The argo-new scratch server: User scratch space on the master node is, as is the case for home directories, located on a separate machine. The single partition is NFS-mounted to the master and to all compute nodes.
There are two types of filesystems on argo-new:
- local partitions with a Linux ext3* filesystem
- remote partitions -- ones on other machines -- available as an Network FileSystem, or NFS*
A ext3 filesystem* (Linux Second EXTended) is one constructed on disk that is physically attached to the machine. Some portion of or all of the local disk is used to contruct a partition which becomes the basis for the filesystem.

An NSF filesystem* (Network FileSystem) is a filesystem that is local to one machine, but made to appear local to other machines by accessing it over a network.

Tigger and icarus vs. argo-new

Though tigger, icarus and argo-new are MIMD* systems, tigger and icarus are shared memory multiprocessor systems whereas argo-new is a distributed memory multicomputer.

Tigger/Icarus

Tigger is a shared memory multiprocessor system (a conventional computer) with multiple processors of the same type (symmetric) connected to multiple memory modules. Access to physical memory and processors is shared among processes. Syncronization among processes is accomplished by reading from and writing to shared memory which is controlled by a locking mechanism (spinlocks and semaphores). Control of the locks is the responsibility of the user. The path that connects processors to physical memory is internal and transparent to users. The data along with the executable code is contained in memory which is shared among the processes. By using threads, portions of a single program may run on different processors.

Argo-new

A distributed memory multicomputer system is a group of complete computers that are connected to one another. Each of the constituent machines, called nodes, is a self contained unit (has its own processor and local memory) capable of acting independently of the other machines in the group. The most common method of programming distributed-memory systems is message passing which permits processes to share data and coordinate their activities.

Argo-new has 64 compute nodes and apprximately 168GB of memory:

16 machines (argo4-4 -> argo1-1) * 4.0GB per = 64GB
+
16 machines (argo8-4 -> argo5-1) * 2.5GB per = 40GB
+
32 machines (argo16-4 -> argo9-1) * 2.0GB per = 64GB

   Computer        Computer        Computer        Computer         Computer
   (Argo1-1)       (Argo1-2)       (Argo1-3)       (Argo1-4)  ...  (Argo4-4)
+------------+  +------------+  +------------+  +------------+   +------------+
| +--------+ |  | +--------+ |  | +--------+ |  | +--------+ |   | +--------+ |
| |Processor |  | |Processor |  | |Processor |  | |Processor |   | |Processor |
| +--------+ |  | +--------+ |  | +--------+ |  | +--------+ |   | +--------+ |
|    |       |  |    |       |  |    |       |  |    |       |   |    |       |
| +--------+ |  | +--------+ |  | +--------+ |  | +--------+ |   | +--------+ |
| |Local   | |  | |Local   | |  | |Local   | |  | |Local   | |   | |Local   | |
| |memory  | |  | |memory  | |  | |memory  | |  | |memory  | |   | |memory  | |
| +--------+ |  | +--------+ |  | +--------+ |  | +--------+ |   | +--------+ |
+------------+  +------------+  +------------+  +------------+   +------------+
     |                |              |               |                | 
     |                |              |               |                |

+-----------------------------------------------------------------------------+
| Ethernet network                                                            |
+-----------------------------------------------------------------------------+

     |                |              |               |                |
     |                |              |               |                |
  (Argo5-1)       (Argo5-2)       (Argo5-3)       (Argo5-4)  ...  (Argo8-4)
+------------+  +------------+  +------------+  +------------+   +------------+
| +--------+ |  | +--------+ |  | +--------+ |  | +--------+ |   | +--------+ |
| |Processor |  | |Processor |  | |Processor |  | |Processor |   | |Processor |
| +--------+ |  | +--------+ |  | +--------+ |  | +--------+ |   | +--------+ |
|    |       |  |    |       |  |    |       |  |    |       |   |    |       |
| +--------+ |  | +--------+ |  | +--------+ |  | +--------+ |   | +--------+ |
| |Local   | |  | |Local   | |  | |Local   | |  | |Local   | |   | |Local   | |
| |memory  | |  | |memory  | |  | |memory  | |  | |memory  | |   | |memory  | |
| +--------+ |  | +--------+ |  | +--------+ |  | +--------+ |   | +--------+ |
+------------+  +------------+  +------------+  +------------+   +------------+

Master Node

Overview

Clients log into the master (also called the front-end), create/edit/compile programs, and submit them for execution on one or more of the compute nodes. Clients do not run their programs on the master. Repeat: user programs MUST NOT be run on the master . Instead, they are submitted to the compute nodes for execution.

Hardware

1U rack mounted system case (The depth of the system case is measured in U units. 1U = 1.75 inches.)
Dual AMD Opteron processors
4GB of memory
Dual NIC* (Network Interface Card, Intel Gigabit: eth0/eth1) with jumbo frames on eth0.
- eth0: 172.16.0.20
- eth1: 128.248.121.63
Two internal 72GB SCSI disks (sda and sdb)

Operating System

RedHat Linux 9.0 (Kernel level: 2.4.20-8smp)

Master Node Filesystems

There are two types of filesystems on argo:

local partitions with a Linux ext3* filesystem
remote partitions -- ones on other machines -- available as an Network FileSystem, or NFS*

A ext3 filesystem* (Linux 2nd EXTended) is one constructed on disk that is physically attached to the machine. Some portion of or all of the local disk is used to contruct a partition which becomes the basis for the filesystem.

An NFS filesystem* (Network FileSystem) is a filesystem that is local to one machine, but made to appear local to other machines by accessing it over a network.

Filesystems on the master node	Size	Available to compute nodes?	Access Method	Explanation
boot	120MB	No	local	local partition - assessible only to master
home/homesXX	1.2TB	Yes	NFS	home area on argo's home server. Click here for more information
opt	10GB	No	local	local partition - assessible only to master
root	2GB	No	local	local partition - assessible only to master
scratch	1.2TB	Yes	NFS	scratch area on argo-new's scratch server
tmp	10GB	No	local	temporary storage - assessible only to master
usr	10GB	No	local	local partition - assessible only to master
usr/common	20GB	Yes	NFS	Common area for software
var	10GB	No	local	local partition - assessible only to master

Compute Nodes

Overview

The compute nodes have ONE AND ONLY ONE purpose: run user programs. Clients do not have login access to the compute nodes. From the master node, clients submit programs to torque which executes them on one or more compute nodes. While clients are prohibited from logging into compute nodes, they may issue remote shell commands from the master to retrieve information. For example:

Hardware

64 Nodes	Manufacturer	Size	#/Type of processor	Memory	NIC	Disk (GB)	High-speed interconnect
16: argo4-4 - argo1-1	Sun	1U	Dual AMD Opterons	4GB	Dual-port GigE	One 72 SCSI	No
16: argo8-4 - argo5-1	Sun	2U	One 2.8GHz Xeon	2.5GB	Dual-port GigE	One 36 SCSI	No
16: argo12-1 - argo9-1	Sun	1U	Two 3.0GHz Xeon	2.0GB	Dual-port GigE	Two 36 SCSI	No
16: argo16-4 - argo13-1	Sun	1U	Two 3.0GHz Xeon	2.0GB	Dual-port GigE	Two 36 SCSI	No

Operating System

RedHat Linux 9.0 (Kernel level: 2.4.20-8smp)

Home Server

Overview

User home directories on the master node are physically located on a separate machine whose sole purpose is to act as a file server. There are four disk partitions of 300GB each on the server. Each partition is NFS-mounted to the master and all compute nodes:

Partition	Mount point on the argo file server	Filesystem - master and compute
/dev/sda1	/mnt/array_a	/home/homes50
/dev/sdb1	/mnt/array_b	/home/homes51
/dev/sdc1	/mnt/array_c	/home/homes52
/dev/sdd1	/mnt/array_d	/home/homes53

To see which filesystem your home directory is in, type:

echo $HOME

Hardware

Single Pentium III 550 MHz processor
- 16K L1* (cache memory on the chip)
- 512 L2* (cache memory on separate chip)
384MB DRAM Memory
Single port NIC* (Intel Gigabit: eth0) with jumbo frames
- eth0: 172.16.X.X
One 5GB Quantum disk for booting
Ten 60GB Maxtor disks for cluster home space

Operating System

RedHat

Network

Overview

Intranodal communication (communication among the nodes) is more demanding than nodal communication with the external environment. Each node may need to interact with other nodes, either in concert or independently. The purpose of the communcation may be:

To share small or large amounts of data,
To synchronize operations.

In the former case, a high bandwidth* communication path may be needed. In the latter case, low latency* communication is generally best. Nodal communication may also be classified by the initiator. While most communication is among user processes, there is also communcation for system activities, referred to as out-of-band-management.

Ethernet

Each node, master and compute, is connected to an ethernet network and the NIC within a compute node is gigabit. The ethernet NIC may be used for interprocess communication using vanilla MPI as well as for out-of-band-management (basically anything - NIS, NFS, etc - except interprocess communication):

+-----------+           Master            Switch          Switch           Switch           Switch
| Outside   |        +----------+      +---------+      +---------+      +---------+      +---------+
|  World    |  <--> eth1      eth0 <-->| | | | | | <--> | | | | | | <--> | | | | | | <--> | | | | | |         
+-----------+        +----------+      +---------+      +---------+      +---------+      +---------+
                                            |                |                |                |  
                                            |                |                |                |
                                            V                V                V                V
                                       +---------+      +---------+      +---------+      +---------+ 
                                       | argo4-4 |      | argo8-4 |      | argo12-4|      | argo16-4|
                                       +---------+      +---------+      +---------+      +---------+ 
                                       | argo4-3 |      | argo8-3 |      | argo12-3|      | argo16-3|
                                       +---------+      +---------+      +---------+      +---------+
                                       |   ...   |      |   ...   |      |   ...   |      |   ...   |
                                       +---------+      +---------+      +---------+      +---------+
                                       | argo1-1 |      | argo5-1 |      | argo9-1 |      | argo13-1|    
                                       +---------+      +---------+      +---------+      +---------+

Master:: 128.248.121.63 (eth1)
Master:: 172.16.0.20 (eth0)
Compute:: 172.16.zone.node (eth0)

Argo-new Compute Cluster Previous: Contents Next: Getting Started

2007-11-13 ACCC Systems Group