	Academic Computing and Communications Center

The qsub command

Overview

All user jobs are run by torque.

The qsub command is how you submit a job, ANY job - serial, parallel, interactive, or non-interactive - to torque.

Syntax

b cannot run in partition DEFAULT (idle procs do not meet requirements:

The format of the qsub command is:

qsub -V script_file

where script_file is a text file containing, among other things:

the name of your executable program
options that tell torque how to run your program

Repeat: the operand to the qsub command MAY NOT BE an executable program (a binary file).

WRONG:

For example

Doing so will result in your program not running. Your job will be assigned an id and will APPEAR as if it is running if you immediately do a qstat command:

Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
2256.argo-new.c jsmith   staff    delta        6427  64  --    --  336:0 R   --
   argo4-4/0

Looks like it's running - the status column (S) contains an R for running. But, after a minute or so, the job will cancel and a message like the following will be in your standard error file:

2256

The 2256 in the message is the id; your id, obviously, will be something else.

More Detailed information about script files and qsub

Below is a sample text file, called my_script, that contains some basic directives (commands) to torque:

#!/bin/csh
#PBS -m be
#PBS -e ***homedir***/a.out.error
#PBS -o ***homedir***/a.out.output
#PBS -N delta
***homedir***/a.out

You submit the text file to torque using the qsub command:

qsub -V my_script

After submitting a job for execution, the job is assigned a number, an id, which identifies it and is used with other commands to, among other things, monitor the job status (is my job running, is it awaiting execution, etc).

You are not limited to just what you see in the sample script. Other commands are available including but not limited to shell commands like cd, ls, grep, and rm. The sample has some basic directives -- to get you started.

Each line that has a torque directive must start with #PBS. Shell commands, on the other hand, do not.

Let's take a detailed look at the script.

Line #1 of the script: #!/bin/csh

The first line tells torque to run the script under the C-shell. You are not required to use the C-shell. If you want to run your script under the bash shell, replace the line with #!/bin/bash. Or, don't include either line and let the system use your login, command shell.

The line is not a torque directive, unlike the next four lines, and does not start with #PBS.

Line #2 of the script: #PBS -m be

The second line is a directive (it starts with #PBS) and it directs torque to inform you via email (-m) when your job starts (b for begins) and when it ends (e for ends).

The line is not required; it may be altered or removed entirely. Here are two other permutations:

Directive	Description
#PBS -m b	Let me know when my job starts but not when it end
#PBS -m e	Let me know when my job ends but not when it starts

Just because you've submitted your job for execution doesn't mean it will begins running immediately. As was explained here, your job may wait until requested resources become available. Or, it may never run because you requested too many resources and/or resources that will never be available to you. That's why the email, letting you know when your program begins executing, is useful. If you submit a job that will run for an extended period of time (days or weeks), it's convenient to be informed of its completion.

The email is not sent to a maildrop on argo but rather to your UIC maildrop (***netid***@uic.edu). To change the email destination, modify the content of the .forward file in your home directory. If you don't have a .forward file, create one with your maildrop on a single line (netid@uic.edu). Example:

jsmith@uic.edu

Line #3 of the script: #PBS -e homedir/a.out.error

The third line identifies the path to and name of the standard error file for your job. (For a explanation of standard output and standard error files, click here.). As was explained, torque has a default naming scheme for the standard error file. By virtue of line three, you are overriding the default and giving the file a different name. In this case, a.out.error in your home directory. Do not hardcode ***homedir*** into the script:

WRONG:

Replace the ***homedir*** with the path to your home directory. Type the following two commands at the command prompt to get the path to your home directory:

For example, if the output is:

/home/homes50/jsmith

then the line in the script should be:

/home/homes50/jsmith

In the discussion below, always replace ***homedir*** with the path to your home directory.

If you don't include the line and it's not required, then the default scheme is used. As was explained, the name of the default is constructed from the following information:

jobname.ejobid

For example, if you submit the a.out job and the job is assigned the job number 3603, then the error file is a.out.e3603. The advantage of the default is that each job submission creates a new error file and does not replace an existing one. The downside - each new submission creates a new error file, cluttering your home directory with hundreds of them. As you accumulate too many, your response time when you use shell commands like ls or rm is negatively impacted.

You are STRONGLY encouraged to erase error files after reviewing the content.

Line #4 of the script: #PBS -o homedir/a.out.output

The fourth line is for standard output. The same logic that applied to standard error applies to standard out. If you use the default scheme, then a new file is created with each submission. The default name for standard output:

jobname.ojobid

Line #5 of the script: #PBS -N delta

The fifth line assigns the name delta to the job. Assigning a name is not required. If you don't assign a name to the job, then the default scheme is to use the name of your text file as the name of the job.

Line #6 of the script: homedir/a.out

The sixth line is the program to execute. Obviously, this is required. Without the line, your script does nothing. You are strongly encouraged to use the full path to the executable. Sample:

/home/home50/jsmith/a.out

The following is a BAD way to do identify the location of your executable and will, most likely, cause you problems:

./a.out

If you write your own programs, it is best though not required to include the full path to your files in open statements. For example, in Fortran:

Or, include the cd command in your script. Example:

#!/bin/csh
cd ***homedir***/tmp
***homedir***/tmp/a.out

Files opened by the a.out program will be written to the current directory which is ***homedir***/tmp.

You may use environmental variables in your script. For example, the path to your home directory is contained in the variable $HOME. Instead of hardcoding the path to your home, you may substitute the variable $HOME:

instead of

However, if you use environment variables, you must inform torque that you are doing so. The way to do that is use the -V option with qsub:

qsub -V my_script

Best practice: always include the -V option.

Identifying a queue to run your job

The command for submitting a job on the cluster is:

qsub -V my_script

Since you didn't specify a queue, the system will route the job to a default queue.

For staff and faculty, the default is the staff queue.
For students, the default is student_short.

To specify a queue, you have two options:

Include the queue name as an operand to the qsub command:

OR

Put the queue name in the script file (my_script):
and submit the script without the queue name on the qsub line:

There is a second queue for staff (not for students); it is called dedicated. The advantage of dedicated is that you will have exclusive use of a single, dual core, dual cpu machine. Your job will be the only one running on the node assigned to the queue (unlike nodes in any other queue where your running job shares the node with other running jobs). The downside - you get it for only 30 minutes. To use the dedicated queue, just substitute the word dedicated for staff:

qsub -V -q dedicated my_script

-q dedicated

qsub -V my_script

Using more than one machine/processor

The number of nodes you request to run your job is specified, like the destination, either on the qsub line or in the script. If you are running a serial job (a serial job uses just one node and one core), which is the default, you don't have to specifcally request the single node. The following request for a single node is unnecessary:

-l nodes=1

It's the same as doing:

qsub -V -q staff my_script

If you will be running a parallel job using multiple nodes, then you must request (hardcode) the number nodes you want allocated:

Put the number of nodes on the qsub line:
Put the number of nodes in the script file:
and submit the script without the number of nodes on the qsub line:

In both cases, the user is requesting four nodes.

You should not request a particular node. Depending on a variety of factors, the system administrator for the cluster may move nodes from one queue to another. There is no guarantee that a particular node remains in a particular queue. If you reqeust a node by name and that node is not in the requested/default queue, then the job will not run because the resource - the node - is not available. Users should direct a job to a queue and not to nodes:

  Wrong: qsub -V -l nodes=argo1-1 my_script
  Right: qsub -V -1 nodes=1 -q student_long my_script

  Wrong: qsub -V -l nodes=argo1-1+argo1-2 my_script
  Right: qsub -V -l nodes=2 -q student_long my_script

  Wrong: qsub -V -l nodes=argo1-1:ppn=2+argo1-2:ppn=2 my_script
  Right: qsub -V -l nodes=2:ppn=2 -q student_long my_script

You want multiple cores. Multiple cores are requested by means of the ppn operand which follows the request for the number of nodes. Same as before: you may use one of the two formats to make the request. For the purposes of brevity, only the first method is detailed; you should be able to construct the second method from examples of other qsub arguments.

Use 3 nodes but only one core per node:
Use 3 nodes and two core per node:
Use 1 node and two cores on it:

The ppn value may be included in the script file instead of on the command line. Examples:

ppn=2

Obviously, you specify the ppn only when it is greater than one, one being the default.

As stated before, one core is the default. If you don't specify the number of cores, then only one is assigned. The following two command invocations do the same thing; both request one core (the first includes the request; the second, let the system default to one core):

is the same as

qsub -V -l nodes=1 -q staff my_script

The requested cores will be on the same machine. If you request one machine with four cores, the system will look for a single machine having four cores. If it fails to find a matching machine, it will NOT attempt to allocate two machines and two cores on each of the two machines. Instead, your job does not run because the system will be unable to satisfy your request. The resources (not just cores but other requested resources including but not limited to nodes) may be owned by another job and, when that job completes and the resource(s) become available, they will be assigned to your job. However, if you request resources that will never be available ("I want eight cores on one node" is an example of a nonexistent resource), your job will never run. For more detailed information about why run program might not run, click here.

A VERY IMPORTANT POINT

If your program or software is neither parallel nor distributive, specifying more than one node and/or more than one processor DOES ABOSLUTELY NOTHING other than waste resources that could be used by some other client.

If you've written your own program using a programming language like C or Fortran (or any other language) and you haven't included a paradigm such as MPI, then:

If you are not familiar with the terms MPI, OpenMPI, MPICH2, or LAM or if you don't know what the terms parallel or distributive mean, then:

If your software doesn't specifically mention that it parallel/distributive, then:

Gaussian is an example of a parallel/distributive package; GaussView, produced by the same company that wrote Gaussian, is not. If your program is not parallel/distributive, then it is SERIAL. That means it execute on a single processor on a single node.

If the software or program is SERIAL, do not include the nodes and ppn components on the qsub command. Including them will NOT (REPEAT, WILL NOT) make your program run faster.

What are the limits regarding multiple processors and machines

Currently, there is a chart that is displayed when you login. If you failed to notice the chart or if it scrolled off the screen, you may display it along with other login messages by using the following argo command:

cat /etc/motd

At some point, that chart will be removed from "message of the day file" (motd). There are commands that you can use to display queue limits:

Maximum number of nodes a student may use in the student_short queue:

To see other policies (rules) of the student_short queue:

qmgr -c "list queue student_short"

The two samples use the student_short queue. You may substitute the name of other queues (student_medium, student_long, staff, dedicated) to get the corresponding settings for them:

qmgr -c "list queue staff" | grep resources_max.nodect
qmgr -c "list queue staff" | grep resources_max.ncpus
qmgr -c "list queue staff"

How long can my job run

Queues have two values regarding how long a job may run: a default and a maximum. The default is used when a user does not specify how long to run the job, called walltime (more about this in a moment). The maximum is exactly that: regardless of how much time a user requests, a job is permitted to run no longer than the maximum. The maximums (per queue) are shown in the output of the qstat -q command (the walltime column):

The defaults are shown using the qmgr command:

  qmgr -c "list queue student_short"  | grep resources_default.walltime 
  qmgr -c "list queue student_medium" | grep resources_default.walltime
  qmgr -c "list queue student_long"   | grep resources_default.walltime
  qmgr -c "list queue staff"          | grep resources_default.walltime
  qmgr -c "list queue dedicated"      | grep resources_default.walltime

Walltime is specified, like the queue, as either an operand on the qsub command or as a value in the text file. In your first run, let the system use the default. If you are running an interactive job, specify a value considerably less since you know how long you plan to interact with the software. Walltime MUST be requested using the format HHH:MM:SS and the system interprets the values from right to left. Examples:

Put the walltime on the qsub line:

Put the walltime in the script file:
and submit the script without the walltime on the qsub invocation:

There is a gotcha regarding walltime. As was stated, the system reads the walltime specification from right to left and ALL three fields MUST be included even if just to contain zeroes. As an example, the following format does not request 720 hours; instead, it asks for only 12 hours:

walltime=720:00

The first field (00) is the number of seconds requested and the second field (720) is the number of minutes. 720 minutes (divided by 60 minutes per hour) translates to 12 hours. To request 720 hours:

walltime=720:00:00

There is no command to tell you how long a job will run (in other words, what to enter as the walltime). You just have to estimate.

Documentation

For more information about the qsub command, see the man page (man qsub).

2010-8-27 ACCC Systems Group