|
ACADEMIC COMPUTING and COMMUNICATIONS CENTER | |||||||
|
|
|
|
|
|
|
|
|
There is a software package on borg called the (L)oad (S)haring (F)acility, hereafter referred to as LSF, that clients must use to dispatch and run programs. The LSF User's Guide Version 3.1 in postscript format is available as file lsf3.1_batch.user.ps in the directory /usr/local/lsf/doc.
The following table lists in alphabetical order some of the more useful LSF commands. For detailed explanations, click on the command in the table and/or view the appropriate man page on borg. Examples have also been provided for most of the commands.
| Command | Explanation | Examples |
|---|---|---|
| bbot | Change the order in which your batch jobs are dispatched. | Yes |
| bhist | Display execution history of batch jobs. | Yes |
| bhosts | Display information about the batch server hosts. | Yes |
| bjobs | Display the status and other information about batch jobs. | Yes |
| bkill | Terminate a pending or running batch job. | Yes |
| bmod | Modify the options of a previously submitted batch job. | No |
| bparams | Display generic configuration parameters, including the default queue. | Yes |
| bpeek | Display the standard output and standard error that has been produced by an unfinished batch job | Yes |
| bqueues | Display queue information. | Yes |
| bresume | Resume a batch job that was suspended (also see bstop). | Yes |
| bsub | Submit a job for batch execution (also see xbsub). | Yes |
| bstop | Suspend a batch job (also see bresume). | Yes |
| bswitch | Moves pending/running jobs from queue to queue. | Yes |
| btop | Change the order in which your batch jobs are dispatched. | Yes |
| lsclusters | Display general configuration information about the borg cluster. | Yes |
| lshosts | List information about LSF host configuration. | Yes |
| lsid | Display cluster and current master hosts. | Yes |
| lsinfo | List resources available in the borg cluster. | Yes |
| lsload | Print current load information of host in borg cluster. | Yes |
| lsrun | Run a job interactively. | Yes |
| xbsub | A GUI for submitting a job to the LSF batch system (also see bsub). | No |
| xlsbatch | A GUI for monitoring jobs and other aspects of the LSF batch system. | No |
| xlsmon | A GUI for cluster load monitoring. | No |
| xbmod | A GUI for modifying the options of a batch job (also see bmod). | No |
Display the history of one or more jobs in the LSF Batch system.
JOB-BASED:
bhist [-b] [-w] [-l] [-a] [-d] [-p] [-s] [-t] [-f logfile | -n num_logfiles] [-C time0,time1] [-S time0, time1] [-D time0, time1] [-q queue] [-m host] [-u user | all] [-J job] [-P project] [-N spec] [jobId | "jobId[index]" ...]CHRONOLOGY-BASED:
bhist [-t] [-f logfile] [-T time0, time1]
| -b | Display the job history in a brief format. Default: display summary format. |
| -w |
Display the job history in a wide format without truncation. |
| -l | Display the job history in a (long) multi-line format, giving
detailed information about each job. Default: display summary format. |
| -a | Display all, both finished and unfinished jobs. Default: finished jobs are not displayed. |
| -d | Display only the finished jobs. |
| -p | Display only the pending jobs. |
| -s | Display only the suspended jobs. If option -l or -b is also specified, show the reason why each job was suspended. |
| -t | Display job events chronologically. Default: display events occurring in the past week. |
| -f logfile | Specify the file name of the event log file. Either an absolute or relative path name may be specified. Default: use the system event log file lsb.events. |
| -n num_logfile | Specify the number of event log files that are searched.
A value of 0 causes all log files in the cluster's log directory
to be searched. Default: 1. |
| -C time0,time1 | Display only those jobs whose completion or exit times were
between time0 and time1. Default: display all jobs that have completed or exited. |
| -S time0,time1 | Display only those jobs whose submission times were between
time0 and time1. Default: display all jobs that have been submitted. |
| -D time0,time1 | Display only those jobs whose dispatch times were between
time0 and time1. Default: display all jobs that have been dispatched. |
| -q queue | Display jobs submitted to the specified queue only. Default: display all queues. |
| -m host | Display jobs dispatched to the specified host only. Default: display all hosts. |
| -user | all | Display jobs submitted by the named user or by all users. Default: display the jobs submitted by the user who invoked this command. |
| -J job | Display all jobs with the specified name. |
| -P project |
Display only jobs belonging to the named project. |
| -N spec | Display CPU time consumed by the job. The appropriate CPU scaling factor for the specified host, or defined host model, is used to normalize the actual CPU time consumed by the job. |
| jobId | "jobId[index]" ... | Display the job or jobs with the specified jobId or "jobId[index]", where "jobId[index]" specifies selected job array elements. |
| -T time0,time1 | Consider only those job events occurring between time0 and time1. |
Display information about the server hosts in the LSF Batch system.
HOST-BASED:
bhosts [-R res_req] [-w | -l] [host ... | cluster ...]RESOURCE-BASED:
bhosts -s [resource ...]
| -R res_req | Display information about the hosts that meet the specified
resource requirements. Default: display information for all hosts in the local cluster. |
| -w | Display fields in a wide format without truncation. |
| -l | Display information in a (long) multi-line format. |
| host | cluster... | Display information for the named host(s) or cluster. Default: display information for all hosts in the local cluster. |
| -s | Display numeric shared resource information for the host on a per-resource basis. |
| resource | Specifies the numeric shared resource(s) about which information
will be displayed. Default: all numeric shared resources. |
Display the status and other information about jobs in the LSF Batch system.
bjobs [-w | l] [-a] [-d] [-p] [-s] [-r] [-A] [-q queue] [-m host | cluster] [-u user | all] [-J jobname] [-P project] [-N spec] [jobId | "jobId[index]" ...]
| -w | Display fields in a wide format without truncation. |
| -l | Display information in a (long) multi-line format. |
| -a | Display information about all jobs, including unfinished jobs (pending, running or suspended) and recently finished jobs. |
| -d | Display only recently finished jobs. |
| -p | Display only pending jobs, with the reasons they were not dispatched during the last dispatch turn. |
| -s | Display only suspended jobs, showing the reasons for suspension. |
| -r | Display only running jobs. |
| -A |
Display summarized information about job arrays. |
| -q queue | Display only jobs in the named queue. |
| -m host | cluster ... | Display only jobs dispatched to the named host or host group. |
| -m user | all | Display jobs submitted by the named user or by all users. Default: display the jobs submitted by the user who invoked this command. |
| -J jobname | Display all jobs with the specified name. |
| -P project | Display only jobs belonging to the named project. Default: display jobs belonging to all projects. |
| -N spec | Display CPU time consumed by the job. The appropriate CPU scaling factor for the specified host, or defined host model, is used to normalize the actual CPU time consumed by the job. |
| jobId | "jobId[index]" ... | Display the job or jobs with the specified jobId or "jobId[index]", where "jobId[index]" specified selected job array elements. |
Send a signal to one or more unfinished batch jobs.
bkill [-l] [-s value | name] [-q queue] [-m host] [-u user | all] [-J jobname] [jobId | "jobId[index]" ...]
| -l | Display the set of signal names supported. (This is a subset of those supported by /bin/kill and is platform dependent). |
| -s value | name | Send the signal specified by value or name to the specified
jobs. Default: SIGKILL or 9. |
| -q queue | Send a signal to the jobs in the specified queue. |
| -m host | Send a signal to the jobs dispatched to the specified host or host group. Ignored if a job ID other an 0 is specified. |
| -u user | all | Send a signal to the jobs submitted by the named user. Ignored if a job ID other than 0 is specified. |
| -J jobname | Send a signal to the named jobs. Ignored if a job ID other than 0 is specified. |
| jobId | "jobId[index]" ... | Send a signal to the specified jobs or job array elements. |
Modify the options of a previously submitted batch job. bmod uses a subset of the bsub options. To reset an option to its default value, use the option string followed by n.
bmod [-x | -xn] [-r | -rn] [-N | -Nn] [-B | -Bn] [-q queue ... | -qn]
[-m host ... | -mn] [-n minimum[maximum] | -nn] [-R res_req | -Rn]
[-J jobname | -Jn] [-b begin | -bn] [-t terminate | -tn] [-i infile | -in]
[-o outfile | -on] [-e errfile | -en] [-u mail_user | -un]
[[-f "lfile op [rfile]"] ... | -fn]] [-c cpu[/host] | -cn]
[-W run [/host] | -Wn] [-F file | -Fn] [-M mem | -Mn]
[-D data | -Dn] [-S stack | -Sn] [-C core | -Cn]
[-k dir [period] | -Kn] [-L shell | -Ln] [-G user_group | -Gn]
[-P project | -Pn] [-w depend | -wn] [-E "pre_exec [arg ...]" | -En]
[-Z "new_command"] jobid
| -x | Exclusive execution mode. |
| -r | Specifies that the job can be rerun. |
| -N | Notify the submitter by mail when the job finishes. If -o option is not given, the standard output from the job is included in the message. |
| -B | Notify the submitter by mail when the job is dispatched and begins execution. |
| -K | Submit a batch job and wait for the job to complete. |
| -q queue ... | Submit the job to the specified queue(s). Default: the system default queue. |
| -m host[+[level]] ... | Limit candidate hosts for executing this job to the specified host(s). '+' is used to specify preference. Level is a positive number specifying the preference level of a host. |
| -n min[,max] | The minimum and maximum number of processors requested to
run a (parallel) job. Default: one processor. |
| -R res_req | Execute the task on a host that meets the resource requirements.
Default: type==local unless a model or Boolean resource is specified, in which case type==any order[r1m:pg]. |
| -J jobname | Assign the specified character string to the job. |
| -b begin | Assign the specified character string to the job. |
| -t terminate | The job termination deadline. Default: allow the job to run as long as its resource limits permit. |
| -i infile | The job gets its standard input from the specified file.
Default: /dev/null. |
| -o outfile | Write the standard output of the job to the specified file. Default: sent by mail to the user. |
| -e errfile | Write the standard error output of the job to the specified file. |
| -u user | Send mail to a specified email address. Default: send mail to the submitting user according to the format specified in the lsf.conf file. |
| -f "[lfile op [rfile]]" ... | Copy a file between the local (submission) host and the remote (execution) host. Op is one of '>', '<','<<', '><' (or '<>'). op is illegal without at least one of lfile or rfile. |
| -E "pre_exec [arg ...]" | Execute the pre-execution command on the host to which the job is dispatched to run (or on the first host selected for a parallel job) before actually running the job. If the pre-execution command exits with 0, start the primary job; otherwise the job remains pending. |
| -c CPU[/spec] | Set the total CPU time limit for this job. Default: no limit. The optional spec defines the appropriate host CPU scaling factor. |
| -W run[/spec] | Set the wall-clock run time limit of this job. Default: no limit. Spec is the same as in the -c option. |
| -F file | Set a per-process (soft) file size limit for each process
belonging to this job. Default: no soft limit. |
| -M mem | Set the total process resident set size limit in KB for the
whole job. Default: no limit. |
| -D data | Set a per-process (soft) data segment size limit for each
of the processes that belong to this job. Default: no soft limit. |
| -S stack | Set a per-process (soft) stack segment size limit for each
of the processes that belong to this job. Default: no soft limit. |
| -C core | Set a per-process (soft) core file size limit for all the
processes that belong to this job. Default: no soft limit. |
| -k dir [period] | This job can be checkpointed. The checkpoint directory is a relative or absolute path name, and is used for restarting the job (see brestart) Multiple jobs can checkpoint into the same directory. Optionally, a checkpoint period in minutes may be specified. |
| -w depend | Only when the dependency condition is satisfied (TRUE),
will the job be considered for dispatch. The expression is composed of job
or system conditions using '&&', '||' and '!' logic operators.
Conditions are job status, calendar, file event, or user event. Job status conditions: started(jobId | jobName), done(jobId | jobName), exit(jobId | jobName), and ended(JobId | jobName) Calendar conditions (calendars are only available with LSF JobScheduler): calendar(cal_spec) or cal(cal_spec) File event conditions: file(arrival(location) | exist(location) | size(location) | age(location)) User event conditions: event(spec) or ev(spec) |
| -L shell | Specifies the name of the login shell used to initialize the execution environment. |
| -P project | Specifies the name of the project that this job will be charged to. |
| -G user_group | Specifies the name of the LSF user group to which the job will belong. |
Display information about the configurable LSF Batch system parameters. By default only display the most interesting parameters.
bparams [-l]
| -l | Display all parameters along with brief descriptions and their corresponding names. |
Display the standard output and standard error that has been produced by an unfinished batch job in the LSF Batch system.
bpeek [-f] [-q queue | -m host | -J jobname | jobId | "jobId[index]"]
| -f | Display the output of the job using the command "tail
-f". Default: use the command "cat". |
| -q queue | Display the output of the most recently submitted job in the specified queue. |
| -m host | Display the output of the most recently submitted job that has been dispatched to the specified host. |
| -u user | all | Send a signal to the jobs submitted by the named user or by all users. Ignored if a job ID other than ) is specified. |
| -J jobname | jobId | "jobId[index]" | Display the output of the most recently submitted job that
has the specified job name, the output of the job with the given ID, or
the output of the specified job array element. Default: display the output of the most recently submitted job that satisfies options -q, -m, or -J. |
Display information about batch queues in the LSF Batch system.
bqueues [-w l -l | -r] [-m host | cluster] [-u user] [queue ...]
| -w | Display fields in a wide format without truncation. |
| -l | Display information in a (long) multi-line format. |
| -r | Same as -l, but also display recursively the share account tree of the fairshare queue (if it is defined). |
| -m host | cluster | Display queues from which jobs may be dispatched to the named
host or host group. Default: all hosts. Otherwise, display the queues in the named remote cluster. |
| -u user | Display the queues to which the named user or user group
may submit batch jobs. Default: all users. |
| queue | Display information about the named queue(s). |
Suspend or resume one or more unfinished batch jobs.
Sample bstop commands and output
Sample bresume commands and outputbstop [-q queue] [-m host] [-u user] -J [jobname] [jobId | "jobId[index]" ...]
bresume [-q queue] [-m host] [-u user] -J [jobname] [jobId | "jobId[index]" ...]
| -q queue | Stop or resume the job(s) in the specified queue. |
| -m host | Stop or resume the job(s) dispatched to the specified host
or host group. Ignored if a job ID other than 0 is specified. |
| -u user | Stop or resume the job(s) submitted by the named user. Ignored if a job ID other than 0 is specified. |
| -J jobname | Stop or resume the named job(s). Ignored if a job ID other than 0 is specified. |
| jobId | "jobId[index]" ... | Stop or resume the specified job(s) or job array elements. |
Submit a job for execution in the LSF Batch system. If no command is given, bsub reads job commands from the standard input. If the standard input is a controlling terminal, bsub prompts for commands.
bsub [-H] [-x] [-r] [-N] [-B] [-I | -Ip | -Is | -K] [-q queue ...]
[-m host[=[level]] ...] [-n min[,max]] [-R res_req] [-J jobname]
[-b begin] [-t terminate] [-i infile] [-o outfile] [-e errfile] [-u user]
[[-f "lfile op [rfile]"] ...] [-E "pre_exec [arg ...]"] [-c cpu[/spec]]
[-W run[/spec]] [-F file] [-M mem] [-D data] [-S stack] [-C core]
[-k "dir [period]"] [-w depend] [-L shell] [-P project]
[-G user_group [command [arg ...]]
| -H | Hold the job in the PSUSP state when the job is submitted. |
| -x | Exclusive execution mode. |
| -r | Specifies that the job can be rerun. |
| -N | Notify the submitter by mail when the job finishes. If -o option is not given, the standard output from the job is included in the message. |
| -B | Notify the submitter by mail when the job is dispatched and begins execution. |
| -I | Submit an interactive job. bsub blocks until the job is terminated.
See -i, -o, and -e options. Default: job interacts with user's terminal. |
| -Ip | Submit an interactive job and create a pseudo-terminal when the job starts. |
| -Is | Submit an interactive job and create a pseudo-terminal with shell mode support when the job starts. Shell mode CTRL-C and CTRL-Z keys. |
| -K | Submit a batch job and wait for the job to complete. |
| -q queue ... | Submit the job to the specified queue(s). Default: the system default queue. |
| -m host[+[level]] ... | Limit candidate hosts for executing this job to the specified host(s). '+' is used to specify preference. level is a positive number specifying the preference level of a host. |
| -n min[,max] | The minimum and maximum number of processors requested to
run a (parallel) job. Default: one processor. |
| -R res_req | Execute the task on a host that meets the resource requirements.
Default: type==local unless a model or Boolean resource is specified, in which case type==any order[r1m:pg]. |
| -J jobname | Assign the specified character string to the job. |
| -b begin | Assign the specified character string to the job. |
| -t terminate | The job termination deadline. Default: allow the job to run as long as its resource limits permit. |
| -i infile | The job gets its standard input from the specified file.
Default: /dev/null. |
| -o outfile | Write the standard output of the job to the specified file. Default: sent by mail to the user. |
| -e errfile | Write the standard error output of the job to the specified file. |
| -u user | Send mail to a specified email address. Default: send mail to the submitting user according to the format specified in the lsf.conf file. |
| -f "[lfile op [rfile]]" ... | Copy a file between the local (submission) host and the remote (execution) host. op is one of '>', '<','<<', '><' (or '<>'). op is illegal without at least one of lfile or rfile. |
| -E "pre_exec [arg ...]" | Execute the pre-execution command on the host to which the job is dispatched to run (or on the first host selected for a parallel job) before actually running the job. If the pre-execution command exits with 0, start the primary job; otherwise the job remains pending. |
| -c cpu[/spec] | Set the total CPU time limit for this job. Default: no limit. The optional spec defines the appropriate host CPU scaling factor. |
| -W run[/spec] | Set the wall-clock run time limit of this job. Default: no limit. spec is the same as in the -c option. |
| -F file | Set a per-process (soft) file size limit for each process
belonging to this job. Default: no soft limit. |
| -M mem | Set the total process resident set size limit in KB for the
whole job. Default: no limit. |
| -D data | Set a per-process (soft) data segment size limit for each
of the processes that belong to this job. Default: no soft limit. |
| -S stack | Set a per-process (soft) stack segment size limit for each
of the processes that belong to this job. Default: no soft limit. |
| -C core | Set a per-process (soft) core file size limit for all the
processes that belong to this job. Default: no soft limit. |
| -k dir [period] | This job can be checkpointed. The checkpoint directory is a relative or absolute path name, and is used for restarting the job (see brestart) Multiple jobs can checkpoint into the same directory. Optionally, a checkpoint period in minutes may be specified. |
| -w depend | Only when the dependency condition is satisfied (TRUE),
will the job be considered for dispatch. The expression is composed of job
or system conditions using '&&', '||' and '!' logic operators.
Conditions are job status, calendar, file event, or user event. Job status conditions: started(jobId | jobName), done(jobId | jobName), exit(jobId | jobName), and ended(JobId | jobName) Calendar conditions (calendars are only available with LSF JobScheduler): calendar(cal_spec) or cal(cal_spec) File event conditions: file(arrival(location) | exist(location) | size(location) | age(location)) User event conditions: event(spec) or ev(spec) |
| -L shell | Specifies the name of the login shell used to initialize the execution environment. |
| -P project | Specifies the name of the project that this job will be charged to. |
| -G user_group | Specifies the name of the LSF user group to which the job will belong. |
Switch one or more unfinished (running or suspended) jobs from on queue to another.
bswitch [-q queue] [-m host] [-u user] -J [jobname] [jobId | "jobId[index]" ...]
| -q queue | Switch job(s) from the specified queue. Ignored if a job ID other than 0 is specified.. |
| -m host | Switch job(s) dispatched to the named host or host group.
Ignored if a job ID other than 0 is specified. |
| -u user | Switch job(s) submitted by the named user. Ignored if a job ID other than 0 is specified. |
| -J jobname | Switch the named job(s). Ignored if a job ID other than 0 is specified. |
| destination | The destination queue to which job(s) will be switched. |
| jobId | "jobId[index]" ... | Switch the specified job(s) or job array elements. |
Move a pending job to the top (beginning) or bottom (end) of its queue. This only effects the user's own jobs.
Sample btop commands and output
Sample bbot commands and outputbtop jobId | "jobId[index]" [position]
bbot jobId | "jobId[index]" [position]
| jobId | "jobId[index]" | Move the specified job or job array element. |
| position | A positive number that indicates the target position of the job relative to the beginning (for btop) or end (for bbot) of the queue. |
Display general configuration information about LSF clusters; sample commands and output.
lsclusters [-l] [cluster ...]
| -l | Display information in a (long) multi-line format. |
| cluster ... | Display information about the named cluster(s). Default: all clusters. |
Display information about LSF host configuration, including: host name, host type, host model, CPU normalization factor, number of CPUs, total memory, and available resources, on either a per-host or per-resource basis.
HOST-BASED:
lshosts [-w | -l] [-R res_req] [host ... | cluster ...]RESOURCE-BASED:
lshosts -s [resource ...]
| -w | Display information in wide format. |
| -l | Display information in a (long) multi-line format. |
| -R res_req | Display information about the hosts that meet the specified
resource requirements. Default: display information for all hosts |
| host... | cluster... | Display configuration information for the named host(s)
or cluster(s). Default: display information for all hosts. |
| -s | Display static shared resource information on a per-host basis. |
| resource ... | Specifies the static shared resource(s) about which information
will be displayed. Default: all static shared resources. |
Display the LSF release version, the name of the local load sharing cluster, and the name of its master LIM host.
lsid
Display information about the LSF configuration, including available resources, host types and host models.
lsinfo [-l] [-r] [-m] [-t] [resource ...]
| -l | Display information in a (long) multi-line format. |
| -r | Display information about configured resources. |
| -m | Display information about configured host models. |
| -t | Display information about configured host types. |
| cluster ... | Display information about the named cluster(s). Default: all clusters. |
Display load information on hosts, in order of increasing load, on either a per-host or per- resource basis.
HOST-BASED:
lsload [-N | -E] [-l] [-R res_req] [-I index] [-n number] [host ... | cluster ...]RESOURCE-BASED:
lsload -s [resource ...]
| -N | Return normalized CPU run queue length load indices. CPU
run queue values are normalized using the number of CPUs and the CPU factors
of the hosts. Default: return raw load indices. Options -N and -E are mutually exclusive. |
| -E | Return effective CPU run queue length load indices. CPU run
queue values are normalized using the number of CPUs of the hosts. Default: return raw load indices. Options -N and -E are mutually exclusive. |
| -l | Display information in a long format. |
| -R res_req | Display information about the hosts that meet the specified
resource requirements. Default: type==any order[r1m:pg]. |
| -I index | Display only the specified load indices. List of load index names are separated by ':'(r1m:pg:ut). |
| -n number | Display up to the specified number of hosts that best satisfy
the resource requirements. Default: display all hosts that satisfy the resource requirements. |
| host... | cluster... | Display information for the named host(s) or cluster(s).
Default: display information for all hosts. If -R is specified, display all hosts that satisfy the resource requirements. |
| -s | Display the load on a per-resource basis. |
Submit a task to the LSF system for execution, possibly on a remote host.
lsrun [-S] [-P] [-l] [-v] [-m host ... | cluster ...] [-R res_req ...]
| -S | Create a pseudo-terminal with shell mode support when starting a remote task. Shell mode support is required for applications which redefine the CTRL-C and CTRL-Z keys. |
| -P | Create a pseudo-terminal when starting a remote task. |
| -l | Always execute the task locally if remote execution fails.
Default: execute locally only if the local host also satisfies the resource requirements. |
| -v | Verbose mode. The name of the selected host is displayed. |
| -m host | cluster... | Specify the candidate host(s) or cluster(s) for executing the task. If none of the candidate hosts satisfy the resource requirements, the task will not run. If only one host is specified, this option overrides the -R option and the task will be executed on that host. |
| -R res_req |
Execute the task on a host that meets the resource requirements. If -R
and -m are not specified, the resource requirements from the remote
task list are used. |
| 2001-1-29 ACCC Systems Staff |
|
