This page lists how to use various resource strings to request the proper hardware from LSF, as well as other elements necessary to efficiently use the cluster.
See the following video segment on finding specs for nodes on the cluster.
Resource strings are used to ensure that a job will be placed on nodes that have the resources the job requires. There are resources that are native to LSF, such as the number of cores or memory needed, and there are resources that have been defined locally for the Henry2 cluster, such as model name or node type.
Resource strings may contain a number of sections:
To specify a queue, use
In general, users should not specify a queue. When no queue is specified, LSF will choose the most appropriate queue based on the number of cores and time requested from the set of default queues.
The exceptions are partner queues and specialty queues, which are queues with special resources.
The queues available to a user can be displayed by using
bqueues -u user_name.
The properties of a queue can be displayed by using
bqueues -l queue_name.
Queue priority is determined by several factors including fair share priority, queue priority, and time of submission.
Shared memory jobs must be placed on a single node, or host. Some memory intensive MPI jobs or hybrid parallel jobs must limit the number of tasks per node.
To specify a span type, use
|hosts||Maximum number of hosts to confine tasks|
|ptile||Maximum number of tasks per host|
Set a usage case, such as higher memory. Usage is per host. See the generic LSF template for syntax information.
To specify a usage type, use
To specify a resource, use
LSF will not show an error if the user specifies a combination of resources that do not exist. For example, -R "select[hc model==Gold6130]" would result in job pending indefinitely as the Gold6130 model processors have 16 cores and hc requests processors with six cores.
The following is a list of the types of resources available on Henry2 and a description of each.
The required cores per node is specified by the number of cores per processor. Each node has two of the same model Intel Xeon processors. A specification of hc would select a node with two six-core processors, i.e., a node with 12 cores.
|qc||Processor model with four (quad) cores|
|hc||Processor model with six (hexa) cores|
|oc||Processor model with eight (octa) cores|
|tc||Processor model with ten cores|
|twc||Processor model with twelve cores|
|stc||Processor model with sixteen cores|
Software compiled on one type of architecture may not run on another type of architecture, resulting in an error of illegal instruction. LSF resources may be used to specify the instruction set architecture (ISA).
|sse||Processor model with SSE instructions|
|sse2||Processor model with SSE2 instructions|
|ssse3||Processor model with SSSE3 instructions|
|sse4_1||Processor model with SSE4 v1 instructions|
|sse4_2||Processor model with SSE4 v2 instructions|
|avx||Processor model with AVX instructions|
|avx2||Processor model with AVX2 instructions|
Similar to the ISA compatibility issues described above, a given software may not be compatible with all models of GPU.
|rtx2080||Node with attached Nvidia RTX 2080 GPU|
|gtx1080||Node with attached Nvidia GTX 1080 GPU|
|p100||Node with attached Nvidia P100 GPU|
|k20m||Node with attached Nvidia K20m GPU|
|m2070||Node with attached Nvidia M2070 GPU|
|m2070q||Node with attached Nvidia M2070Q GPU|
|m2090||Node with attached Nvidia M2090 GPU|
The type of interconnect may be specified.
LSF will not show an error if a job is placed in a queue not containing the specified interconnect. For example, when using ib, the job must be placed on a queue containing nodes with InfiniBand. Queues available to all users that have ib include standard_ib and mixed_ib.
Model definitions used for Henry2 nodes. These correspond to Intel Xeon model numbers of the processors on the nodes. Each node has two of the same model Intel Xeon processors. Here is a site with filter and search capabilities that lists processor model specifications.
To specify a specific model of processor, use
The following is a list of the model numbers currently available on Henry2.
Run a job with 48 tasks (-n 48) on four nodes with 12 cores per node.
bsub -n 48 -W 120 -R "select[hc] span[ptile=12]" < job_script_name
or using special select syntax
bsub -n 48 -W 120 -R "hc span[ptile=12]" < job_script_name
Nodes have two processors and the resource name defined for nodes with 6-core processors is hc. This job would fully occupy 4 nodes.
Run a job with 50 tasks with the tasks distributed 10 per node.
bsub -n 50 -W 200 -R "span[ptile=10]" < job_script_name
This resource string does not specify anything about the node selection criteria beyond needing 10 cores on each node. If the job were scheduled on nodes with 12 cores per node it is possible that LSF would schedule other jobs on the nodes being used for this job to occupy the remaining cores. In general, it is desirable to fully utilize nodes to avoid potential contention from other jobs.
See the generic template for creating a detailed batch script for more information and examples about how to specify LSF resources.
uname -a # OS and kernel info cat /etc/centos-release # Linux distribution lscpu # CPU info cat /proc/cpuinfo # processor info cat /proc/meminfo # memory info nvidia-smi # GPU info