IBM Spectrum LSF (LSF) is used for job queuing and scheduling on the HPC cluster compute nodes. All jobs run on the cluster compute nodes must be submitted to LSF using the bsub command.
To assist users to manage their jobs and organize work plans it may be helpful to understand the factors affecting how jobs are selected for scheduling. This information is current as of August 2020, but is subject to change in response to changing work load on the cluster.
First, the highest priority project is identified based on share priority. Next, a user from that project is selected round-robin. If the user has more than one job waiting, a job is selected from the highest priority queue. If the user has more than one job waiting in that queue, the job with the earliest submission time is selected and assigned the available job slot.
Each project is assigned a base share. The project's share priority is reduced based on the number of processors currently being used by the project, by the CPU time used recently by the project, and by the total run time used recently by the project. Changes in number of processors is reflected immediately in the project's share priority. That is, as soon as a job completes the share priority would increase to reflect fewer processors in use. CPU and run time adjustments decay away over time - currently on the order of a day. So a project that used a large amount of CPU time in the morning would continue to have a somewhat lower share priority throughout the afternoon.
Share priority turns out to be the most important factor in determining which project has the highest priority for the next available job slot. Individual users within a project are selected round-robin. The bhpart command shows current share priorities and the bugroup command shows project membership of each individual user.
A project's base share is computed by taking the total number of job slots on university compute nodes and dividing by the number of projects. Partner projects, that is projects of faculty who have purchased cluster nodes, have the number of processors they have purchased added to the base share for their project. Because base shares must be specified as integer values the computed base shares are normalized such that non-partner base shares are equal to one.
Each LSF queue is assigned a priority. Jobs in higher priority queues tend to be scheduled first. Queue priority tends to have the most effect for selecting between jobs belonging to users within the same project. Queue priority is listed in descending priority order by the bqueues command.
All of the above listed factors being equal, jobs are scheduled on a first in, first out basis.
This is not an absolute process. Jobs waiting a very long time can accumulate sufficient priority to overcome low queue priorities, but in most cases this is the observed LSF scheduling behavior.