Running Jobs on Private Condos

As mentioned on the page Running Jobs, users who are authorized to use private condos can submit the jobs on their private condos by adding below two PBS directives in their job script.

#PBS -A ACF-UTK0124 #PBS -l qos=condo

While submitting the jobs on private condos, it is crucial to use the correct set of PBS directive such as partition names and features. For example: a UTIA private condo with an associated account (ACF-UTK-0124) is created by reserving one Skylake node which falls under the skylake partition and feature.

#PBS -l partition=skylake #PBS -l feature=skylake

Defining any other partition such as beacon or sigma along with UTIA private condo PBS directives will results in the failed submission of the job. The other alternative is leave the partition names and features to be off and use the default ones.

Below we have created few example PBS jobs scripts by considering different possible scenarios of requesting ISAAC resources using the project account associated to a private condo.

Example 1: Job script to land the job on the UTIA private condo node using the default partition name and features.

 #PBS -S /bin/bash
 #PBS -A ACF-UTK0124
 #PBS -M abc@vols.utk.edu

 #PBS -l nodes=1:ppn=16
 #PBS -l walltime=24:00:00 <- Can be up to 28 days.
 cd $PBS_O_WORKDIR
 module load test/example

 #Command to run the application in serial or parallel.
 ./a.out

By default, QoS attribute is condo for UTIA private condo node and partition name is skylake, therefore, job will land on the UTIA private condo even if we do not mention these PBS directives in the job script. Note that Job will be subject to the condo node limits (1 node) and wallclock (28 days).

Example 2:  Job script to land the job on the UTIA private condo node or the institutional node, whichever can be scheduled first.

 #PBS -S /bin/bash
 #PBS -A ACF-UTK0124
 #PBS -M nduan1@vols.utk.edu
 #PBS -l qos=overflow <- this overrides the default qos=condo
 #PBS -l partition=skylake <- you can leave this out still if the job  ends up with qos=condo or qos=overflow (skylake will be implied)
 #PBS -l nodes=1:ppn=16
 #PBS -l walltime=24:00:00
 cd $PBS_O_WORKDIR
 module load test/example

 #Command to run the application in serial or parallel.
 ./a.out

In this example, instead of using default QoS ( which is condo), we used QoS attribute to be overflow. This will overwrite the default QoS directive for UTIA condo and the job will be subject to condo node limits depending upon which partition it lands on. It will be only one node if the job lands on the UTIA private condo with a maximum wallclock of 28 days and greater than one if the job lands on the institutional condo with a maximum wallclock of 24 hours.

Example 3: Job script to land the job only on the institutional beacon nodes with project account belonging to the private condo.

 #PBS -S /bin/bash
 #PBS -A ACF-UTK0124
 #PBS -M abc@vols.utk.edu
 #PBS -l qos=campus 
 #PBS -l partition=beacon 
 #PBS -l nodes=1:ppn=16
 #PBS -l walltime=24:00:00 
 cd $PBS_O_WORKDIR
 module load test/example

 #Command to run the application in serial or parallel.
 ./a.out

Note that in the above script we are requesting the scheduler to use project account for the private condo, but are interested to use the beacon node. We can do that by adding the PBS directive ‘#PBS -l partition=beacon ‘ to the job script. To force the scheduler to use beacon partition, PBS directive ‘#PBS -l qos=campus’ needs to be added. This will override the default value of QoS attribute which is condo.

Example 4: Jobscript to land the job only on the institutional beacon nodes using the qos=long-utk.

 #PBS -S /bin/bash
 #PBS -A ACF-UTK0124
 #PBS -M abc@vols.utk.edu
 #PBS -l qos=long-utk 
 #PBS -l partition=beacon 
 #PBS -l nodes=1:ppn=16
 #PBS -l walltime=72:00:00 
 cd $PBS_O_WORKDIR
 module load test/example

 #Command to run the application in serial or parallel.
 ./a.out

As mentioned in example 2, the maximum wallclock time for the institutional condo (qos=campus) is 24 hours. However, there are few beacon nodes which are dedicated to run the jobs for 3 days. The qos corresponding to these nodes is long-utk. Use the above script to submit the job for more than 24 hours under the institutional condo. Note that there are only beacon nodes dedicated to run the jobs under the qos=long-utk. Therefore, partition name should be beacon, otherwise the Torque scheduler will not allow the submission of the job.