Job Arrays

Overview

Teaching: 15 min
Exercises: 0 min

Questions

• How do I run similar job in parallel?

Objectives

• Learn how to use job arrays to parallelize tasks.

The queue system can do a lot of work for you, without you having to rewrite the same script.

Job Arrays are similar to the arrays you worked with earlier when you were learning to write shell scripts.

The -t option of qsub

The -t option can be used to create a specified number of queue submissions that all perform nearly the same computational task.

Valid arguments for the -t option can be as follows:

 -t 1-10
 -t 1-100
 -t 1,2,3,5,8,13,21
 -t 1-10,23,42

Internally to your scripts, this creates an environmental variable $PBS_ARRAYID that is set to the values you specify to the -t option.

Let’s make and save a file - job_array_test.sh:

touch /home/ssander/jobarrayFile_$PBS_ARRAYID.txt

Now that we’ve made the file, let’s submit it to the queue and have it run 10 times:

qsub -l nodes=1:ppn=2 -l walltime=48:00:00 -q batch -t 1-10 job_array_test.sh

Exercise

Write a script that uses the $PBS_ARRAYID environmental variable to write the following string to a file that is also numbered by $PBS_ARRAYID:

“This is file” $PBS_ARRAYID

and run the script for the values 1-5 and 7,9,11,13.

Solution

job_array_excercise.sh: echo "This is file" $PBS_ARRAYID > filenumber_$PBS_ARRAYID.txt qsub -l nodes=1:ppn=2 -l walltime=48:00:00 -q batch -t 1-5,7,9,11,13 job_array_excercise.sh

Key Points