Job examples


General rules & script format

  • Each line begining with #PBS is interpreted as qsub command line option (read man qsub for available options)
  • Other lines begining with # are comments ( e.g. ##PBS is a comment)
  • All other lines are executed in a user's home shell started by the queueing system (so there is no need to put #!/bin/sh line in the script - it would start unneccessarily two nested shells)


Note: all examples below assume bash as a user shell.

Serial job

A simple script with comments

## Job name
#PBS -N test
## Select queue
#PBS -q short
## Request 1h of real (wall clock) time
#PBS -l walltime=1:0:0
## Request 200MB of memory
#PBS -l mem=200mb
## email me on begin, abort, end of the job
#PBS -m bae
## files which store standard output and error streams from the job
#PBS -o stdout.txt
#PBS -e stderr.txt
cd $PBS_O_WORKDIR
# run code redirecting stdout and stderr to a file (assuming tcsh is your shell)
./my_code >& out.txt

 

Important: Please note that the standard out and err streams from the code are redirected to a file despite the specification of standard out and err for the job. This is very important unless stdout/stderr from your code is less than a few MB. The job output is spooled locally on the execution node and copied to the user working directory only after the job completes. Since the spool size is small (a few GB) you can overfill the disk and crash all the jobs on the node. With redirection approach you avoid this and in addition you can monitor out.txt during runtime.

Array of serial jobs

It is possible to start N copies of a job with one qsub command, e.g. qsub -t 3-20,25-100 will start 94 jobs with id's from 3 to 20 and from 25 to 100 . The id is available in the job as a shell variable PBS_ARRAYID. It can be exploited to start many similiar jobs like in the example below:

## Job name
#PBS -N test
## Select queue
#PBS -q short
## Request 1h of real (wall clock) time
#PBS -l walltime=1:0:0
## Request 200MB of memory
#PBS -l mem=200mb
## files which store standard output and error streams from the job
#PBS -o stdout.txt
#PBS -e stderr.txt
cd /my_working_directory/$PBS_ARRAYID
# perhaps create initial condifions depending on $PBS_ARRAYID
./mycode >& out-$PBS_ARRAYID.txt


Serial jobs with Quality of Service

Priviliged users can take advantage of the Quality of Service feature (see User guide) to increase priority of a job waiting in the queue. To submit a job under specific QOS type qsub -W x=QOS:qname job_script or add following line to the job_script : #PBS -W x=QOS:qname. qname is the name of QOS account.

Parallel MPI job

Parallel jobs must use queue ibpara.
Please do not use mpd daemon (mpirun) nor standard mpiexec nor mpirun_rsh. Old /opt/mpiexec/bin/mpiexec is obsolete too. Use /opt/hydra/bin/mpiexec.hydra command - it cooperates with torque so taks are spawned according to '-l nodes' specification (no -np nor -machinefile is needed).
 

#PBS -N testmpi
#PBS -q ibpara
## request 2 nodes with 8 cpus available for tasks
#PBS -l nodes=2:ppn=8
## Request 500MB of memory per task
#PBS -l pmem=500mb
#PBS -o stdout.txt
#PBS -e stderr.txt
## email me only on job end
#PBS -m e
module load mvapich2-x86_64
cd $PBS_O_WORKDIR
/opt/hydra/bin/mpiexec.hydra  ./my_code >&  out.txt


Parallel OpenMP job

This applies also to Mathematica jobs (however do not use more than 4 threads).

#PBS -N testomp
#PBS -q ibpara
## request a node with 4 free cpus
#PBS -l nodes=1:ppn=4
## Request 500MB of memory per task
#PBS -l pmem=500mb
#PBS -o stdout.txt
#PBS -e stderr.txt
#PBS -m e


export OMP_NUM_THREADS=`cat $PBS_NODEFILE | wc -l`
cd /work/psk/my_directory
./my_code >& out.txt

Interactive jobs

To start an interactive job on the node with gpu card

qsub -I -q interactive -l partition=oldpart -l nodes=1:gpus=1

Note: there is one node in partition oldpart with 2 gpus - it should be used for testing and debugging CUDA codes; if you omitt partition parameter, the job goes to the defpart - there are 4 nodes with faster gpus but they should be used for production runs