/
Physnodes - 4) Job script files

Physnodes - 4) Job script files

06 Nov, 2020

Job submission parameters and example script files

To submit a script use the qsub <scriptname> command

 

Submitting a job with qsub

-bash-4.1$ qsub serial_job
Your job 10217 ("serial_job") has been submitted
-bash-4.1$ qstat
job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
10217 0.00000 serial_job abs4 qw 06/02/2014 12:29:57 1

 

Job Script Directives

The default executing shell is the borne shell (/bin/sh). If you require a different shell (e.g. /bin/csh) then this can be specified in a #!/bin/csh directive at the top of the script.   

Directives to the batch scheduler allow the jobs to request various resources. Directives are preceded by #$, so for instance to specify the current working directory add #$ -cwd to the script.

Option

Description

default

-l h_rt=hh:mm:ss

The wall clock time (amount of real time needed by the job). This parameter must be specified, failure to include this parameter will result in an error message

Required

-l h_vmem=memory

Sets the limit of virtual memory required (for parallel jobs per processor). If this is not given it is assumed to be 2GB/process. If you require more memory than 1GB/process you must specify this flag. e.g. #$ -l h_vmem=12G will request 12GB memory.

2G

-l h_stack=memory

Sets the stacksize to memory.

unlimited

-help

Prints a list of options

 

-l nodes=x[,ppn=y][,tpp=z]

Specifies a job for parallel programs using MPI. Assigns whole compute nodes. x is the number of nodes, y is the number of processes per node, z is the number of threads per process.

 

-l np=x[,ppn=y][,tpp=z]

Specifies a job for parallel programs using MPI. Assigns whole compute nodes. x is the number of processes, y is the number of processes per node, z is the number of threads per process.

 

-pe ib np

Specifies a job for parallel programs using MPI, np is the number of cores to be used by the parallel job.

 

-pe smp np

Specifies a job for parallel programs using !OpenMP or threads. np is the number of cores to be used by the parallel job.

 

-hold_jid prevjob

Hold the job until the previous job (prevjob) has completed - useful for chaining runs together, resuming runs from a restart file.

 

-l placement=type

Choose optimal for launching a process topology which provides fully non-blocking communication, minimising latency and maximising bandwidth. Choose good for launching a process topology which provides 2:1 blocking communications. Choose scatter for running processes anywhere on the system without topology considerations.

good

-t start-stop

Produce an array of sub-tasks (loop) from start to stop, giving the $SGE_TASK_ID variable to identify the individual subtasks.

 

-cwd / -l nocwd

Execute the job from the current working directory; output files are sent to the directory form which the job was submitted. If -l nocwd is used, they are sent to the user's home directory.

-cwd

-m be

Send mail at the beginning and at the end of the job to the owner.

 

-M email_address@<domain>

Specify mail address for -m option. The default <username>@york.ac.uk will automatically redirect to your email address at you parent institution, which was used for your registration on the facility.

<username>@york.ac.uk

-V

Export all current environment variables to all spawned processes. Necessary for current module environment to be transferred to SGE shell.

Recommended

-P project

Account usage to particular project. If user belongs to multiple projects, specifying the project name is compulsory. Can be omitted for users with single project membership

Default project for members of single project. Not specified for users with multiple projects

-o directory

place output from job (stdout) in directory. Directory must exist

current working directory

-e directory

Place error output from job (stderr) in directory Directory must exist

current working directory

Environment Variables

In addition to those environment variables specified to be exported to the job via the -v or the -V option (see above)  qsub, qsh, and qlogin add the following variables with the indicated values to the variable list (for a complete list see: man qsub)

Variable

 

SGE_O_HOME

the home directory of the submitting client.

SGE_O_PATH

the executable search path of the submitting client.

SGE_O_WORKDIR

the absolute path of the current working directory of the submitting client.

SGE_STDERR_PATH

SGE_STDOUT_PATH

the pathname of the file to which the standard error/standard output stream of the job is diverted

SGE_TASK_ID

the index number of the current array job task (see -t option above). This is an unique number in each array job and can be used to reference different input data records, for example.

 

 

SGE_TASK_FIRST

SGE_TASK_LAST

SGE_TASK_STEPSIZE

the index number of the first/last/increment array job task

ENVIRONMENT

the variable is set to BATCH to identify that the job is being executed under SGE control

JOB_ID

a unique identifier assigned by SGE

JOB_NAME

The jobname set by -N.

NSLOTS

the number of slots (normally the same as the number of cores) for a parallel job.

RESTARTED

this variable is set to 1 if a job was restarted either after a system crash or an interruption.