...
Directives to the batch scheduler allow the jobs to request various resources. Directives are preceded by #$, so for instance to specify the current working directory add #$ -cwd to the script.
Option | Description | default |
-l h_rt=hh:mm:ss | The wall clock time (amount of real time needed by the job). This parameter must be specified, failure to include this parameter will result in an error message | Required |
-l h_vmem=memory | Sets the limit of virtual memory required (for parallel jobs per processor). If this is not given it is assumed to be 2GB/process. If you require more memory than 1GB/process you must specify this flag. e.g. #$ -l h_vmem=12G will request 12GB memory. | 2G |
-l h_stack=memory | Sets the stacksize to memory. | unlimited |
-help | Prints a list of options |
-l nodes=x[,ppn=y][,tpp=z] | Specifies a job for parallel programs using MPI. Assigns whole compute nodes. x is the number of nodes, y is the number of processes per node, z is the number of threads per process. |
-l np=x[,ppn=y][,tpp=z] | Specifies a job for parallel programs using MPI. Assigns whole compute nodes. x is the number of processes, y is the number of processes per node, z is the number of threads per process. |
-pe ib np | Specifies a job for parallel programs using MPI, np is the number of cores to be used by the parallel job. |
-pe smp np | Specifies a job for parallel programs using !OpenMP or threads. np is the number of cores to be used by the parallel job. |
-hold_jid prevjob | Hold the job until the previous job (prevjob) has completed - useful for chaining runs together, resuming runs from a restart file. |
-l placement=type | Choose optimal for launching a process topology which provides fully non-blocking communication, minimising latency and maximising bandwidth. Choose good for launching a process topology which provides 2:1 blocking communications. Choose scatter for running processes anywhere on the system without topology considerations. | good |
-t start-stop | Produce an array of sub-tasks (loop) from start to stop, giving the $SGE_TASK_ID variable to identify the individual subtasks. |
-cwd / -l nocwd | Execute the job from the current working directory; output files are sent to the directory form which the job was submitted. If -l nocwd is used, they are sent to the user's home directory. | -cwd |
-m be | Send mail at the beginning and at the end of the job to the owner. |
-M email_address@<domain> | Specify mail address for -m option. The default <username>@york.ac.uk will automatically redirect to your email address at you parent institution, which was used for your registration on the facility. | <username>@york.ac.uk |
-V | Export all current environment variables to all spawned processes. Necessary for current module environment to be transferred to SGE shell. | Recommended |
-P project | Account usage to particular project. If user belongs to multiple projects, specifying the project name is compulsory. Can be omitted for users with single project membership | Default project for members of single project. Not specified for users with multiple projects |
| -o directory | place output from job (stdout) in directory. Directory must exist | current working directory |
| -e directory | Place error output from job (stderr) in directory Directory must exist | current working directory |
Environment Variables
In addition to those environment variables specified to be exported to the job via the -v or the -V option (see above) qsub, qsh, and qlogin add the following variables with the indicated values to the variable list (for a complete list see: man qsub)
| Variable |
|---|
| SGE_O_HOME | the home directory of the submitting client. |
| SGE_O_PATH | the executable search path of the submitting client. |
| SGE_O_WORKDIR | the absolute path of the current working directory of the submitting client. |
SGE_STDERR_PATH SGE_STDOUT_PATH | the pathname of the file to which the standard error/standard output stream of the job is diverted |
| SGE_TASK_ID | the index number of the current array job task (see -t option above). This is an unique number in each array job and can be used to reference different input data records, for example. |
SGE_TASK_FIRST SGE_TASK_LAST SGE_TASK_STEPSIZE | the index number of the first/last/increment array job task |
| ENVIRONMENT | the variable is set to BATCH to identify that the job is being executed under SGE control |
JOB_ID | a unique identifier assigned by SGE |
| JOB_NAME | The jobname set by -N. |
| NSLOTS | the number of slots (normally the same as the number of cores) for a parallel job. |
| RESTARTED | this variable is set to 1 if a job was restarted either after a system crash or an interruption. |
General queue settings
There are several default settings for the batch queue system:
...
Shared memory jobs should be submitted using the -pe smp <cores> flag. The number maximum number of cores that can be requested for shared memory jobs is limited by the number of cores available in a single node, in the case of YARCC physnodes this is 16 cores. Note that if you are running OpenMP job, the OMP_NUM_THREADS environment variable is automatically set to the requested number of cores by the batch system. To run a 16 process shared memory job, for 1 hour.
...
Please note that currently the majority of YARCC of physnodes is comprised of 16, 20, and 24 core nodes with between 64GB and 256GB of memory.
...