• Your workload is taking a long time to run (>6hrs)

• It uses all your machine's resources (compute cores or memory)

• You need lots of memory

• You need GPUs

• You are either using or producing a lot of data

• You think you could cut your job into smaller chunks and process them at the same time
• You know you want to analyse larger datasets in the future.

Connecting to research0

1. Start PuTTY
2. Select 'research0' from the 'Saved Sessions'
3. Click 'Open'
4. The first time you connect you will get a security alert showing the fingerprint of the server, labeled as 'ssh-rsa' or 'ssh-ed25519'. Check the fingerprint below for the label shown and click "Yes" to proceed if it matches.

ssh-rsa 2048 5c:43:e5:e6:57:e0:4d:9f:f8:b5:ca:52:2f:30:39:ef research0
ssh-rsa 2048 bb:1f:6e:58:fa:d7:23:0d:ae:b2:b2:e8:62:a0:e7:5c research1
ssh-rsa 2048 13:42:77:57:ad:33:67:12:a4:8f:d3:26:24:37:2c:e0 research2
ssh-rsa 2048 99:da:16:61:09:e0:19:1f:53:0e:2b:e9:2a:22:50:99 research3
ssh-rsa 2048 ac:7e:1e:2a:05:d8:a1:3b:cf:b5:77:48:d0:bb:8a:22 teaching0
ssh-rsa 2048 bb:de:33:ff:07:23:6b:0a:73:ad:1f:8a:57:b2:c7:77 teaching1

ssh-ed25519 5a:c9:c1:76:16:00:42:45:f9:e5:bd:63:5d:87:db:8a research0
ssh-ed25519 f4:51:59:b7:b5:74:1c:14:a7:2c:78:1c:11:1f:72:cc research1
ssh-ed25519 76:93:b3:2d:e0:73:cb:25:a5:9e:c7:bd:ce:76:8f:a2 research2
ssh-ed25519 64:b2:f0:11:93:fc:d7:ea:12:bb:90:bc:7d:06:75:cc research3
ssh-ed25519 f4:ff:10:59:1e:5f:21:10:14:59:6e:04:16:63:7d:95 teaching0
ssh-ed25519 bc:73:f8:4f:40:7a:6b:41:73:90:f6:77:2b:4f:aa:d5 teaching1

Section

Graphical login using x2go Note The previous methods using NX Client and eXceed are now deprecated. Section border true Column width 50% x2go client is available on all IT Services Managed Windows systems. It is pre-installed on Classroom PCs; users on Office PCs can install it from Run Advertised Programs / Software Center. It appears under "Internet Tools" on the Start Menu and there are pre-defined menu entries for research0 and teaching0. Column width 50% Section border true Column width 50% On unmanaged PCs you can download the installer from the x2go Website. You will need to configure the settings as follows for research0 (or replace research0 with the name of the server you want to use): If the "New session" panel does not appear, select the menu item "Session | New session..." Enter "research0" in the 'Session name' field at the top Enter "research0.york.ac.uk" to the 'Server: Host:' field Change the 'Session Type' to XFCE Click "OK" (all other settings can stay on default). Column width 50% Section border true Column width 50% If you wish to change any settings: Cancel any login dialogs Click the pull-down on the corner of the panel and select "Session preferences..." Column width 50% Section border true Access from a Mac Go to the Finder on your Mac, find Applications, open the Utilities folder in Applications and then start the Terminal app from the Utilities folder. (You may wish to add Terminal to your Dock.) Then type the following, using your university username (abc123) instead ofusername. You don't need to type the $; this is an example of a prompt, which tells us the terminal is ready for us to type something. You should see something similar when you open Terminal, though it may be a bit longer (it may show your username for example). Just type anything from after the $. Panel title Logging into research0 $ ssh username@research0.york.ac.uk For a graphical login, install and configure the x2go client software as described in the unmanaged Windows section above. teaching0 and the other servers can be accessed in the same manner. Access from a UNIX server or desktop To login from a terminal window, type the following from your local machine with your university username (abc123): Panel title Logging into research0 $ ssh username@research0.york.ac.uk  If you require X forwarding, type: Panel title Logging research0 with X forwarding $ ssh -X username@research0.york.ac.uk You will be prompted for your IT Services password. Accessing teaching0 is done in the same manner.

******************************************************************************
***                   THE UNIVERSITY OF YORK IT SERVICES                   ***
***                                                                        ***
***                       THIS IS A PRIVATE COMPUTER                       ***
***                UNAUTHORISED ACCESS STRICTLY PROHIBITED                 ***
******************************************************************************

login: user001
password: 
Last login: Mon Sep  8 14:12:44 2014 from gallifrey.york.ac.uk
-bash-4.1$

When you see the prompt type the following command (you can also copy and paste the command into your terminal).

echo I love York

What happens?  The terminal should write out "I love York".  echo is a small program that takes a series of arguments and repeats them to the user.

Now try typing the following commands.

ls
ls -l

What do you see? Something like this?

-bash-4.1$ ls
bin  Chemistry  Desktop  examples  Experiments  intel  jobs  logs  tmp
-bash-4.1$ ls -l
total 296
drwxr-xr-x 2 abs4 csrv        4096 Jun 24 09:39 bin
drwxr-xr-x 3 abs4 csrv        4096 Jun  6 09:23 Chemistry
drwxr-sr-x 2 abs4 elecclust   4096 Mar 11 10:53 Desktop
drwxr-xr-x 3 abs4 csrv        4096 Jun 30 12:21 examples
drwxr-xr-x 5 abs4 csrv        4096 May 23 11:34 Experiments
drwxr-xr-x 3 abs4 csrv        4096 Aug 14 12:26 intel
drwxr-sr-x 3 abs4 elecclust   4096 Aug 15 12:49 jobs
drwxr-xr-x 2 abs4 csrv      266240 Aug 15 13:48 logs
drwxr-xr-x 3 abs4 csrv        4096 Aug 14 14:50 tmp
-bash-4.1$ 

The ls command lists all of the files in your current directory. A directory is equivalent to a folder.  The ls -l command gives you a long list, showing more information about each file or folder such as who is the owner of the file, who can access it, when it was last accessed. You can use most Linux programs in a number of ways by adding extra options.  Here for ls -l we added the -l option. If you need to know more about a program you can use either of the following:

ls --help
man ls

ls can also take arguments as input. For example, to see the contents of a directory in your current directory, pass the name of the directory to ls as an argument. You can combine options and arguments, like the second command below:

ls Desktop
ls -l Desktop

Filesystem organisation

The file system is the component of the operating system that organises data into files. These files are organised into directories (just like directories in Windows Explorer or the Mac Finder).

When you have logged in you will be placed in a directory which is called your home directory. To find the name of the directory use the program pwd (print working directory). 

Now try running the following commands. Here we have shown the prompt (-bash-4.1$) followed by the command to type on one line, then the output of the command on the following line (e.g after the -bash-4.1$, the first command is pwd, and the output is /home/userfs/e/ejb573; do not type the prompt as well, only the command). Your output will be slightly different as it will display the path to your home directory. 

-bash-4.1$ pwd
/home/userfs/e/ejb573
-bash-4.1$ cd /home
-bash-4.1$ pwd
/home
-bash-4.1$ cd
-bash-4.1$ pwd
/home/userfs/e/ejb573
-bash-4.1$ cd ..
-bash-4.1$ pwd
/home/userfs/e
-bash-4.1$ cd 
-bash-4.1$ pwd
/home/userfs/e/ejb573
-bash-4.1$ 

The output of the pwd command, /home/userfs/e/ejb573, is called a pathname, and this specifies the location of user ejb573's home directory. The first '/' in the pathname is called the root directory, the top-level directory in the hierarchy. Names following the '/' are directory names. Directories within directories are called sub-directories. Path names can also specify the location of files. The last part of a pathname (after the last /) is typically the name of a file or directory.

The cd program lets you change your working directory to another directory in the file system. cd with no arguments places you back in your home directory. The special argument '..' means the directory above your current directory (known as the parent directory).

Creating, moving and copying files and directories

You can create new files with the touch program and new directories with mkdir. You can move or copy files or directories to other locations with the mv (move) and cp (copy) programs.

First, let's create a new file and directory.

-bash-4.1$ touch afile
-bash-4.1$ touch bfile
-bash-4.1$ mkdir new-dir
-bash-4.1$ ls
afile bfile  Chemistry  examples     intel  logs     tmp
bin    Desktop    Experiments  jobs   new-dir

 You should now see the files afile and bfile and the directory new-dir.  Let's experiment with afile and new-dir:

-bash-4.1$ ls
afile  bin        Desktop   Experiments  jobs  new-dir
bfile  Chemistry  examples  intel        logs  tmp
-bash-4.1$ mv afile new-dir
-bash-4.1$ cp bfile new-dir
-bash-4.1$ ls
bfile  Chemistry  examples     intel  logs     tmp
bin    Desktop    Experiments  jobs   new-dir
-bash-4.1$ ls new-dir
afile  bfile
-bash-4.1$ mv new-dir/afile .
-bash-4.1$ ls
afile  bin        Desktop   Experiments  jobs  new-dir
bfile  Chemistry  examples  intel        logs  tmp
-bash-4.1$ 

This example creates a new directory, 'new-dir', We then move the file 'afile' to it and create a copy of 'bfile'. We then move the file 'afile' back to our current working directory. The '.' argument in "mv new-dir/afile ." means the current working directory, so this command moves 'afile' to your working directory.

Copying a directory is a little more complicated as the directory may contain files and directories. We use cp's '-r' option (recursive) to do this.

-bash-4.1$ ls
afile  bin        Desktop   Experiments  jobs  tmp
bfile  Chemistry  examples  intel        logs
-bash-4.1$ ls tmp
icc-start  ifort-start  logs       mpi-stop  start  test
icc-stop   ifort-stop   mpi-start  new-dir   stop
-bash-4.1$ cp tmp/test .
cp: omitting directory `tmp/test'
-bash-4.1$ cp -r tmp/test .
-bash-4.1$ ls 
afile  bin        Desktop   Experiments  jobs  test
bfile  Chemistry  examples  intel        logs  tmp
-bash-4.1$ ls test
test.c  test.cpp  test.f  test.f90  test.x
-bash-4.1$ 

In this example we wish to copy the contents of the directory 'tmp/test' into the current directory. cp will not copy a directory; we have to use the '-r' (recursive) option to tell cp to copy all files and directories within the directory.

Deleting files and directories

You can use the rm (remove) program is used to delete files. Please be careful with these commands, because Linux has no undo!

rm filename

To remove a directory and all of its contents use the rm -r (recursive) option to the rm command. To be safe and check the files before you remove them use -ri (recursive and interactive) options.

rm -ri dirname

File contents can be displayed with several different Linux programs. cat (concatenate files) will print your file to the screen, but this is not a good idea when the files are large. Instead, we can use less to view the file one screen at a time. Try them both on an existing file on the research/teaching servers (here we are using a file we already have called snark2).

 -bash-4.1$ cat snark2

The Hunting of the Snark
By Lewis Carroll
Fit the First
            The Landing

"Just the place for a Snark!" the Bellman cried,
   As he landed his crew with care;
Supporting each man on the top of the tide
   By a finger entwined in his hair.

"Just the place for a Snark! I have said it twice:
   That alone should encourage the crew.
Just the place for a Snark! I have said it thrice:
   What I tell you three times is true."

-bash-4.1$ less snark
The Hunting of the Snark
By Lewis Carroll
            Fit the First
            The Landing
"Just the place for a Snark!" the Bellman cried,
   As he landed his crew with care;
Supporting each man on the top of the tide
   By a finger entwined in his hair.
"Just the place for a Snark! I have said it twice:
   That alone should encourage the crew.
Just the place for a Snark! I have said it thrice:
   What I tell you three times is true."
The crew was complete: it included a Boots—
   A maker of Bonnets and Hoods—
A Barrister, brought to arrange their disputes—
   And a Broker, to value their goods.
A Billiard-marker, whose skill was immense,

To move through the file with less, or to quit back to the command line, use the following keys:

• SPACE - display the next screen of text
• q - quit displaying the file
• b - skip backwards through he file
• /pattern - search for text in the file

There are many text editors available on Linux. The easiest to use is probably nano:

nano filename

To edit text within nano, just type; to move around the file, use the arrow keys; to exit nano, type Control+X (^X is shown in the shortcuts at the bottom of the screen); to save the file, type Control+O (^O) and hit Enter when prompted for a filename. You might explore the other shortcuts shown at the bottom to try out some other features.

You might also try vi or emacs, which are harder to learn and use but very powerful once you get used to them. If you are using X forwarding (see above), you might also try gedit, which is a graphical text editor.

There are different exercises on copying files dependent on what operating system you use on your local machine.

-bash-4.1$ touch cfile
-bash-4.1$ touch dfile
-bash-4.1$ mkdir new-dir2
-bash-4.1$ mv dfile new-dir2
-bash-4.1$ ls
cfile  Chemistry  examples     intel  logs     tmp
bin    Desktop    Experiments  jobs   new-dir2

-bash-4.1$ #For an individual file
-bash-4.1$ scp cfile abc123@research0.york.ac.uk:~/ 

-bash-4.1$ #For a folder with lots of files
-bash-4.1$ scp -r new-dir2 abc123@research0.york.ac.uk:~/

-bash-4.1$ ls
cfile new-dir2

-bash-4.1$ #For an individual file
-bash-4.1$ scp dfile abc123@research0.york.ac.uk:~/new-dir2/

-bash-4.1$ #For an individual file
-bash-4.1$ scp abc123@research0.york.ac.uk:~/cfile . 

-bash-4.1$ #For a folder with lots of files
-bash-4.1$ scp -r abc123@research0.york.ac.uk:~/new-dir2 .
-bash-4.1$ ls

man scp 

rsync -avz new-dir2 abc123@research0.york.ac.uk:~/

man rsync 

Running applications on the command line.

You will need to login to the research machines with X forwarding enabled.  If you followed the steps in Exercise 1 you will have already set this up.  From linux or MacOS ssh with the -X flag

ssh -X abc123@research0.york.ac.uk:~/

Once logged in (either with putty or ssh) type matlab on the command line and wait for the gui to pop up. 

matlab

There are a number of programs installed this way. If you have ever used linux on the classroom PCs all the programs available there are also available here.

Working with modules

Modules allow us to support many permutations of application versions, built with different compiler versions and technologies. We can thus support new application versions, or optimised builds, without disruption to users who still require earlier editions.  There are many programs installed for use on these machines.  To see what programs are available type 'module avail' in the terminal. Do you see a list of available software?

ejb573@research0:~$ module avail
--------------------- /usr/lib/Modules/3.2.10/modulefiles ----------------------
dot         module-git  module-info modules     null        use.own
----------------------- /usr/lib/Modules/uoymodulefiles ------------------------
its-hdd      its-net      Ubuntu/16.04 Ubuntu/18.04
--------------- /opt/york/phys/pkg/Modules/modulefiles/compilers ---------------
clang/3.8.0(default)  gcc/5.4.0(default)    gcc/7.4.0
clang/4.0.0           gcc/7                 intel/17.0.0(default)
gcc/4.8.3             gcc/7.1.0             miniconda/3.7
gcc/4.9.0             gcc/7.3.0             Tcl/8.6.1
------------------ /opt/york/phys/pkg/Modules/modulefiles/mpi ------------------
mpich/3.2      mpich2/1.5     openmpi/1.10.7 openmpi/2.1.1
--------------- /opt/york/phys/pkg/Modules/modulefiles/libraries ---------------
cube/4.3.5           otf2/2.0             slepc/3.6.3
CUDA/10.1.160        petsc/3.4.5          slepc/3.7.4
CUDA/7.5.18          petsc/3.5.4          slepc/3.9.1
CUDA/8.0.61.p02      petsc/3.6.4          slepc-complex/3.11.2
fftw/3.3.6(default)  petsc/3.7.6          slepc-complex/3.5.4
fftw/3.3.8           petsc/3.8.4          slepc-complex/3.7.4
ghost/1.0            petsc/3.9.0          spglib/1.10.4
hdf5/1.8.16          petsc-complex/3.11.4 spglib/1.9.10
hdf5/1.8.18          petsc-complex/3.5.4  sundials/2.7.0
lapack/3.7.0         petsc-complex/3.7.7  szip/2.1.1
lz4/1.9.1            scalasca/2.3.1       Tk/8.6.1
MESASDK/22Aug2018    scorep/3.1           uda/1.0.0
netcdf/4.4.1.1       slepc/3.4.4          zlib/1.2.11
opari2/2.0.2         slepc/3.5.4

If you need to search for an application run module avail application-name on command line.  How many versions of python are there available on the research/teaching machines?

ejb573@research0:~$ module avail python
------------------------- /opt/yarcc/Modules/compilers -------------------------
python/2.7.11   python/3.4.0    python/3.6.0    python/3.6.5-bm
python/2.7.14   python/3.5.1    python/3.6.4
python/2.7.6    python/3.5.3    python/3.6.5

Now let us load a module.   On the command line run the following command

ejb573@research0:~$ python --version
Python 2.7.17

The default, system version Python version is 2.7.17.  We actually want a newer version of python to run our code.  Let us load a new version of Python with the module system

ejb573@research0:~$ module load python/3.6.5
ejb573@research0:~$ python --version
Python 3.6.5

We have now loaded a new version of python.  To check this you ran python --version.  You can also use module add python/3.6.5 to add a module.

To remove the module run

ejb573@research0:~$ module unload python/3.6.5
ejb573@research0:~$ python --version
Python 2.7.17

A cluster consists of many (hundreds or thousands) rack mounted computers called nodes. It would be similar to having hundreds of desktop computers sitting in the same room and able to talk to each other.  Clusters are often accessed via login nodes, which can send jobs to the other nodes in the cluster. Your commands will not be run immediately, but will be sent to a queue, and run when there is space on the cluster to run them.

The Viking cluster is Linux based and can be accessed in a similar manner to the research servers, but instead of accessing, say, research0.york.ac.uk, you would access viking.york.ac.uk. We have extensive wiki pages on how to get started on Viking. Please see /wiki/spaces/RCS/pages/39159378 for further details and if you ever need help email itsupport@york.ac.uk where one the research computing team can assist you. 

Viking is a multidisciplinary facility, supporting a broad spectrum of research needs, free at the point of use to all University of York researchers. Viking is as much a facility for learning and exploring possibilities as it is a facility for running well-established high-performance computing workloads. In this light, we encourage users from all Faculties, backgrounds and levels of ability to consider Viking when thinking about how computing might support their research.

To access Viking you need a user account, and your user account needs to be associated with a project code. Project codes are typically given to research groups; ask your PI for their code, and if they don't have one, ask them to fill in the Project Application Form. Once you have a project code, you can apply for a user account by filling in the User Application Form. It can take 24 hrs for new accounts to be processed. The Viking Wiki has more help on How to access Viking.