High Performance Computing

chadwick cluster Overview

The Liverpool chadwick high performance computing (HPC) cluster is available for demanding research computing tasks and is a collection of many individual computer nodes, each with multiple cores (computing elements which typically do one task each). Each of these may appear little or no faster than some high-end desktop systems but they have a higher than usual amount of memory each and various other hardware advantages over desktop PCs. In total, there are around 1800 cores available.

The processor nodes and cores are designed to work together in parallel, with fast communication between them, using a large shared disk storage area (i.e. all the nodes see essentially the same storage). This might well allow demanding calculations or data manipulations, if written to work in parallel on many cores at once, to run 10 or more times faster, and typically significantly more efficiently in various respects — including your time once you are used to the system. The management of the system is done for you and advice is available on how best to use it to your work.

Users normally access the system by logging in to a login node (sometimes called the head node) over the network. The login node is for light use, such as editing input files, compiling your own programs and, most importantly, submitting jobs to run the relevant tasks on one or more of the many compute nodes connected to it. Programs which run for a long time or take a lot of memory should not be run on the login node and there are some restrictions on what you can do on it to prevent other users being affected. More background information can be found on HPC cluster hardware webpage.

The cluster has management software to support shared use by many users simultaneously. This sort of batch environment may be unfamiliar, but is fairly straightforward. To do some work, it is necessary to compose a script (or often a command will do it for you) and type a command to submit the job to the batch queue. It will then run with whatever resources (such as number of cores) you've asked for. The job sits in a queue along with other users' jobs until the resources are available to run it (which might be immediately if the cluster is lightly loaded). It will then run unattended — perhaps while you're asleep — and eventually produce some output. You can either check on its progress or get email when it has finished.

The operating system used on the cluster is GNU/Linux (a type of UNIX) — not Microsoft Windows. It is typically (but not necessarily) used by typing at a "command line" in a "terminal emulator" window, not with a GUI environment (although GUIs are available). The main tasks you are likely to want to do are:

If you are completely new to UNIX/Linux then there is an absolute beginners tutorial introduction available on this site. A brief overview of Linux with links off to related topics can be found at Opensource.com. Another introduction to Linux and its basic commands can be found at Surrey EE Unix Tutorial (at the starting level, Linux, Mac Darwin and UNIX commands are identical)

A short, command summary can be found at Beginner Linux Tutorial. There are other starting topics on this web-site as well.

There is also a 25 minute video posted at Linux Tutorial Series - Part 01 that claims to provide an introduction to Linux.

There are variety of ways off logging in to the cluster which will depend on what system you are using (e.g. Windows PC, Mac laptop) and whether you are on campus or not. These are detailed instructions available on the Accessing ARC resources page. This also has information on how to transfer files to and from the chadwick cluster

What you need to do when ysing the cluster is usually fairly simple, even if it looks intimidating initially, and people quickly get used to it, especially if — as for most users — it's a question of running one of two existing programs for all your work. It may also be possible to automate or abstract much of this by using scripts to glue together sub-tasks.