Basic User's Guide

What is the supercomputer?

The computing cluster (supercomputer) of HSE University consist of computing nodes connected by a high-speed communication network. Each computing node has its own RAM and SSD and also it is connected to shared network storage. Special-parallel software is used to make independent computing nodes to work as a single computer.
Work with a computing cluster consists of three stages:

The user prepares on his own computer the source data for the available application software of the supercomputer or writes his own parallel program.
The user connects to the head node of the cluster via the SSH protocol, downloads the source data, and puts his task in the queue for calculation.
The user downloads the result from the computing cluster to his own computer and studies it locally.

Attention! The supercomputer is not a separate server. When connecting to cluster.hpc.hse.ru, users are connected to the head node, on which users cannot run calculations. It only acts as an interface for queuing tasks. The direct execution of calculations takes place on computing nodes.

Remote access to the supercomputer

Remote connection to the supercomputer is carried out through the head node using the SSH protocol at cluster.hpc.hse.ru, port 2222.
In GNU/Linux and MacOS, the ssh application is used for connection; for Microsoft Windows, the freeware PuTTY program is recommended.
Logging in begins with the login: system prompt: In response to it, enter the user name and press the [Return/Enter] key. Then the system will ask for the user password. Enter the password and press the [Return/Enter] key. Please note that when entering the password, characters on the screen are not printed, however, the password itself is entered.
To change users password after logging in, users should use the passwd command and enter the new password twice in response to the invitation. Entered characters will not be displayed.

Transferring files to the supercomputer

To transfer files to the computing cluster, use programs that support the SFTP protocol.

WinSCP is recommended for Microsoft Windows, scp for Linux, scp / Cyberduck / Mountain Duck programs for macOS.

File storage

In accordance with the rules, the user independently provides a backup of his data. It’s not allowed to store the data that is no longer involved in calculations on the supercomputer.
User directories look like /home/{username}. User directories are located on the Luster parallel data storage. The maximum amount of data for a user is determined by the disk quota determined during registration (after that the quota can be increased if necessary). In addition, a local directory for temporary files (/tmp) is available on each computing node and is located on the node’s SSD. After the program finishes, temporary user files in the /tmp node are deleted automatically.

Setting environment

To manage versions of application software packages and libraries, the Environmental Modules software package is installed on the supercomputer. It allows usets to flexibly configure environment variables for using certain software versions and tracking their dependencies (including starting in batch mode using sbatch). Also, the use of Environment Modules allows the user to flexibly manage different versions of the application (for example, the user can quickly switch between different versions of compilers and versions of the MPI library).

Using Environment Modules

Basic Environment Modules commands

module avail – list available modules
module list – list loaded modules
module load module1 – load module module1
module unload module1 – unload module module1
module swap module1 module2 – replace the loaded module module1 with module2
module purge – unload all loaded modules
module whatis module1 – show module1 information
module save [env_name] - save the current set of loaded modules under the name env_name. If you do not specify a name, the set will be overwritten by the default set
module resotre [env_name] - load a set of saved modules
module describe [env_name] - show the composition of the set of saved modules

Environmental Modules case studies

View loaded modules:

$ module list
Currently Loaded Modules:
1) autotools 2) prun/1.2 3) gnu7/7.3.0 4) openmpi3/3.1.3 5) ohpc

View available modules:

$ module avail
- /opt/ohpc/pub/modulefiles -
CUDA/10.0 cnpy_lib/1.0 hpcx/hpcx-stack
Python/Anaconda_v10.2019 gnu7/7.3.0 (L) hpcx/hpcx (D)
EasyBuild/3.7.1 fltk/v1.3.5 hwloc/1.11.10
Where:
D: module, loaded by default;
L: currently loaded module.

Unloading all modules and loading CUDA/10.2 and Python/Anaconda_v10.2019:

$ module purge
$ module add CUDA/10.2 Python/Anaconda_v10.2019
$ module list
Currently Loaded Modules:
1) CUDA/10.2 2) Python/Anaconda_v10.2019

Saving the default module set:

$ module save
Saved current collection of modules to: "default"

Compiling programs on the supercomputer

To compile programs on the HSE University computing cluster, the following are installed:

Proprietary Intel Parallel Studio XE Cluster Edition for GNU/Linux.
A set of free GNU Compiler Collection compilers.

The use of the Intel compiler and libraries is preferable, because it provides a significant acceleration of the execution of programs on Intel central processors of the supercomputer.

Instructions for compiling programs on the supercomputer

The compiler is selected by loading the appropriate module:
module load gnu8 openmpi3
or
module load INTEL/parallel_studio_xe_2020_ce

To compile parallel MPI applications using Intel Parallel Studio compilers, use the mpiicc, mpiicpc, mpiifort commands:

C: mpiicc [options] -o program.out file1.c file2.c
C++: mpiicpc [options] -o program.out file1.cpp file2.cpp
Fortran: mpiifort [options] -o program.out file1.f90 file2.f90

Detailed information on using Intel Parallel Studio compilers is available on the official Intel website.
To compile parallel MPI applications using the GNU Compiler Collection + OpenMPI, use the mpicc, mpixx, mpifort commands:

C: mpicc [options] -o program.out file1.c file2.c
C++: mpicxx [options] -o program.out file1.cpp file2.cpp
Fortran: mpifort [options] -o program.out file1.f90 file2.f90

Detailed information on using the GNU Compiler Collection and OpenMPI compilers is available on the official website.
When compiling, it is recommended to specify special optimization flags to speed up program execution. The -O2 optimization flag activates vectorization and basic loop optimization. In most cases, this flag ensures the best performance of the program.
When using programs with a large number of floating-point calculations or processing bulk data sets, it is recommended to use the -O3 flag, which activates a more aggressive conversion of loops and conditional expressions. More information on optimization flags can be found on the Intel Parallel Studio website and the GNU Compiler Collection.
When compiling using Intel compilers, it is also recommended to specify the -xHOST flag. It allows the compiler to use the maximum set of instructions available for processors on the supercomputer. This flag optimizes the application even more but reduces its portability to other platforms (all the nodes of the HSE University computing cluster have processors of the same generation with the same set of instructions).
An example of using optimization flags when compiling MPI programs:
mpiicpc -xHOST -O2 /opt/ophc/pub/examples/mpi/hello.c -o ./program_name
mpicxx -O2 /opt/ophc/pub/examples/mpi/hello.c -o ./program_name
To compile regular (non-parallel) programs using Intel Parallel Studio, use the following commands:

C: icc [options] -o program.out file1.c file2.c
C++: icpc [options] -o program.out file1.cpp file2.cpp
Fortran: ifort [options] -o program.out file1.f90 file2.f90

To compile regular (non-parallel) programs using the GNU Compiler Collection, use the following commands:

C: gcc [options] -o program.out file1.c file2.c
C++: g++ [options] -o program.out file1.cpp file2.cpp
Fortran: gfortran [options] -o program.out file1.f90 file2.f90

Queuing a task

To run calculations on the supercomputer, users should use Slurm job management system. This system (task scheduler) manages available resources and places tasks on suitable free computing nodes. In accordance with the rules, it is forbidden to perform tasks that require significant computing resources on the head node.
We recommend using the sbatch command to queue a task. The user should prepare a script with the necessary keys and command to use sbatch:

#!/bin/bash
#SBATCH <sbatch key> <value>
#SBATCH <sbatch key> <value>
<user commands>

Another way to start a task in a queue is with the srun command, which executes the task interactively. For example: srun [options] my_mpi_task
Detailed information on launching tasks is available in a separate instruction.

View Queue Status

Users can view the current status of the task queue using the squeue command.

To view the status of only user tasks, use the short mj command.

Removing a task from the queue

To remove a task from the queue, use the scancel [task number] command.

Basic GNU/Linux OS commands

List of basic commands in GNU/Linux:

pwd – print the full path of the current directory
cd [path] - change the current directory
ls – show directory contents
cp <source> <recipient> - copy file
mv <source> <recipient> - move the file
rm <path> - delete the file
cat <path> - display the contents of the file in the terminal
mcedit [path] - open a text file editor
exit – exit the shell (or disconnect from the supercomputer)

Help on any command on GNU/Linux can be obtained using the command man <command_name> or <command_name> --help.

About maintenance work

Scheduled maintenance work on the supercomputer is carried out monthly on the last business day of the month (Mon-Fri) from 9:30 to 18:00. For the duration of maintenance work, the task queue is suspended, and access for users is closed. Active tasks are removed. Tasks that support restarting (see the Restart tasks section in the startup instructions) are restarted after completion of work.

Scheduled maintenance work is carried out to provide users with stable cluster operation for a whole month.

In case of problems

To resolve questions arising on the use of the HSE University computing cluster, contact the Supercomputer modeling department by creating a ticket on the support portal or by e-mail at hpc@hse.ru. The ticket should describe in detail the problem that has arisen and the steps for reproducing it.

Have you spotted a typo?
Highlight it, click Ctrl+Enter and send us a message. Thank you for your help!
To be used only for spelling or punctuation mistakes.

Supercomputer Modeling Unit