Documentation

Users

This website can only be accessed by authenticated users. There are three types of users:

Viewers - these are users with read-only access to the system. They are able to view clusters, applications and jobs.
Normal Users - in addition to what Viewers can do, they are able to prepare and submit jobs to run on clusters.
Cluster Admins - these are the users with administration privileges. In addition to what Normal Users can do, they can set up credentials, create and manage clusters, and set up/install applications on clusters. Cluster Admins are normally the owners of cloud resources, e.g. owners of GCP projects, or somebody who are delegated with sufficient permission.

The preferred way to access this system is through Google identities. Admin users may also create user accounts.

Credentials

This section is for Cluster Admin users. Normal user should skip this section as the admin users should have prepared the credentials already.

This system helps you to automate many activities on your chosen cloud platform. For this to work, register your cloud credentials with this system. Credentials will be validated against the cloud platforms and saved for future uses.

Cloud credentials come in different forms depending on which cloud provider you are using. Typically they can be found from the web interface of your cloud provider or obtained from command line.

GCP

The preferred way accessing GCP resources from this system is through a Service Account. To create a service account: your account must have sufficient permissions. If you are not the Owner or Editor of the GCP project, follow the instructions below. When certain permissions are missing, GCP will give clear error messages. Note the permissions required, locate them in this page, and identify suitable roles that provide them. Ask the project Owner to assign those extra roles to your user account. Some relevant roles include Service Account User, Service Account Admin, and Service Account Key Admin.

From GCP console

Log in to your GCP console and select the GCP project to host this work.
From the main menu, select IAM & Admin, then Service Accounts.
Click the CREATE SERVICE ACCOUNT button.
Name your service account, optionally provide a description, and then click the CREATE button.
Grant your service account the following roles: Editor, Security Admin.
You may give some human users access to this service account but that is not required by this system. Clock Done button.
Locate the new service account from the list, click Manage Keys from Actions menu.
Click ADD KEY, then Create new key. Select JSON as key type, and click the CREATE button.
Save the generated JSON content which should be pasted in the credential creation form here on this website.

From command line

It is assumed that you have the gcloud command-line tool installed on your development system, or use GCP cloud shell which has this tool pre-installed.

Clusters

In the simplest setting, an organisation may create one HPC cluster per cloud platform. This is because one cluster can support multiple machine types. There are, of course, many good reasons to create multiple clusters on the same platform, e.g. for project management purpose, or to map cloud usages to organisational structures.

For each cluster, admin users can choose the suitable machine types for the organisation's workloads and impose resource limits, e.g. the maximum number of compute nodes for each machine types.

At any time, each cluster is in one of the following status:

New: Cluster is being newly configured by a user through the web interface.
Creating: Cluster is being created (i.e. hardware is being brought up online).
Initialising: Cluster is being initialised (i.e. software is being installed and environment is being prepared). By default, these clusters use a CentOS 7 based operating system with software preconfigured to support distributed MPI and hybrid jobs.
Ready: Cluster is ready for jobs.
Terminating: Cluster is being terminated.
Stopped: Cluster is stopped (can be restarted).
Deleted: Cluster has been deleted (cannot be restarted). Historical information of this cluster will be retained in the database.

Depending on the current cluster status, actions are made available on the website to perform management tasks, e.g. cluster can be created when it is in 'New' status or paused (stopped) when it is in 'Ready' status. Most management tasks are performed asynchronously in the background. For example, cluster creation may take more than 20 minutes to complete. Visual indication is available on the website when background tasks are ongoing.

When a cluster is ready, applications can be installed by Cluster Admin users.

Applications

Applications can be installed on HPC clusters using either Spack or custom means. Spack, an established package management system for HPC, contains build recipes of most widely used applications, including almost all popular open-source packages. For applications not yet covered by the Spack package repository, e.g. codes developed in-house, or commercial packages that require complex set-up, a script-based approach can be used register them to the system.

Note that an application in this system refers to a unique binary installation of a package, as identified by a software version, and a specific target architecture. Spack has intrinsic support on both of these factors, generating multiple binaries as required. Spack also builds variants of the same software, e.g. packages with optional features switched on. Experienced users should be able to specify such using Spack specifiers.

Spack applications

Each application is in one of the following status:

New: Application is being newly configured by a user through the web interface. At this stage, only a database record gets created in the system.
Preparing: Application build is triggered from the website and information is being passed on to the cluster.
Queueing: The job to build this application in queueing on the target cluster.
Installing: The job to build this application in running on the target cluster. Spack is fully responsible for building this application and managing its dependencies.
Ready: Spack build has completed successfully and the application is ready to run on the current cluster.
Failed: Spack has somehow failed to build this application. See Troubleshooting section below for debugging information.

A typical work flow for installing a new Spack application is as follows:

From the cluster list page, identify the target cluster, and click Install Spack Application from its Actions menu. Alternatively, click the Install Spack Application button from the cluster detail page.
In the Create a new application form, in the Spack name field, type a keyword and use the auto-completion function to choose the Spack package you want to install. The Name and Version fields get populated automatically. If Spack supports multiple versions of the same application, click the dropdown list to select the desired version.
Spack supports variants - applications built with customised compile-time options. These may be special compiler flags, or optional feature that must be switched on manually. Advanced users may supply additional specs using the optional Spack spec field.
The Description field is populated automatically from the information found in the Spack repository.
Choose a Install instance from the dropdown list. Some applications can benefit from parallel build. If this is known to be the case, an instance with higher number of vCPUs may speed up the installation process.
Select one or more Instance type from the list. This field specifies the target machines on which this application installation is valid and able to run. Spack generates the most suitable binaries depending on this information. For example, if the c2 family is chosen on GCP, Spack understands that the CPU architecture is Intel "cascadelake" and generates the optimised binary for it accordingly. If both c2 and n2d (AMD Rome architecture) are selected (Ctrl click for multiple selection), Spack will generate common "x86-64" binaries that can run on both Intel and AMD processors. Cluster Admins decides which machine types are available for each cluster. Users are responsible to choose the most suitable machine types for application builds, striking a balance between performance and compatibility.
Click Save button. At this stage, only a database record has been created in the system. On the next page, click the Edit button to modify the application settings; click the Delete button to delete this record. Click the Spack install button to actually start building this application on the cluster. This last step can take quite a while to complete depending on applications. A visual indication is given on the application detail page until the installation job gets completed.
Successfully installed application will have its status updated stating so. A New Job button becomes available from the Actions menu on the application list page, or from the application detail page.

Other applications

Jobs

Troubleshooting

SSH access

SSH access to cluster head nodes may be necessary for debugging purposes.

Currently, Cluster Admin users may SSH into clusters via the service machine (machine hosting this website) using a special user account called 'citc'. This account is automatically set up when a new cluster is being created.

Cluster Admins have the option to add authorized users when creating new clusters. If these ordinary users have their public SSH keys registered with the system, they can also gain access to the clusters.

Log files

/root/ansible-pull.log - log for cluster set-up
/mnt/shared/benchmarks/install_<application_id>_<machine_type> - logs for Spack application installations