Resources for Power Systems HPC users from IBM at no charge
In order to support the growing community of users and developers building and running their HPC codes on IBM Power Systems or equivalent OpenPOWER servers, IBM provides a number of their software development tools at no charge. These are ‘Community Edition’ versions of the tools, which are generally fully functioning products, although some have limitations: for instance the maximum number of servers it can be installed on. In all cases, there is a user agreement and a license for the product, and the software comes with no official support from IBM. There are also fully licensed versions of all of these products, that come complete with a support contract with IBM entitling you to access the extensive support capabilities and experiences of the IBM Technical Support Services team.
There are also similar products available from third party vendors or the open source communities — but I won’t go into all of the options here.
Compilers
Although a lot of scientific computing applications these days are becoming more data-centric, and therefore being written in interpreted languages like Python and R, there is still a lot of code out there that requires compiling. To this end, there are a number of options for C/C++ and Fortran compilers on Power Systems hardware.
OS supplied GNU compilers
The base Operating system image comes with the standard GNU compilers and libraries, although these will be at the level that was current when that OS release was made available, which can be several steps behind the open source versions. These are installed from the standard OS repositories using tools like yum and apt-get.
IBM Advance Toolchain for Linux on Power
The Advance Toolchain provides up to date versions of the GNU compilers and libraries, available to download and install from IBM. These are fully tested by IBM, and provide support for the latest hardware advantages and compiler functionality. As well as the gcc, gcc++ and gfortran compilers, this package includes optimised runtime libraries for POWER8 and POWER9 processors. Other libraries are included to help get the best performance out of the POWER architecture. along with several supporting libraries. There are also tools for debugging (eg gdb) and performance analysis (eg valgrind and itrace).
For more information: https://developer.ibm.com/linuxonpower/advance-toolchain/
IBM XL Compilers for Linux — community editions
The IBM XL compiler family are used to generate code that makes the best use of the latest POWER9 architecture, allowing you to maximise the performance of your code and the utilisation of your hardware. With full support for offloading computation and data to accelerators like NVIDIA GPUs using CUDA, and also OpenMP, the XL compilers can be used to build complex applications. There are two Community Edition products available, the XL C/C++ for Linux compiler and the XL Fortran for Linux compiler. To help you get the absolute best performance out of your code, both compilers support multiple levels of optimisations, with extensive optimisation reports and debug capabilities — including profile directed feedback and interprocedural analysis.
To download the Community editions of these tools:
XL C/C++: https://www.ibm.com/uk-en/marketplace/xl-cpp-linux-compiler-power
XL Fortran: https://www.ibm.com/uk-en/marketplace/xl-fortran-linux-compiler-power
For more information on installation, setup, and usage:
XL C/C++: https://www.ibm.com/support/knowledgecenter/en/SSXVZZ_16.1.1/com.ibm.compilers.linux.doc/welcome.html
XL Fortran: https://www.ibm.com/support/knowledgecenter/en/SSAT4T_16.1.1/com.ibm.compilers.linux.doc/welcome.html
NVIDIA GPUs and CUDA
For applications that make use of GPU acceleration, you will need a compiler that allows you to make use of the GPUs in a system. The standard compiler for NVIDIA devices is NVCC, which is part of the freely available CUDA runtime from NVIDIA. With code written to make use of the CUDA runtime, compilers like the XL family will generally offload the GPU computation to the NVCC compiler to build the final binaries. NVIDIA also provide a set of tools for debugging and analysing the performance of CUDA code.
Software development tool kits
Although many modern Linux distributions include a wide range of tools to assist with software development and testing, if you are building more complex applications it can often be helpful to have a full development environment. A well built Software Development Kit (SDK) will provide an integrated environment for building, compiling, debugging, and testing code.
IBM Software Development Kit for Linux on Power
The IBM SDK for Linux on Power brings together many of the other tools listed here, providing a full range of application development assistance. It also includes capabilities like the Migration Advisor to help port code from other architectures (like x86_64) to make the most effective use of the Power Systems hardware. The SDK also provides platform emulation capabilities and cross-compilers to allow for development on systems that don’t use the ppc64le architecture.
The IBM SDK for Linux on Power is available at no charge:
https://developer.ibm.com/linuxonpower/sdk/
Multi-node runtimes (MPI)
Most HPC applications require the ability to scale across multiple systems, and multiple NUMA nodes within servers. The common option for implementing this in HPC code is to use MPI. Again, there are options available to users for building MPI capable applications.
OS supplied MPI
Open source tools like OpenMPI are commonly available with Linux distributions, and can be installed using the standard package management tools. There are also other MPI implementations available through the common repositories, like the MPICH tools.
Spectrum MPI
IBM again provides a community edition of the optimised MPI tools for Power Systems hardware running Linux. The Spectrum MPI implementation is based on Open MPI, but with advanced features and optimisations for the hardware capabilities in the POWER processor architectures. It supports a range of networking options, including high speed Ethernet and InfiniBand interconnects.
To download the community edition of Spectrum MPI:
https://www.ibm.com/uk-en/marketplace/spectrum-mpi
For more information on installation, setup, and usage:
https://www.ibm.com/support/knowledgecenter/en/SSZTET_10.3/welcome.html
Mathematical acceleration libraries
Most scientific computing workloads make extensive usage of common mathematical subroutines, including linear algebra, matrix operations, transformations and many more. Again, there are a number of options available to support these common tasks, offloading them to external libraries. These libraries can then make use of hardware acceleration capabilities without the need to expressly code for it.
OS supplied libraries
The base operating system provides open source libraries that have been compiled for the hardware, and provided as packages in the standard repositories. Libraries like OpenBLAS and Lapack are commonly available, although the optimisation for the hardware depends on the Linux distribution installed.
Mathematical acceleration subsystem (MASS)
As part of the Advance Toolchain for Linux on Power (see compilers section above), IBM provides the Mathematical Acceleration Subsystem (MASS) libraries which contain mathematical functions for C, C++, and Fortran applications. These libraries are tuned to give the best performance on specific POWER architectures, including POWER8 and POWER9 based systems. These allow you to get higher performance for your code using the common GNU compilers or the XL compiler family.
Information on usage can be found here:
https://developer.ibm.com/linuxonpower/advance-toolchain/adv-tool-usage/
Engineering and Scientific Subroutine Library (ESSL)
Providing more advanced mathematical functions and subroutines, the IBM Engineering and Scientific Subroutine Library (ESSL) is available in a community edition at no charge. This includes heavily optimised subroutines for a range of mathematical functions running on POWER based hardware. This can be utilised by the GNU compilers, the XL family of compilers, and third party compilers. There are high performance equivalents of most BLAS and Lapack functions, as well as a lot more complex functions used in engineering and scientific applications.
To download the community edition of ESSL:
https://epwt-www.mybluemix.net/software/support/trial/cst/welcomepage.wss?siteId=589&tabId=1174&w=1
For more information on installation, setup, and usage:
https://www.ibm.com/support/knowledgecenter/en/SSFHY8_6.2/navigation/welcome.html
Job schedulers
The final useful tool for HPC sites is a job scheduler that allows users to submit batch jobs to a queue to be deployed on appropriate servers within a cluster of systems. Although there are free and open source schedulers and queue managers, there are often benefits to using commercial products — including improved performance, better management capabilities, or more flexible deployment options.
IBM Spectrum LSF
The IBM offering that covers job scheduling and batch queues is IBM Spectrum LSF. This is available as a standalone tool, or packaged as a Spectrum LSF Suite to include other valuable tools, like cluster provisioning and management tools, application management, and license management across all running jobs. Although primarily used for batch workloads, you can also deploy interactive jobs through the same queues and scheduler, and some of the latest additions to the Spectrum LSF feature set also allow container based workloads and even integration with Kubernetes.
Although there is a no-charge trial available for the Spectrum LSF Suite for HPC, there is also a Community Edition which can be run on clusters of up to 10 nodes, running up to 1000 jobs.
To download Spectrum LSF Suite Community Edition:
https://epwt-www.mybluemix.net/software/support/trial/cst/programwebsite.wss?siteId=680&h=null&p=null
For further information about Spectrum LSF and the Spectrum LSF Suites:
https://www.ibm.com/support/knowledgecenter/SSZUDP_10.2.0/suite_welcome/lsf_suite_kc_overview.html
