Red Hat teams up with US Department of Energy Laboratories

Open source giant Red Hat has announced a collaboration with multiple US Department of Energy (DOE) laboratories to improve cloud-native standards and practices in high-performance computing (HPC).

Those labs will include the Lawrence Berkeley National Laboratory, Lawrence Livermore National Laboratory, as well as the Sandia National Laboratories.

As adoption of HPC expands beyond traditional use cases, the aim is to bridge the gap between HPC and cloud environments and lay the foundations for the arrival of exascale supercomputers.

The collaboration will bring together Red Hat’s expertise in cloud-native innovation with the laboratories’ knowledge of massive-scale HPC deployments. By establishing a common foundation of technology best practices, the pair will look to utilize standardized container platforms to link HPC and cloud computing footprints.

Ultimately, that will help to fill potential gaps in building cloud-friendly HPC applications, while also creating common usage patterns for industry, enterprise and HPC deployments, Red Hat said.

“The HPC community has served as the proving ground for compute-intensive applications, embracing containers early on to help deal with a new set of scientific challenges and problems,” explained Chris Wright, senior vice president and chief technology officer at Red Hat.

“That led to the lack of standardization across various HPC sites creating barriers to building and deploying containerized applications that can effectively span large-scale HPC, commercial and cloud environments, while also taking advantage of emerging hardware accelerators.

“Through our collaboration with leading laboratories, we are working to remove these barriers, opening the door to liberating next-generation HPC workloads.”

Red Hat says it will do this by focusing on advancing four key areas that address current gaps: standardization, scale, cloud-native application development, and container storage.

Among the planned collaborative projects, Red Hat and the National Energy Research Scientific Computing Center (NERSC) will work together on bringing standard container technologies to HPC. The pair will work on enhancements to Linux-based daemonless container engine Podman, to enable it to replace NERSC’s custom development runtime, Shifter.

Red Hat is also expanding its collaboration with Sandia National Laboratories to explore deployment scenarios of Kubernetes-based infrastructure at extreme scale, as well as teaming up with the Lawrence Livermore National Laboratory to bring HPC job schedulers, such as Flux, to Kubernetes through a standardized programmatic interface.

Additionally, Red Hat and the three DOE labs said they are aiming to reimagine storage for containers and develop a set of standard interfaces to manage various container image formats and provide access to distributed file systems.

As a result, HPC sites will be able to abstract the “immense complexities their environments can present” and benefit the range of U.S. exascale machines being deployed by the DOE, Red Hat said.

“High-performance computing infrastructure must adapt to the requirements of today's heterogeneous workloads, including workloads that use containers,” commented Earl Joseph, PhD, chief executive officer at Hyperion Research.

“Red Hat’s partnership with the DOE labs is designed to allow the new generation of HPC applications to run in containers at exascale while utilizing distributed file system storage, providing a strong example of collaboration between industry and research leaders."