Image
How SIDE is helping Germany's Federal Waterways Engineering and Research Institute prepare its shallow water solver for next-generation supercomputers.
Supercomputing in Deutschland (SIDE), part of the EuroCC project, strengthens high-performance computing (HPC) and related competencies in Germany. Formed of the High-Performance Computing Center Stuttgart (HLRS), Juelich Supercomputing Center (JSC), Leibnitz Supercomputing Center (LRZ), Sicos BW, and the Gauss Centre for Supercomputing (GCS), it supports industry, academia, and public institutions in utilizing these technologies, promotes knowledge transfer, and enhances European digital competitiveness.
Scientific partners involved
The Jülich Supercomputing Centre (JSC) at Forschungszentrum Jülich operates one of Europe's most powerful supercomputing infrastructures and hosts JUPITER, the first European exascale system. Through funded projects and strategic collaborations, JSC advances the development of modular exascale architectures and supports the scaling of scientific applications towards next-generation supercomputers. In this collaboration, JSC contributes its expertise in high-performance computing, performance profiling, and GPU-porting methodologies. The Federal Waterways Engineering and Research Institute (BAW - Bundesanstalt für Wasserbau) serves as the technical and scientific advisor to the German Federal Ministry of Transport and the Federal Waterways and Shipping Administration. As Germany's centre of excellence for waterway engineering, BAW develops and operates advanced numerical models to support safe and efficient navigation. Among these is UnTRIM2, a three-dimensional shallow-water model used for coastal and estuarine applications.
Technical / scientific challenge
Within a Proof-of-Concept (PoC) collaboration supported by SIDE, BAW partnered with JSC to explore the acceleration potential of their self-developed iterative Krylov-space Conjugate Gradient (CG) solver used in the UnTRIM2 model, which is used to simulate hydrodynamics and scalar transport in the German Bight and is computationally intensive. Originally designed for CPU-based execution on unstructured grids, the CG solver had never been evaluated for modern heterogeneous HPC architectures. The goal was to assess its GPU readiness and define a realistic, sustainable modernization strategy as a first step toward GPU porting. Additionally, BAW sought practical knowledge transfer to increase in-house experience with GPU technologies, leading to the organization of a dedicated two-day workshop focused on GPU porting concepts, hands-on training, and discussions of the feasibility and efficiency of OpenMP offloading for the UnTRIM2 code and other in-house codes.
Solution
To address the first GPU porting challenge, JSC implemented and evaluated a series of Conjugate Gradient (CG) solver variants on its HPC infrastructure, in particular on the JUWELS Booster system. Both the standard CG solver and its Block-Jacobi preconditioned version (PCG) were evaluated in multiple parallel paradigms: a serial baseline, a shared-memory OpenMP CPU version, and a GPU-accelerated implementation using OpenMP offloading.
This systematic approach enabled direct comparisons of performance and scalability across architectures and provided clear insights into how both the CG and the preconditioned solver could benefit from GPU acceleration. The scaling experiments showed typical results regarding the CPU and GPU performance of the CG solver. While the CPU is faster for smaller problem sizes, the GPU outperforms it for larger ones. A directive-based offloading strategy using OpenMP was selected as the most sustainable short-term solution, as it aligned with the existing hybrid MPI-OpenMP programming model already used in the UnTRIM2 codebase.
Performance analysis was carried out using NVIDIA’s profiling and tracing tool Nsight Systems and the measurement infrastructure Score-P to identify computational hotspots, memory-access behaviour, and scaling limitations on both CPUs and GPUs. Because both JUWELS Booster and BAW’s systems rely on NVIDIA A100 GPUs, the evaluation on JSC’s system yielded realistic, transferable performance results. The PoC culminated in a clear GPU-readiness assessment, a validated performance baseline, and a dedicated two-day hands-on workshop delivered by JSC to address the second challenge of providing knowledge transfer that will support BAW's future GPU-porting efforts.
Scientific impact
The SIDE supported PoC marked the first systematic effort to evaluate the GPU offloading potential of the linear Conjugate Gradient solver within the UnTRIM2 model on modern HPC systems. Through scaling measurements and profiling on both CPUs and GPUs, the collaboration provided clear insights into where computational time is spent and which algorithmic components constrain scalability.
These findings give BAW a solid, evidence-based foundation for future development decisions. Instead of pursuing ad hoc porting efforts, the institute can now prioritize the solver components with the greatest acceleration potential, enabling a more efficient, targeted modernization strategy.
Beyond the specific application, the collaboration illustrates how performance engineering and HPC expertise can significantly reduce the risks associated with updating legacy scientific software. It establishes a practical, reproducible pathway for transitioning complex, CPU-oriented simulation codes to heterogeneous HPC architectures.
This approach also demonstrates how expertise in GPU offloading, in particular, can be developed relatively quickly through knowledge transfer and support from a trained HPC centre.
Benefits
- Clear, evidence-based assessment of GPU-porting feasibility for the CG and PCG solver
- Identification of key challenges faced during porting from CPU to GPU
- Reduced technical risk for future modernization of the UnTRIM2 code
- Practical knowledge transfer through hands-on GPU-porting training using OpenMP
Runtime comparison for CPU vs. GPU
Figure 1 shows a runtime comparison of the CG solver (CPU vs. GPU). The double-logarithmic plot below compares OpenMP runs with 24 threads on one CPU and an OpenMP offloading run on one GPU, with varying problem size n. This comparison scenario was run on the JUWELS Booster at JSC on one node. It can be seen that as the problem size increases, the GPU performs better than a single 24-core CPU. This is expected behaviour for this kind of algorithm, as the GPU performs better when its occupancy is higher.
Figure 1: Runtime comparison between one CPU vs one GPU with increasing problem size n.