Image
In this success story, EuroCC Denmark supported the European Spallation Source (ESS) in running large-scale neutron simulations on the EuroHPC LUMI-C cluster to prepare its NMX instrument for protein crystallography, well before the first beam is delivered. The project expanded simulation capacity from 10⁷ neutron rays on local resources to up to 10¹² on LUMI-C, enabling high-fidelity dataset generation that accelerates the facility’s scientific readiness.
Organisations involved
European Spallation Source ERIC (ESS), ESS is a next-generation, multi-disciplinary neutron research facility being constructed in Lund, Sweden, while the data Management & Scientific Computing centre are located north of Copenhagen, Denmark. Once operational, ESS will become the world’s most powerful pulsed neutron source, enabling groundbreaking research in structural biology, materials science, chemistry, energy, and engineering. ESS is designed around a long-pulse spallation source, providing uniquely high brightness and flux for neutron scattering experiments.
Technical/scientific Challenge
Neutron protein crystallography (NPX) is a rapidly expanding field used to determine hydrogen positions in biological macromolecules — information that is critical for understanding enzyme mechanisms, drug binding, and protein function at the atomic level. The NMX instrument at ESS is designed to achieve unprecedented neutron flux, flexible detector geometry, and the ability to study proteins with large unit-cell parameters (≥ 300 Å), opening the door to investigations that are currently beyond the reach of existing neutron sources.
To fully characterise NMX before beam operations begin, ESS needed to generate large-scale simulated diffraction datasets. These were produced using McStas, a Monte Carlo neutron ray-tracing simulation suite (version 3.5.x, written in C with MPI support) developed by Danish neutron scientists and maintained by the ESS Data Management and Scientific Computing (DMSC) division.
The computational demands of these simulations were immense. Modelling the large number of Bragg reflections (10⁵–10⁶) expected from protein crystals required extremely large neutron populations, between 10⁹ and 10¹² simulated rays per run. Each production simulation generated between 100 and 250 GB of detailed event-mode output in NeXus format, with peak memory usage reaching approximately 2.5 TB across nodes. Instrument geometry, choppers, detectors, and sample environment parameters needed to be updated frequently, meaning simulations had to be re-run many times during the design optimisation process.
Running these simulations on local infrastructure was not feasible. The scale of neutron populations, the data volumes involved, and the need for MPI scaling across hundreds to thousands of CPU cores made access to large-scale HPC resources essential.
Solution
EuroCC Denmark facilitated access to the EuroHPC LUMI-C cluster, where ESS was allocated 4,000 CPU node-hours with strong MPI support. The project followed a structured six-month development plan divided into three phases.
In the first phase, McStas workflows were ported to LUMI-C and MPI scaling was benchmarked. The team validated that McStas binaries could scale to 1,000 – 2,000 threads without performance degradation, and established that the parallel filesystem could handle the high write speeds required for event-mode data output.
During the second phase, ESS ran full NMX diffraction simulations with neutron populations of 10⁹ to 10¹⁰ rays. These large-scale runs generated synthetic Bragg diffraction datasets for protein crystals, producing the high-fidelity data needed to characterise NMX detector response, sensitivity, and instrument geometry.
In the final phase, the focus shifted to optimising MPI performance and file I/O strategy, enabling scaling to approximately 12,000 concurrent processes. Outputs were written in HDF5/NeXus format for downstream analysis, and instrument parameter comparisons were conducted to refine the NMX design.
Throughout the project, EuroCC Denmark provided guidance on navigating the EuroHPC application process and technical support for adapting the McStas workflows to the LUMI-C environment. Storage requirements were met with 1 TB of scratch/project storage for simulation outputs and intermediate files.
Business impact
By leveraging HPC, ESS can assess and optimise NMX performance early, reducing commissioning time and enabling staff training with realistic datasets. Design decisions on detector geometry, sample environments, and shielding are now informed by high-fidelity simulations. This accelerates scientific readiness, increases output in the facility’s early years, and strengthens global neutron research.
More broadly, the project contributed to the global neutron research community by pushing the performance boundaries of McStas, a widely used open-source simulation tool. The scaling improvements achieved on LUMI-C benefit not only ESS but also other neutron facilities worldwide that depend on McStas for instrument design and optimisation.
Benefits
- Scale: Simulations expanded from 10⁷ rays (local) to 10¹⁰–10¹² rays (LUMI-C)
- Accuracy: Ability to test detector geometry and protein crystal unit cell sizes up to 300 Å
- Performance: MPI scaling up to ~12,000 threads across many nodes
- Impact: Supports global neutron research through improved McStas performance
Figure: Simulated Laue diffraction of rubredoxin protein crystal (ca. 125,000 reflections) using the NMX instrument parameters.