Processor Choice for Extended MODFLOW 6

[geo-admin-joe]

I'm trying to choose a processor type for an HPC cluster for calculations with Extended MODFLOW 6.

What I'm told by the groundwater scientists (I'm "just" an admin / computer scientist):

• The model has thousands of parameters and several millions of active model cells where the groundwater flow equation has to be solved.
• The main workload will be the calibration of the model.
• The calibration comprises thousands of individual runs of the model, trying out different combinations of parameter values. One run needs only one processor core but up to 50 GB of RAM.

Since the application is memory-intensive, I'm only considering AMD CPUs, because current AMDs have 12 memory channels vs current Intel's 8 memory channels.

I'm now looking at the F and X variants of AMD Epyc processors. The F variant has a higher frequency, which could speed up calculations, but the X variant has a larger cache, which could speed up memory access. The X variant is only available for 4th-generation processors, while the F variant is available in the 5th generation. The 5th generation has faster nominal memory access (DDR5-6400 vs. DDR5-4800).

What is more advisable: Going for higher frequency plus faster memory access or going for larger cache?

[mjr-deltares]

Hi @geo-admin-joe ,

I am by no means an expert on these architectures, but to my knowledge, multiple studies with (extended) MODFLOW on the Dutch national supercomputer (first Cartesius=>Intel and later Snellius=>AMD) have confirmed that the application is very much memory-bound. In order to avoid saturating the memory bandwidth, it makes sense to minimize the number of cores per channel and maximize the bandwidth of those channels. I don't know if AMD can provide this, but there is something called the HPCG benchmark https://www.hpcg-benchmark.org/ that is fairly representative of what happens inside MODFLOW.

[pya]

@mjr-deltares agreed. But you might get away without installing and configuring MPI. Are there significant performance differences between the serial and parallel version for purely serial runs?

[mjr-deltares]

I would expect the differences to be small but you're right, not having to install PETSc and MPI could be an advantage indeed.

[geo-admin-joe]

@pya Thanks for the hint. In our project, we also have phases when the calicbrated model will be used for simulations. Then we'll need parallel computation with Extended MODLFOW / MPI anyway. Still, the main computational load with will be in the calibration phases with the individual model runs, so it might be worth investing some time to examine if these phases can be shortened by using "regular" MODFLOW.

[geo-admin-joe]

@mjr-deltares Thanks for the info! I found this study: https://gmd.copernicus.org/articles/17/275/2024/
Do you have any others in mind? If yes, would it be possible to provide links to them?

[mjr-deltares]

Hi @geo-admin-joe , that's the main one I had in mind. It says:

"For example, the linear conjugate gradient solver that is used for solving the groundwater flow equation dominates runtime and is strongly memory-bound because of the required sparse matrix–vector multiplications (Gropp et al., 1999). This means that, starting from a certain number of cores, the competition for memory bandwidth hampers a parallel performance, and contention is likely to occur (Tudor et al., 2011). Typically, this directly relates to the available memory channels within a multicore CPU, linking the RAM (random access memory) and processor cores."

Another one that is also representative, but based on a proprietary version of MODFLOW 2005 is https://doi.org/10.1016/j.envsoft.2021.105092. It states that:

"On Cartesius, we use a maximum of four cores per node (hence two
cores per CPU) based on trial-and-error testing that has indicated that
run-times are shortest using this number of cores. Using a maximum of
four cores per node results in 20 idle cores during computation and a
relatively low core utilization of 17%. The reason why using more cores
in our trial-and-error analyis results in higher run times can likely be
explained by large memory requirements for our model applications and
the competition of processor cores within a multi-core CPU for the main
memory (Tudor et al., 2011)."

[p-ortega]

@geo-admin-joe just for reference, there is a similar discussion but for smaller Clusters in Pyemu's repo pypest/pyemu#568