My group uses Abinit on the San Diego supercomputing cluster, and we’ve been doing some longwave calculations that require heavy IO on wavefunction files, so we have been working with one of their staff members to compile Abinit 10.0.5 with parallel IO enabled. We’ve been having some trouble, and I was wondering if you had any input. Essentially, we are able to successfully compile Abinit with the enable-mpi-io flag and have a working Netcdf library (which itself has been compiled with parallelism enabled), but our job fails when we actually try to run a job with iomode=3. Here is a summary of our issue written up by the staffer I’ve been working with, who is able to summarize the issue better than I can:
I compiled abinit 10.0.5 for a user at SDSC (www.sdsc.edu) on our expanse cluster ( User Guide ) with --enable-mpi-io set.
you need a consistent set of hdf5 netcdf and netcdf-fortran libs with the same parallelization scheme and compilers/mpi versions/everything. In particular for fortran modules the compiler has to be identical to that used to compile abinit itself. I don’t see a module for hdf5, but it’s probably auto-loaded by the netcdf libs. Could you check this, and also provide the content of the (not commented with #) lines in the config.h file?
I think that the pnetcdf library is not actually used at all: it’s a separate project from the netcdf-c compiled with the mpicc compiler.
On my macOS machine, ldd abinit gives the following pertinent lines (installed with homebrew in this case, you should have some equivalent paths in your modules)
ldd src/98_main/abinit
/opt/homebrew/opt/netcdf-fortran-parallel/lib/libnetcdff.7.dylib (compatibility version 7.0.0, current version 7.1.0)
/opt/homebrew/opt/netcdf-parallel/lib/libnetcdf.19.dylib (compatibility version 19.0.0, current version 19.0.0)
/opt/homebrew/opt/hdf5-parallel/lib/libhdf5.310.dylib (compatibility version 314.0.0, current version 314.0.0)
So from the lib file names you can’t tell whether they are installed with mpi. The abinit configure script make a check on that with h5cc -showconfig, then grepping for parallel flags
General Information:
HDF5 Version: 1.10.7
Configured on: 2023-04-16
Configured by: Unix Makefiles
Host system: Linux-4.18.0-425.13.1.el8_7.x86_64
Uname information: Linux
Byte sex: little-endian
Installation point: /cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/hdf5-1.10.7-5o4oibcvimawkbouxpkomqvhikvsnnnm
Compiling Options:
------------------
Build Mode: RelWithDebInfo
Debugging Symbols: OFF
Asserts: OFF
Profiling: OFF
Optimization Level: OFF
Linking Options:
----------------
Libraries:
Statically Linked Executables: OFF
LDFLAGS:
H5_LDFLAGS:
AM_LDFLAGS:
Extra libraries: m;dl
Archiver: /usr/bin/ar
Ranlib: /usr/bin/ranlib
Languages:
----------
C: YES
C Compiler: /cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/openmpi-4.1.3-oq3qvsvt5mywjzy7xzrfeh6eebiujvbm/bin/mpicc 10.2.0
CPPFLAGS:
H5_CPPFLAGS:
AM_CPPFLAGS:
CFLAGS: -std=c99 -Wno-implicit-function-declaration -fstdarg-opt -fmessage-length=0
H5_CFLAGS: -Wall;-Wcast-qual;-Wconversion;-Wextra;-Wfloat-equal;-Wformat=2;-Winit-self;-Winvalid-pch;-Wmissing-include-dirs;-Wno-c++-compat;-Wno-format-nonliteral;-Wshadow;-Wundef;-Wwrite-strings;-pedantic
AM_CFLAGS:
Shared C Library: YES
Static C Library: YES
Fortran: ON
Fortran Compiler: /cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/openmpi-4.1.3-oq3qvsvt5mywjzy7xzrfeh6eebiujvbm/bin/mpif90 10.2.0
Fortran Flags:
H5 Fortran Flags: -Waliasing;-Wall;-Wcharacter-truncation;-Wextra;-Wimplicit-interface;-Wsurprising;-Wunderflow;-pedantic;-ffree-form;-fimplicit-none;-std=f2008;-Warray-temporaries;-Wintrinsics-std;-Wimplicit-procedure;-Wreal-q-constant;-Wfunction-elimination;-Wrealloc-lhs;-Wrealloc-lhs-all;-Wno-c-binding-type;-Wuse-without-only;-Winteger-division;-Wfrontend-loop-interchange
AM Fortran Flags:
Shared Fortran Library: YES
Static Fortran Library: YES
C++: ON
C++ Compiler: /cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/openmpi-4.1.3-oq3qvsvt5mywjzy7xzrfeh6eebiujvbm/bin/mpic++ 10.2.0
C++ Flags: -fstdarg-opt -fmessage-length=0
H5 C++ Flags: -Wall;-Wcast-qual;-Wconversion;-Wctor-dtor-privacy;-Weffc++;-Wextra;-Wfloat-equal;-Wformat=2;-Winit-self;-Winvalid-pch;-Wmissing-include-dirs;-Wno-format-nonliteral;-Wnon-virtual-dtor;-Wold-style-cast;-Woverloaded-virtual;-Wreorder;-Wshadow;-Wsign-promo;-Wundef;-Wwrite-strings;-pedantic
AM C++ Flags:
Shared C++ Library: YES
Static C++ Library: YES
JAVA: ON
JAVA Compiler: /cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/openjdk-11.0.12_7-27cv2psb6xthjbwmjlx53otdbjxch6wg/bin/javac
Features:
---------
Parallel HDF5: ON
Parallel Filtered Dataset Writes: ON
Large Parallel I/O: ON
High-level library: ON
Build HDF5 Tests: OFF
Build HDF5 Tools: ON
Threadsafety: OFF
Default API mapping: v110
With deprecated public symbols: ON
I/O filters (external): DEFLATE DECODE ENCODE
MPE:
Direct VFD:
Mirror VFD:
(Read-Only) S3 VFD:
(Read-Only) HDFS VFD:
dmalloc:
Packages w/ extra debug output:
API Tracing: OFF
Using memory checker: OFF
Memory allocation sanity checks: OFF
Function Stack Tracing: OFF
Use file locking: best-effort
Strict File Format Checks: OFF
Optimization Instrumentation: