# Severe inefficiencies in parallel computing

**URL:** <https://discourse.abinit.org/t/severe-inefficiencies-in-parallel-computing/3535>\
**Category:** Ground state\
**Tags:** parallelism\
**Created:** [June 27, 2024, 3:09am UTC](https://discourse.abinit.org/t/severe-inefficiencies-in-parallel-computing/3535 "2024-06-27T03:09:00Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![iavas](https://discourse.abinit.org/letter_avatar_proxy/v4/letter/i/6de8d8/32.png) [@iavas](https://discourse.abinit.org/u/iavas)\
**Post date:** [June 27, 2024, 3:09am UTC](https://discourse.abinit.org/t/severe-inefficiencies-in-parallel-computing/3535/1 "2024-06-27T03:09:00Z")

</div>

Hello all,

I’m currently attempting to do some parallel computing. In [this previous post](https://discourse.abinit.org/t/atomic-forces-not-decreasing-for-structural-optimization/3529), I found the computation to be suspiciously slow. So I tried the #2 case in [this tutorial](https://docs.abinit.org/tutorial/paral_bandpw/) as a test.

According to the information in the tutorial, the calculation of the second input file, which contains 5 SCF steps, should be completed in about a minute using 64 processes in parallel. However, when I did it in practice, it took me more than 2 minutes to complete a single SCF step. Something must be very wrong, but I have no clue how to locate the problem. The log, input, and output files are below (I’ve manually killed this job after SCF step 3, but time information is still in the log file):  
[tparal\_bandpw\_02.log](https://discourse.abinit.org/uploads/short-url/jJ8kltQmVwaW6wM1A3r0gZsejF9.log) (78.6 KB)  
[tparal\_bandpw\_02.abi](https://discourse.abinit.org/uploads/short-url/tolSP2D7SCmKSYfAPiZ7ovkbLnK.abi) (10.2 KB)  
[tparal\_bandpw\_02.abo](https://discourse.abinit.org/uploads/short-url/tW0RIvAaRUDi4yGryo6sTZiWSK9.abo) (42.2 KB)

How do I fix this problem? Or is there any other information I can provide to help pinpoint the problem? Thank you very much.

---

<div class="post-metadata">

**Author:** ![beuken](https://discourse.abinit.org/user_avatar/discourse.abinit.org/beuken/32/93_2.png) [@beuken](https://discourse.abinit.org/u/beuken)\
**Post date:** [June 27, 2024, 11:08am UTC](https://discourse.abinit.org/t/severe-inefficiencies-in-parallel-computing/3535/2 "2024-06-27T11:08:55Z")

</div>

Hi,  
just in case…

> export OPENBLAS\_NUM\_THREADS=1  
> export GOTO\_NUM\_THREADS=1  
> export OMP\_NUM\_THREADS=1  
> export MKL\_NUM\_THREADS=1

---

<div class="post-metadata">

**Author:** ![iavas](https://discourse.abinit.org/letter_avatar_proxy/v4/letter/i/6de8d8/32.png) [@iavas](https://discourse.abinit.org/u/iavas)\
**Post date:** [June 27, 2024, 2:07pm UTC](https://discourse.abinit.org/t/severe-inefficiencies-in-parallel-computing/3535/3 "2024-06-27T14:07:21Z")

</div>

I think this might be where the problem is, I tried to use a single node and the job finished pretty fast. If one “node” in my system has less than 64 CPUs and I have to use two nodes for this, but my ABINIT executable was compiled without OpenMP support, will it cause a problem like this?

---

<div class="post-metadata">

**Author:** ![beuken](https://discourse.abinit.org/user_avatar/discourse.abinit.org/beuken/32/93_2.png) [@beuken](https://discourse.abinit.org/u/beuken)\
**Post date:** [June 27, 2024, 5:03pm UTC](https://discourse.abinit.org/t/severe-inefficiencies-in-parallel-computing/3535/4 "2024-06-27T17:03:24Z")

</div>

Regarding the netlib library, did you compile from source or install a netlib package?  
LINALG libraries (OpenBlas, MKL, etc.) are generally compiled with OMP.

---

<div class="post-metadata">

**Author:** ![iavas](https://discourse.abinit.org/letter_avatar_proxy/v4/letter/i/6de8d8/32.png) [@iavas](https://discourse.abinit.org/u/iavas)\
**Post date:** [June 30, 2024, 4:25pm UTC](https://discourse.abinit.org/t/severe-inefficiencies-in-parallel-computing/3535/5 "2024-06-30T16:25:40Z")

</div>

Hi,  
The cluster admin compiled this software for me, and then compiled a supposed-to-be OMP version after I asked for help. I used it to try to repeat the [Hybrid Parallelism](https://docs.abinit.org/tutorial/paral_bandpw/#hybrid-parallelism-mpiopenmp) part of the tutorial, testing three sets of parameters:  
[tparal\_bandpw\_04\_mpi48\_thread1.abi](https://discourse.abinit.org/uploads/short-url/vaqTYCtknuTN5gJTeL4RToJlD2t.abi) (10.2 KB)  
[tparal\_bandpw\_04\_mpi64\_thread1.abi](https://discourse.abinit.org/uploads/short-url/jUWZTIdHIBkjR48prcbkpj3vkVr.abi) (10.2 KB)  
[tparal\_bandpw\_04\_mpi32\_thread2.abi](https://discourse.abinit.org/uploads/short-url/sYEueCNy1TExewh4sG8n1KhoO5L.abi) (10.2 KB)  
[tparal\_bandpw\_04\_mpi48\_thread1.abo](https://discourse.abinit.org/uploads/short-url/tPJKbEKupbuTeBXPjltfL2UFP9n.abo) (192.3 KB)  
[tparal\_bandpw\_04\_mpi64\_thread1.abo](https://discourse.abinit.org/uploads/short-url/chTRNbnbuPyvFShVwGZ2PY9aw3N.abo) (193.3 KB)  
[tparal\_bandpw\_04\_mpi32\_thread2.abo](https://discourse.abinit.org/uploads/short-url/pYFBDAaVyCazWc9DpKztEjWVhpW.abo) (192.8 KB)

Our cluster has only 56 CPUs per node. The result I found is that only when using one node (OMP\_NUM\_THREADS=1) and total cores 48 \< 56, the 48 core 1 thread case, the Proc time is about 200 seconds. For the 64c1t and 32c2t cases, their Proc time is much longer.  
And in all three cases, the overall time is still way longer than the result in that tutorial.