MD Step time in flare-otf is extremely long

[FromColleWithLove]

Hi, I write here regarding a problem I am having performing active learning on AuAl systems.

When performing active learning with flare-otf, the time of "MD Steps" is enormous, up to hours.

Input

supercell:
        file: 100_surf.pwi
        format: espresso-in
        replicate: [1,1,1]
        jitter: 0
flare_calc:           
        gp: SGP_Wrapper
        file: AuAl_offline/myotf_AuAl_flare.json
dft_calc:
        name: Espresso
        kwargs:
                command: "mpirun -np 1 pw.x < AuAl.pwi >AuAl.pwo"
                pseudopotentials:
                        Au: Au.pbesol-dn-rrkjus_psl.0.3.0.UPF
                        Al: Al.pbesol-n-rrkjus_psl.1.0.0.UPF
                label: AuAl
                tstress: True
                tprnfor: True
                nosym: True
                kpts: "automatic"
                kpts: [7,7,1]
                koffset: [1,1,0]
                input_data:
                        control:
                                disk_io: "low"
                                prefix: "AuAl-otf"
                                pseudo_dir: "potentials/"
                                outdir: ./out
                                calculation: scf
                                restart_mode: from_scratch
                        system:
                                ibrav: 0
                                ecutwfc: 55
                                ecutrho: 440
                                ntyp: 2
                                occupations: "smearing"
                                smearing: "mv"
                                degauss: 0.0147
                        electrons:
                                conv_thr: 3.3e-5 # 1e-5 eV / atom
                                electron_maxstep: 200
                                mixing_beta: 0.7
                                mixing_mode: 'plain'
        params: {}
otf:
        mode: fresh #(or restart)
        md_engine: PyLAMMPS
        md_kwargs:
                command: "mpirun -np 1 lmp_kokkos  -k on g 1 t 16 -sf kk -pk kokkos newton on neigh half"
                specorder: [Al,Au]
                verbose: True
                dump_period: 1
                pair_style: flare
                thermo: 10
                fix: ["1 all nvt temp 500.0 500.0 1.0"]
                #thermo_args: ["step temp ke pe etotal press c_MaxUnc"]
                keep_tmp_files: True
                temp: 500
                tmp_dir: ./tmp
        initial_velocity: 500        
        dt: 0.001 # timestep is 0.5 fs
        number_of_steps: 500000
        output_name: AuAl_otf
        init_atoms: [0,2,4,6,10,12,14,20,25,28,30,32,36,39,42] #the atoms whose atomic environments are used for the first step of learning
        max_atoms_added: -1 #number of atoms added to the set at each DFT call. If -1, allows for all the atoms to be added, if required by uncertainties
        std_tolerance_factor: -0.02 #if the epistemic uncertainty on a force component exceeds this value (in unit of the current noise hyperparameter), DFT is called
        train_hyps: [10000000,10000001]
        write_model: 2 # 0 write never, 1 at end, 2 at each training and end, 3 each time atoms are added and end, 4 write after each trainig and at end, and back up after each write
        update_style: threshold
        update_threshold: 0.001 #when DFT is called, with threshold=update_style, atoms with uncertainties on forces higher than this value will be added (see also max_atoms_added)  
        force_only: False #then, it will use forces,energies and stresses
        no_cpus: 16 
        store_dft_output: [["AuAl.pwo"],"./tmp"]

grep "Time of MD Step" AuAl_otf.out"

gives:

Time of MD Step: 4.48 s
Time of MD Step: 3.59 s
Time of MD Step: 6.84 s
Time of MD Step: 9.67 s
Time of MD Step: 9.48 s
Time of MD Step: 9.60 s
Time of MD Step: 8.81 s
Time of MD Step: 9.17 s
Time of MD Step: 10.73 s
Time of MD Step: 11.04 s
Time of MD Step: 10.21 s
Time of MD Step: 11.08 s
Time of MD Step: 11.83 s
Time of MD Step: 12.04 s
Time of MD Step: 13.75 s
Time of MD Step: 16.40 s
Time of MD Step: 14.45 s
Time of MD Step: 13.91 s
Time of MD Step: 13.00 s
Time of MD Step: 14.46 s
Time of MD Step: 17.28 s
Time of MD Step: 17.16 s
Time of MD Step: 17.74 s
Time of MD Step: 14.65 s
Time of MD Step: 14.15 s
Time of MD Step: 18.05 s
Time of MD Step: 14.90 s
Time of MD Step: 15.26 s
Time of MD Step: 20.99 s
Time of MD Step: 22.24 s
Time of MD Step: 24.75 s
Time of MD Step: 19.55 s
Time of MD Step: 20.74 s
Time of MD Step: 16.59 s
Time of MD Step: 20.12 s
Time of MD Step: 20.24 s
Time of MD Step: 30.18 s
Time of MD Step: 29.22 s
Time of MD Step: 26.32 s
Time of MD Step: 26.68 s
Time of MD Step: 30.99 s
Time of MD Step: 24.86 s
Time of MD Step: 30.07 s
Time of MD Step: 29.65 s
Time of MD Step: 38.05 s
Time of MD Step: 43.63 s
Time of MD Step: 26.15 s
Time of MD Step: 33.64 s
Time of MD Step: 44.57 s
Time of MD Step: 27.54 s
Time of MD Step: 36.27 s
Time of MD Step: 51.22 s
Time of MD Step: 41.92 s
Time of MD Step: 40.27 s
Time of MD Step: 59.43 s
Time of MD Step: 43.66 s
Time of MD Step: 30.73 s
Time of MD Step: 45.19 s
Time of MD Step: 139.00 s
Time of MD Step: 69.07 s
Time of MD Step: 81.58 s
Time of MD Step: 95.25 s
Time of MD Step: 64.66 s
Time of MD Step: 93.25 s
Time of MD Step: 74.02 s
Time of MD Step: 60.86 s
Time of MD Step: 203.48 s
Time of MD Step: 156.00 s
Time of MD Step: 115.43 s
Time of MD Step: 69.35 s
Time of MD Step: 217.28 s
Time of MD Step: 40.28 s
Time of MD Step: 98.65 s
Time of MD Step: 143.31 s
Time of MD Step: 3031.66 s
Time of MD Step: 1476.12 s
Time of MD Step: 2761.84 s
Time of MD Step: 2694.55 s
Time of MD Step: 69.29 s

In a similar case (changing only initial configuration):

Time of MD Step: 4.66 s
Time of MD Step: 3.53 s
Time of MD Step: 6.57 s
Time of MD Step: 7.93 s
Time of MD Step: 8.45 s
Time of MD Step: 8.39 s
Time of MD Step: 8.49 s
Time of MD Step: 10.20 s
Time of MD Step: 9.27 s
Time of MD Step: 10.29 s
Time of MD Step: 10.52 s
Time of MD Step: 11.55 s
Time of MD Step: 10.52 s
Time of MD Step: 12.28 s
Time of MD Step: 11.88 s
Time of MD Step: 13.59 s
Time of MD Step: 12.48 s
Time of MD Step: 14.68 s
Time of MD Step: 16.22 s
Time of MD Step: 15.37 s
Time of MD Step: 18.98 s
Time of MD Step: 24.18 s
Time of MD Step: 18.12 s
Time of MD Step: 15.61 s
Time of MD Step: 26.79 s
Time of MD Step: 21.40 s
Time of MD Step: 23.99 s
Time of MD Step: 27.15 s
Time of MD Step: 22.84 s
Time of MD Step: 30.64 s
Time of MD Step: 35.02 s
Time of MD Step: 31.28 s
Time of MD Step: 33.05 s
Time of MD Step: 24.94 s
Time of MD Step: 23.69 s
Time of MD Step: 42.26 s
Time of MD Step: 36.13 s
Time of MD Step: 29.15 s
Time of MD Step: 19.71 s
Time of MD Step: 29.74 s
Time of MD Step: 21.53 s
Time of MD Step: 40.60 s
Time of MD Step: 90.36 s
Time of MD Step: 56.07 s
Time of MD Step: 54.85 s
Time of MD Step: 55.29 s
Time of MD Step: 57.49 s
Time of MD Step: 99.46 s
Time of MD Step: 73.68 s
Time of MD Step: 59.18 s
Time of MD Step: 85.93 s
Time of MD Step: 78.47 s
Time of MD Step: 113.34 s
Time of MD Step: 63.32 s
Time of MD Step: 182.49 s
Time of MD Step: 266.48 s
Time of MD Step: 476.28 s
Time of MD Step: 495.32 s
Time of MD Step: 708.93 s
Time of MD Step: 345.00 s
Time of MD Step: 248.69 s
Time of MD Step: 248.67 s
Time of MD Step: 1241.97 s
Time of MD Step: 202.23 s
Time of MD Step: 2131.98 s
Time of MD Step: 142.52 s
Time of MD Step: 660.16 s
Time of MD Step: 798.47 s
Time of MD Step: 118.98 s
Time of MD Step: 253.41 s
Time of MD Step: 641.24 s
Time of MD Step: 315.33 s
Time of MD Step: 383.65 s
Time of MD Step: 247.10 s
Time of MD Step: 158.00 s
Time of MD Step: 760.65 s
Time of MD Step: 3116.53 s
Time of MD Step: 142.18 s
Time of MD Step: 6741.88 s

Version

[GCC 13.3.0]
numpy 1.24.4
ase 3.22.0
flare 1.4.2
LAMMPS (4 Feb 2025 - Development - patch_4Feb2025-440-g2240430d5e-modified)

Host machine

OS : Red Hat Enterprise Linux 8.7
CPU : Ice Lake
GPU : NVIDIA Ampere A100

Additional context

In between DFT calls, the LAMMPS calculator works fine.
There is one MPI task with 16 threads, while no_cpus =16, but i doubt it would slow the otf down so much.

[FromColleWithLove]

It might be that the "Time of MD steps" is actually the time taken by all the lammps steps between two DFT calls.

flare/flare/learners/otf.py

Line 541 in 07a6ff5

if self.md_engine == "PyLAMMPS":

However, performances still seem to be low.

Performance: 0.572 ns/day, 41.988 hours/ns, 6.616 timesteps/s, 297.703 atom-step/s
83.9% CPU use with 1 MPI tasks x 16 OpenMP threads

[omissis]

Performance: 0.513 ns/day, 46.749 hours/ns, 5.942 timesteps/s, 267.385 atom-step/s
85.4% CPU use with 1 MPI tasks x 16 OpenMP threads

I have very similar results when running on 16 cpu processors so just mpirun -np 16 lmp - around 270 atom-step/s