[Apbs-users] Parallel runs giving vastly different results from serial run

[J.Dziedzic <J.D...@so...>]

   Dear APBS users,

   I am observing weird results when attempting to use the parallel
features of APBS. Following the example for the actin dimer, I have
set up a seemingly simple calculation for H2O with a continuous
charge distribution.

   I get the expected result for the serial case, about -31 kJ/mol
and the output dx potential files look reasonable. The serial input
file looks like this:

--- cut here --- THIS WORKS
read
     mol pqr water_null.pqr
     charge dx water_continuous.dx
end

elec name solvated
     mg-auto
     dime 193 193 193
     nlev 4
     fglen 26.1818181817 26.1818181817 26.1818181817
     cglen 26.1818181817 26.1818181817 26.1818181817
     fgcent 12 12 12
     cgcent 12 12 12
     mol 1
     lpbe
     bcfl zero
     pdie 1.0
     sdie 80.0
     chgm spl2
     srfm smol
     srad 1.4
     swin 0.3
     sdens 10.0
     temp 298.15
     calcenergy total
     calcforce no
     usemap charge 1
     write pot dx sol-potential
     write charge dx sol-charge
end

elec name reference
     mg-auto
     dime 193 193 193
     nlev 4
     fglen 26.1818181817 26.1818181817 26.1818181817
     cglen 26.1818181817 26.1818181817 26.1818181817
     fgcent 12 12 12
     cgcent 12 12 12
     mol 1
     lpbe
     bcfl zero
     pdie 1.0
     sdie 1.0
     chgm spl2
     srfm smol
     srad 1.4
     swin 0.3
     sdens 10.0
     temp 298.15
     calcenergy total
     calcforce no
     usemap charge 1
     write pot dx ref-potential
     write charge dx ref-charge
end
print elecEnergy solvated - reference end
quit
--- cut here ---

   The charge map that is passed in contains the charge of both the
atomic cores and the electron cloud, obtained from a DFT code
and is apparrently correct, because the result agrees with
similar point-charge calculations and experiment. When I
manually integrate the resultant potential dx files and the
charge map, I get a result for the solvation energy that is
consistent with what APBS tells me. The pqr file
water_null.pqr deserves a word of explanation -- this is the
water molecule, stripped of all charge and serves as a placeholder
only, to make APBS happy. The whole charge I'm interested in
_is_ contained in the dx file. Since APBS refuses to accept
a zero-charge PQR file, I have set the charges to something
very small, i.e.

 > cat water_null.pqr
ATOM      1  O   XXX     1      11.537  12.359  12.000 0.001 1.50
ATOM      2  H   XXX     1      11.537  11.402  12.000 0.001 1.20
ATOM      3  H   XXX     1      12.463  12.598  12.000 0.001 1.20

   I do not pass the "true" charges of the cores in the pqr
file, because this leads to wrong boundary conditions if
sdh or mdh is used (since my molecule is neutral, I expect the
potential to be zero at large distances, passing the actual
core charges in the pqr file leads APBS to believe that the
cores should generate the boundary conditions -- the charge map
is ignored when these are determined).

   To reiterate, this works as expected, producing the expected
polar solvation energy (-31 kJ/mol) and the expected output
potential maps.

   Now, when I attempt to run this in parallel, by changing the
input file to look like this:

--- cut here --- THIS BREAKS
read
     mol pqr water_null.pqr
     charge dx water_continuous.dx
end

elec name solvated

     ### serial->parallel changes localized here ###
     mg-para
     dime 97 97 97
     pdime 2 2 2
     nlev 4
     ofrac 0.1
     fglen 26.1818181817 26.1818181817 26.1818181817
     cglen 26.1818181817 26.1818181817 26.1818181817
     fgcent 12 12 12
     cgcent 12 12 12
     ###############################################

     mol 1
     lpbe
     bcfl zero
     pdie 1.0
     sdie 80.0
     chgm spl2
     srfm smol
     srad 1.4
     swin 0.3
     sdens 10.0
     temp 298.15
     calcenergy total
     calcforce no
     usemap charge 1
     write pot dx sol-potential
     write charge dx sol-charge
end

elec name reference

     ### serial->parallel changes localized here ###
     mg-para
     dime 97 97 97
     pdime 2 2 2
     nlev 4
     ofrac 0.1
     fglen 26.1818181817 26.1818181817 26.1818181817
     cglen 26.1818181817 26.1818181817 26.1818181817
     fgcent 12 12 12
     cgcent 12 12 12
     ###############################################

     mol 1
     lpbe
     bcfl zero
     pdie 1.0
     sdie 1.0
     chgm spl2
     srfm smol
     srad 1.4
     swin 0.3
     sdens 10.0
     temp 298.15
     calcenergy total
     calcforce no
     usemap charge 1
     write pot dx ref-potential
     write charge dx ref-charge
end
print elecEnergy solvated - reference end
quit
--- cut here ---

... I run the calculation, then issue mergedx 193 193 193 ...
to obtain the merged dx files.

   These potential dx files, after merging, however, are quite
different from what was obtained previously. I expected
isosurfaces that would be of the same shape, with a minor
artifact at the "seams" between processors. What I get instead
is a very different isosurfaces, only the "seams" are as
expected :(. The discrepancy is huge for smaller values of
the potential, for larger values the distance between the
"right" serially obtained isosurface and the "wrong"
isosurface from parallel calculation diminishes.

   The solvation energy is way off as well -- APBS reports
-324 kJ/mol, whereas direct integration of the dx files
gives me -34 kJ/mol. In the serial calculation, by direct
integration, I could obtain the same value that APBS gave me.

   Using serial APBS with async does not help (behaves
identically to the parallel version)

   I (think I) have _excluded_ two culprits:

1) It is not the fault of mergedx, because even the partial
potential-PEn.dx files do not match the potential obtained
from a serial calculation.

2) Using mdh boundary conditions does not help things.

   Could you please suggest the reason for this weird
behaviour? I have a set of huge calculations (dime=769)
that I was hoping to split in this way, for they are
hopeless to run serially.

   I have posted the input .dx and .pqr files here:
www.mif.pg.gda.pl/homepages/jaca/give/test.tar.gz

   The apbs.in file can be extracted from this message.

   Thank you in advance,
- Jacek Dziedzic