[Rdkit-discuss] parallel conformation generation

[Andrew D. <da...@da...>]

JP mentioned that he would like to use multiple CPUs for his conformation
generation code, which he posted during his talk today. Here's a straight-forward
parallelization of his code. On my laptop, with 4 processes allocated,
I get a 2.5x speedup. I have an Intel Core i7 with 1 CPU and 4 cores.

It uses the concurrent.futures module which is standard in Python 3.2 and later.
Since RDKit doesn't support Python 3.x, I'm actually using the back port for
Python 2.x, which is available from the Python package index. I use it often. I
even include it as part of the chemfp distribution. I presented some examples of
how to use concurrent.futures at the EuroPython 2012 conference. You can see the
presentation and video at

  https://ep2012.europython.eu/conference/talks/concurrentfutures-is-here



During the "developing the community" round-table we talked about posting more
code examples to the list, so here it is.



import sys
from rdkit import Chem
from rdkit.Chem import AllChem

# Download this from http://pypi.python.org/pypi/futures
from concurrent import futures

## On my machine, it takes 39 seconds with 1 worker and 10 seconds with 4.
## 29.055u 0.102s 0:28.68 101.6%   0+0k 0+3io 0pf+0w
#max_workers=1

## With 4 threads it takes 11 seconds.
## 34.933u 0.188s 0:10.89 322.4%   0+0k 125+1io 0pf+0w
max_workers=4

# (The "u"ser time includes time spend in the children processes.
#  The wall-clock time is 28.68 and 10.89 seconds, respectively.)


# This function is called in the subprocess.
# The parameters (molecule and number of conformers) are passed via a Python pickle.
def generateconformations(m, n):
    m = Chem.AddHs(m)
    ids=AllChem.EmbedMultipleConfs(m, numConfs=n)
    for id in ids:
        AllChem.UFFOptimizeMolecule(m, confId=id)
    # EmbedMultipleConfs returns a Boost-wrapped type which
    # cannot be pickled. Convert it to a Python list, which can.
    return m, list(ids)


smi_input_file, sdf_output_file = sys.argv[1:3]

n = int(sys.argv[3])

writer = Chem.SDWriter(sdf_output_file)

suppl = Chem.SmilesMolSupplier(smi_input_file, titleLine=False)

with futures.ProcessPoolExecutor(max_workers=max_workers) as executor:
    # Submit a set of asynchronous jobs
    jobs = []
    for mol in suppl:
        if mol:
            job = executor.submit(generateconformations, mol, n)
            jobs.append(job)

    # Process the job results (in submission order) and save the conformers.
    for job in jobs:
        mol, ids = job.result()
        for id in ids:
            writer.write(mol, confId=id)
        
writer.close()



Enjoy!

				Andrew
				da...@da...