From: Ben K. <ben...@gm...> - 2011-01-13 15:37:43
|
My test code (modified from Mario's), ============================================= #!/usr/bin/env python from services.tables_settings import composite_h5_filename import tables import multiprocessing as mp def _worker(qout = None): fn = composite_h5_filename fp = tables.openFile(fn) print "About to load: ", fn rows = fp.root.t092010.where('(origWac <5.9) & (origWac>5.8)') count = 0 for row in rows: count +=1 print count fp.close() if qout is not None: qout.put("Done") if __name__ == "__main__": print "**** Running from main process:" _worker() print "**** Running from subprocess:" qout = mp.Queue() ps = mp.Process(target=_worker, args=(qout,)) ps.daemon = True ps.start() print qout.get(); ============================================= The test works for me if the where condition only returns a small number of rows (~40), but fails for larger queries (14000 rows). In these cases, the query will will finish on the main process, but the forked process hangs after iterating through about half the total rows on the fork. Admittedly, I don't know much about multiprocessing in python, and this hang might not be caused by pytables. Relevant versions, tables.__version__ Out[2]: '2.2pro' numexpr.__version__ Out[4]: '1.4.1' and python 2.6.6. -Ben |