Re: [Pytables-users] concurrent accesses

[Owen M. <owe...@bc...>]

If you're reading the data out of the file from inside a generator (ie - if
load_items returns a generator that accesses the HDF5 file) then as the
worker processes consume the work items the file is actually being opened
and read from a worker thread in the master process.

Regards,
Owen


On 2 November 2012 21:49, Ben Elliston <bj...@ai...> wrote:

> Hi Francesc
>
> > Hmm, now that I think, Blosc is not thread safe, and that can bring
> > these sort of problems if you use it from several threads (but it
> > should be safe when using several *processes*).
>
> I am using multiprocessing.Pool, like so:
>
> if __name__ == '__main__':
>    pool = Pool(processes=2)              # start 2 worker processes
>    items = load_items ()
>    pool.map (process_items, items)
>
> Ben
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iQIVAwUBUJQx0+ZqTimv57y9AQigFQ//XWpDoIve2PDag4SG/JBu6Y4D5X8pZcfA
> froFDmju5dGtlCaRKUc/puioFRukyD2s5QCV9hvYIGQ2EYkts1eKLQnYRvcV/37D
> J6QVEGUcuqfdj6lnGZSiDHr24rCeT3oGozbYO0/6casyV4iIuRjOzghWgKjV7ko1
> N/dy2UGK0S1S2Ws/OnkzDlbXiZShHfjLw3au2TtCdXaPcA4X1aMs1qLzEzvd+gJb
> MLHy3MVtGCxjtnB3Vzi/2UmgMfB6hFuGugD2Yp2i0SxFIlTS7cIXYB2beL+x+y4Q
> MtcZZO7QOvTvoExRje0BWR0e0BAOumKACKiU7Uq8L0+rwT6NWPfOVfe6KhieHvi/
> bh8oiNl2tekB+UE5JQ6Yi13YwfReyA1M8RFRsrQ3fXCWaQ6+Hx3m8+t/q4bfDhy4
> wFrC4N3hIkqMNI589aju8vVWerSdKDrzqFjcLBp8zfY718qm0ulGz0bBsKNPzr8y
> 7AN+/5StSos29oBTv3p3s7YDLjXoDi4XwZPR0aMt9pO2JrPxVUda+oi8/1AT46RS
> QoyFGmkqotucR22rWXbv9wZdHm9SsxGZakNpeQYxN9uf2aaqP0eodbgmSPx1ST8P
> 8vU+uGqrCb564KW5eJmQCgNZMXo1uCAqCdF5LZT+h1ncWy9//bv9gW2mityDNs6j
> YOqKMSOj9e0=
> =ZvxS
> -----END PGP SIGNATURE-----
>
>
> ------------------------------------------------------------------------------
> LogMeIn Central: Instant, anywhere, Remote PC access and management.
> Stay in control, update software, and manage PCs from one command center
> Diagnose problems and improve visibility into emerging IT issues
> Automate, monitor and manage. Do more in less time with Central
> http://p.sf.net/sfu/logmein12331_d2d
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>