Re: [Pytables-users] threading/GIL

[Francesc A. <fa...@ca...>]

A Dimecres 15 Desembre 2004 18:31, Andrew Straw va escriure:
> It may be more clear to say this patch improves "interactivity" rather=20
> than "performance".  (Even on single CPU systems.) To explain, imagine=20
[...]
> usually I/O bound, not CPU bound).  In fact, Python's GIL is a major=20
> obstacle to writing multi-threaded programs that make use of=20
> multi-CPUs, so I envision this patch having perhaps more significant=20
> impact on single CPU machines.

Aha!, very good explanation. That's been very interesting to know about.

> Note that pytable's realtime compression does make "disk access" much=20
> more CPU bound.  Thus, releasing the GIL may not result in speed=20
> increases on single CPUs with compression turned on.  I haven't dug=20
> into the pytables internals very much, but if the (de)compression can=20
> be done after releasing the GIL, I predict this would improve=20
> performance on a multi-CPU system.  In that case, one CPU can busy=20
> itself with compressing data, while another CPU can perform other=20
> tasks.

I'm afraid that compression/uncompression is driven by HDF5 itself, so, with
the actual patch it will happen in the same thread as the one doing the real
I/O. However, as you already said, pytables online compression is more "disk
access" bound than CPU bound, specially when fast compressors (i.e. LZO) or
fast CPU's are used. With the advent of newer CPUs the bottleneck will be
more and more on "disk access", so it may not be worth the effort trying to
make pure I/O and compression happen in different threads.

> So, I hope this example is relatively clear. As I said, I will endeavor=20
> to write a simple (as possible) example.

That would be great. Thanks!

=2D-=20
=46rancesc Altet
Who's your data daddy? =A0PyTables