Re: [Pytables-users] Clear chunks from CArray

[Giovanni L. C. <glc...@gm...>]

hi Anthony and Antonio,

thanks for the explanations. I was hoping I could do it programmatically, but if 
it is an inherent limitation of HDF5 there is little I can do. Compression plus 
a sensible chunk size should help though.

Best,

Giovanni

On 08/06/2013 04:11 AM, pyt...@li... wrote:
> Hi Anthony, hi Giovanni,
>
> Il giorno 06/ago/2013, alle ore 00:45, Anthony Scopatz<sc...@gm...>  ha scritto:
>
>> On Mon, Aug 5, 2013 at 3:14 PM, Giovanni Luca Ciampaglia<glc...@gm...>  wrote:
>> Hi Anthony,
>>
>> what do you mean precisely? I tried
>>
>> del ca[:,:]
>>
>> but CArray does not support __delitem__. Looking in the documentation I could
>> only find a method called remove_rows, but it's in Table, not CArray. Maybe I am
>> missing something?
>>
>> Huh, it should...  This is definitely an oversight on our part.  If you could please open an issue for this -- or better yet -- write a pull request that implements delitem, that'd be great!
>>
>> So I think you are right that there is no current way to delete rows from a CArray.  Oops!  (Of course, I may still be missing something as well).
>>
>> It looks like EArray also has this problem too, otherwise I would just tell you to use that.
> I'm not sure to understand the problem.
>
> The "truncate" method of arrays can be used to remove rows from an extendable array.
> It seems to me that it is not documented but we should add it to the UG.
>
> CArrays cannot be resized.
>
>> Be Well
>> Anthony
>>   
>>
>> Thank,
>>
>> Giovanni
>>
>> On Mon 05 Aug 2013 03:43:42 PM EDT,pyt...@li...
>> wrote:
>>>> Hello Giovanni, I think you may need to del that slice and then possibly
>>>> repack. Hope this helps. Be Well Anthony On Mon, Aug 5, 2013 at 2:09 PM,
>>>> Giovanni Luca Ciampaglia <glc...@gm...> wrote:
>>>>> Hello all,
>>>>>
>>>>> is there a way to clear out a chunk from a CArray? I noticed that setting
>>>>> the
>>>>> data to zero actually takes disk space, i.e.
>>>>>
>>>>> ***
>>>>> from tables import open_file, BoolAtom
>>>>>
>>>>> h5f = open_file('test.h5', 'w')
>>>>> ca = h5f.create_carray(h5f.root, 'carray', BoolAtom(), shape=(1000,1000),
>>>>> chunkshape=(1,1000))
>>>>> ca[:,:] = False
>>>>> h5f.close()
>>>>> ***
>>>>>
>>>>> The resulting file takes 249K ...
>>>>>
>>>>> Best,
>>>>>
>>>>> --
>>>>> Giovanni Luca Ciampaglia
> HDF5 handles efficiently chunks that have never been written saving some disk space but I doubt that chunks can be "de-initializad".
> If my understanding is correct, once one write some value in a chunk (even if it is the default value) the chunk is allocated at HDF5 level and written to disk.
> At they point one can only change item values.
> Also I doubt that a repack can help in this case (not tested).
>
> The only solution IMO is compression.
>
>
> cheers
>
> --
> Antonio Valentino


-- 
Giovanni Luca Ciampaglia

Postdoctoral fellow
Center for Complex Networks and Systems Research
Indiana University

✎ 910 E 10th St ∙ Bloomington ∙ IN 47408
☞ http://cnets.indiana.edu/
✉ gci...@in...