[Pytables-users] Row.append()

[Jim K. <jim...@sp...>]

I am trying to make this better / faster...
Data comes faster than I can store it on one box.  So My though was to have many boxes each storing their own part in their own table.
Later I would concatenate the tables together with something like this:

dest_h5f = pt.openFile(path + 'big_mater.h5','a')
for source_path in source_h5_path_list:
    h5f = pt.openFile(source_path,'r')
    for node in h5f.root:
        dest_table = dest_h5f.getNode('/', name = node.name)
        print node.nrows
        if node.nrows > 0 and node.nrows < 1000000:   # found I needed to limit the max size or I would crash
            dest_table.append(node.read())
            dest_table.flush()
    h5f.close()
dest_h5f.close()

I could add the logic to iter in chunks over the source data to overcome the crash and  but I suspect there could be a better way.

Take a table in one h5 file and append it to a table in another h5 file.   Looked like Table.copy() would do the trick but don't see how I get it to append to an existing table.

My h5 files have 4 rec arrays all stored in root.

Any suggestions?


________________________________

    Jim Knoll
     DBA/Developer II

     Spot Trading L.L.C
     440 South LaSalle St., Suite 2800
     Chicago, IL 60605
     Office: 312.362.4550
     Direct: 312-362-4798
     Fax: 312.362.4551
     jim...@sp...
     www.spottradingllc.com<http://www.spottradingllc.com/>

________________________________

The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you. Spot Trading, LLC