Menu
▾
▴
Re: [Pytables-users] Pytables-users Digest, Vol 86, Issue 8
|
From: Andreas H. <li...@hi...> - 2013-07-18 07:37:44
|
On 18.07.2013 08:45, Pushkar Raj Pande wrote: > Both loadtxt and genfromtxt read the entire data into memory which is > not desirable. Is there a way to achieve streaming writes? > > Thanks, > Pushkar > > > On Wed, Jul 17, 2013 at 7:04 PM, Pushkar Raj Pande <top...@gm... > <mailto:top...@gm...>> wrote: > > Thanks Antonio and Anthony. I will give this a try. > > -Pushkar > > > On Wed, Jul 17, 2013 at 2:59 PM, > <pyt...@li... > <mailto:pyt...@li...>> wrote: > > Date: Wed, 17 Jul 2013 16:59:16 -0500 > From: Anthony Scopatz <sc...@gm... <mailto:sc...@gm...>> > Subject: Re: [Pytables-users] Pytables bulk loading data > To: Discussion list for PyTables > <pyt...@li... > <mailto:pyt...@li...>> > Message-ID: > > <CAP...@ma... > <mailto:CAPk-6T4Ht9%2BNcDd_1OojrbN4U_6%2BO...@ma...>> > Content-Type: text/plain; charset="iso-8859-1" > > Hi Pushkar, > > I agree with Antonio. You should load your data with NumPy > functions and > then write back out to PyTables. This is the fastest way to do > things. > > Be Well > Anthony > > > On Wed, Jul 17, 2013 at 2:12 PM, Antonio Valentino < > ant...@ti... > <mailto:ant...@ti...>> wrote: > > > Hi Pushkar, > > > > Il 17/07/2013 19:28, Pushkar Raj Pande ha scritto: > > > Hi all, > > > > > > I am trying to figure out the best way to bulk load data > into pytables. > > > This question may have been already answered but I couldn't > find what I > > was > > > looking for. > > > > > > The source data is in form of csv which may require parsing, > type > > checking > > > and setting default values if it doesn't conform to the type > of the > > column. > > > There are over 100 columns in a record. Doing this in a loop > in python > > for > > > each row of the record is very slow compared to just > fetching the rows > > from > > > one pytable file and writing it to another. Difference is > almost a factor > > > of ~50. > > > > > > I believe if I load the data using a C procedure that does > the parsing > > and > > > builds the records to write in pytables I can get close to > the speed of > > > just copying and writing the rows from 1 pytable to another. > But may be > > > there is something simple and better that already exists. > Can someone > > > please advise? But if it is a C procedure that I should > write can someone > > > point me to some examples or snippets that I can refer to > put this > > together. > > > > > > Thanks, > > > Pushkar > > > > > > > numpy has some tools for loading data from csv files like > loadtxt [1], > > genfromtxt [2] and other variants. > > > > Non of them is OK for you? > > > > [1] > > > > > http://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html#numpy.loadtxt > > [2] > > > > > http://docs.scipy.org/doc/numpy/reference/generated/numpy.genfromtxt.html#numpy.genfromtxt > > > > > > cheers > > > > -- > > Antonio Valentino > > > > > > > ------------------------------------------------------------------------------ > > See everything from the browser to the database with AppDynamics > > Get end-to-end visibility with application monitoring from > AppDynamics > > Isolate bottlenecks and diagnose root cause in seconds. > > Start your free trial of AppDynamics Pro today! > > > http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk > > _______________________________________________ > > Pytables-users mailing list > > Pyt...@li... > <mailto:Pyt...@li...> > > https://lists.sourceforge.net/lists/listinfo/pytables-users > > > -------------- next part -------------- > An HTML attachment was scrubbed... > > ------------------------------ > > ------------------------------------------------------------------------------ > See everything from the browser to the database with AppDynamics > Get end-to-end visibility with application monitoring from > AppDynamics > Isolate bottlenecks and diagnose root cause in seconds. > Start your free trial of AppDynamics Pro today! > http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk > > ------------------------------ > > _______________________________________________ > Pytables-users mailing list > Pyt...@li... > <mailto:Pyt...@li...> > https://lists.sourceforge.net/lists/listinfo/pytables-users > > > End of Pytables-users Digest, Vol 86, Issue 8 > ********************************************* > > > > > > ------------------------------------------------------------------------------ > See everything from the browser to the database with AppDynamics > Get end-to-end visibility with application monitoring from AppDynamics > Isolate bottlenecks and diagnose root cause in seconds. > Start your free trial of AppDynamics Pro today! > http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk > > > > _______________________________________________ > Pytables-users mailing list > Pyt...@li... > https://lists.sourceforge.net/lists/listinfo/pytables-users > You could use pandas_ and the read_table function. There, you have nrows and skiprows parameters with which you can easily do your own 'streaming'. .. _pandas: http://pandas.pydata.org/ -- Andreas |
