From: Glenn <jon...@ca...> - 2008-06-17 11:24:23
|
Francesc Alted <faltet <at> pytables.com> writes: > > A Monday 16 June 2008, Glenn escrigué: > > Hello, > > I am storing 400000 rows to an EArray as follows: > > if grp.__contains__('normI'): > > fh.removeNode(grp,'normI') > > fh.createEArray(grp,'normI',Float32Atom(), (0,512), > > expectedrows=800000) > > > > ... populate 400000 rows of normI array ... > > > > When I use it as follows: > > tmp = np.asarray(grp.normI[:,k]) # Grab the k'th column of the > > Earray tmp = SomeCalculation(tmp) #this is very fast > > grp.SomeCArray[:,k] = tmp #this is also very fast, but I am only > > storing ~100 # values, so I'm not sure if it actually has good > > performance or not > > > > > > it is horribly slow, the np.asarray call takes ~30 seconds, which is > > only 32Kbyte/s if only 400000*4 bytes are being read as it should be, > > but 16Mbyte/s if all 512*4*400000 are being read, and then sliced. > > When I check the disk read performance, I see that indeed it is > > reading continuously at around 16 Mbyte/s. Am I doing something > > wrong? > > Mmmm, I think your message above misses some information. Could you > please double check which is exactly the statement showing slow > performance? > > Also, I'm not sure why you are using the: > > tmp = np.asarray(grp.normI[:,k]) > > idiom. It is not: > > tmp = grp.normI[:,k] > > enough? Not that the asarray() call would be slowing down things, it is > just that I'm curious. > > Cheers, > I tried both with and without asarray, same result. My loop looks like this: tic = time.time() for k in range(512): print "start", k, time.time()-tic tmp = np.asarray(grp.normI[:,k]) print "get asarray", k, time.time() - tic tmp = MyCalculation(tmp) print "calc", k, time.time() - tic grp.tpavI[:,k] = tmp print "store", k, time.time() - tic I clearly see that the only part that is taking up any time is reading the data in, because all of the other print statements do not show any increase in time. If this is the correct way of accessing the EArrays, and should be fast, then I will see if I can make a simple example to reproduce the effect. Thanks always for the help, Glenn |