Screenshot instructions:
Windows
Mac
Red Hat Linux
Ubuntu
Click URL instructions:
Rightclick on ad, choose "Copy Link", then paste here →
(This may not be possible with some types of ads)
From: belinda thom <bthom@cs...>  20070103 03:57:46

Hi, Is there a way for me to keep adding the next row of a 2d array to a file via load? (matlab's save had a very useful append option). I managed to get 2d laod/save working, e.g. import numpy as N import pylab as P import bthom.utils as U # a numpy 2d array a = N.arange(12,dtype=N.float).reshape((3,4)) a[0][0] = N.pi # checking out the matploblib save/load stuff P.save("data.csv", a, fmt="%.4f", delimiter=";") aa = P.load("data.csv", delimiter= ";") x,y,z,w = P.load("data.csv", delimiter=";", unpack=True) The above took me longer than it perhaps should have b/c of advice I'd gotten elsewhere recommending trying to keep numpy and pylab separate when possible (to take advantage of all of numpy's features; it seems numpy doesn't even have the alltohandy load/save functionality). When I try similar tricks to write one row at a time, I'm hosed in that the shape is gone: # checking out a way to keep appending fname = "data1.csv" U.clobber_file(fname) #this thing just ensures 0 bytes in file f = open(fname,"a") nrows,ncols = a.shape for i in range(nrows) : P.save(f, a[i,:], fmt="%d", delimiter=";") f.close() aaa = P.load("data1.csv", delimiter= ";") in particular: % cat data1.csv 3 1 2 4 <snip> 11 Thanks in advance, b 
From: Steve Schmerler <elcorto@gm...>  20070103 10:01:09

belinda thom wrote: > Hi, > > Is there a way for me to keep adding the next row of a 2d array to a > file via load? (matlab's save had a very useful append option). > > I managed to get 2d laod/save working, e.g. > > import numpy as N > import pylab as P > import bthom.utils as U > # a numpy 2d array > a = N.arange(12,dtype=N.float).reshape((3,4)) > a[0][0] = N.pi > # checking out the matploblib save/load stuff > P.save("data.csv", a, fmt="%.4f", delimiter=";") > aa = P.load("data.csv", delimiter= ";") > x,y,z,w = P.load("data.csv", delimiter=";", unpack=True) > > The above took me longer than it perhaps should have b/c of advice > I'd gotten elsewhere recommending trying to keep numpy and pylab > separate when possible (to take advantage of all of numpy's features; > it seems numpy doesn't even have the alltohandy load/save > functionality). > > When I try similar tricks to write one row at a time, I'm hosed in > that the shape is gone: > > # checking out a way to keep appending > fname = "data1.csv" > U.clobber_file(fname) #this thing just ensures 0 bytes in file > f = open(fname,"a") > nrows,ncols = a.shape > for i in range(nrows) : > P.save(f, a[i,:], fmt="%d", delimiter=";") > f.close() > aaa = P.load("data1.csv", delimiter= ";") > > in particular: > > % cat data1.csv > 3 > 1 > 2 > 4 > <snip> > 11 > > Thanks in advance, > b > This is because pylab.save() writes every 1Darray (like a[i,:]) as "column vector". In the definition of the save() function: [...] if len(X.shape)==1: origShape = X.shape X.shape = len(X), 1 [...] This reshapes the 1Darray (len(a[i,:].shape) == 1) to a 2Darray of shape Nx1 and a loop over the first axis writes the rows (in this case one element per row) to file. There are several ways to do what you want: ## # generate data a = N.arange(12,dtype=N.float).reshape((3,4)) a[0][0] = N.pi P.save("data.csv", a, fmt="%.4f", delimiter=";") # (A) # rather hackish way, define your own save(), not # really useful, just to show that it works def save2(fname, X, fmt='%.18e',delimiter=' '): if is_string_like(fname): if fname.endswith('.gz'): import gzip fh = gzip.open(fname,'wb') else: fh = file(fname,'w') elif hasattr(fname, 'seek'): fh = fname else: raise ValueError('fname must be a string or file handle') X = N.asarray(X) origShape = None if len(X.shape)==1: origShape = X.shape # >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ## X.shape = len(X), 1 X.shape = 1, len(X) # <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< for row in X: fh.write(delimiter.join([fmt%val for val in row]) + '\n') if origShape is not None: X.shape = origShape fname = "data1.csv" f = open(fname,"a") nrows,ncols = a.shape for i in range(nrows) : save2(f, a[i,:], fmt="%f", delimiter=";") f.close() aaa = P.load("data1.csv", delimiter= ";") print aaa print "" # (B) # do it without a save() function fname = "data2.csv" f = open(fname,"a") nrows,ncols = a.shape delim = ';' fmt = '%f' for i in range(nrows): # just like in pylab.save() f.write(delim.join([fmt %val for val in a[i,:]]) + '\n') f.close() aaa = P.load("data2.csv", delimiter= ";") print aaa print "" # (C) # probably the best: save a 1xn "row vector" per line fname = "data3.csv" f = open(fname,"a") nrows,ncols = a.shape for i in range(nrows) : P.save(f, a[i,:].reshape((1,ncols)), fmt="%f", delimiter=";") f.close() aaa = P.load("data3.csv", delimiter= ";") print aaa HTH  cheers, steve Random number generation is the art of producing pure gibberish as quickly as possible. 