SF.net SVN: matplotlib: [3974] trunk/py4science/workbook/files_etc.tex

[<jd...@us...>]

Revision: 3974
          http://matplotlib.svn.sourceforge.net/matplotlib/?rev=3974&view=rev
Author:   jdh2358
Date:     2007-10-20 14:57:26 -0700 (Sat, 20 Oct 2007)

Log Message:
-----------
finished working with files chapter

Modified Paths:
--------------
    trunk/py4science/workbook/files_etc.tex

Modified: trunk/py4science/workbook/files_etc.tex
===================================================================
--- trunk/py4science/workbook/files_etc.tex	2007-10-20 21:32:59 UTC (rev 3973)
+++ trunk/py4science/workbook/files_etc.tex	2007-10-20 21:57:26 UTC (rev 3974)
@@ -202,3 +202,96 @@
   since 2003}
 \end{figure}
 \par\end{center}
+
+
+\subsection{Loading and saving binary data}
+\label{sec:binary_data}
+
+ASCII is bloated and slow for working with large arrays, and so binary
+data should be used if performance is a consideration.  To save an
+array \texttt{X} in binary form, you can use the numpy
+\texttt{tostring} method
+
+\begin{lstlisting}
+In [16]: import numpy
+
+# create some random numbers
+In [17]: x = numpy.random.rand(5,2)
+
+In [19]: print x
+[[ 0.56331918  0.519582  ]
+ [ 0.22685429  0.18371135]
+ [ 0.19384767  0.27367054]
+ [ 0.35935445  0.95795884]
+ [ 0.37646642  0.14431089]]
+
+# save it to a data file in binary
+In [20]: x.tofile(file('myx.dat', 'wb'))
+
+# load it into a new array
+In [21]: y = numpy.fromfile(file('myx.dat', 'rb'))
+
+# the shape is not preserved, so we will have to reshape it
+In [22]: print y
+[ 0.56331918  0.519582    0.22685429  0.18371135  0.19384767
+0.27367054
+  0.35935445  0.95795884  0.37646642  0.14431089]
+
+In [23]: y.shape
+Out[23]: (10,)
+
+# restore the original shape
+In [24]: y.shape = 5, 2
+
+In [25]: print y
+[[ 0.56331918  0.519582  ]
+ [ 0.22685429  0.18371135]
+ [ 0.19384767  0.27367054]
+ [ 0.35935445  0.95795884]
+ [ 0.37646642  0.14431089]]
+\end{lstlisting}
+
+The advantage of numpy \texttt{tofile} and \texttt{fromfile} over
+ASCII data is that the data storage is compact and the read and write
+are very fast.  It is a bit of a pain that that meta ata like array
+datatype and shape are not stored.  In this format, just the raw binary
+numeric data is stored, so you will have to keep track of the data
+type and shape by other means.  This is a good solution if you need to
+port binary data files between different packages, but if you know you
+will always be working in python, you can use the python pickle
+function to preserve all metadata (pickle also works with all standard
+python data types, but has the disadvantage that other programs and
+applications cannot easily read it)
+
+\begin{lstlisting}
+# create a 6,3 array of random integers
+In [36]: x = (256*numpy.random.rand(6,3)).astype(numpy.int)
+
+In [37]: print x
+[[173  38   2]
+ [243 207 155]
+ [127  62 140]
+ [ 46  29  98]
+ [  0  46 156]
+ [ 20 177  36]]
+
+# use pickle to save the data to a file myint.dat
+In [38]: import cPickle
+
+In [39]: cPickle.dump(x, file('myint.dat', 'wb'))
+
+# load the data into a new array
+In [40]: y = cPickle.load(file('myint.dat', 'rb'))
+
+# the array type and share are preserved
+In [41]: print y
+[[173  38   2]
+ [243 207 155]
+ [127  62 140]
+ [ 46  29  98]
+ [  0  46 156]
+ [ 20 177  36]]
+\end{lstlisting}
+
+
+


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.