Menu
▾
▴
matplotlib-users
|
From: Jahan M. <jah...@gm...> - 2010-10-07 16:12:04
|
Hi, I am an novice-intermediate user of python (I took a 1 semester course in scientific programming with python). I wanted to know what the best way is to manipulate, analyze, and plot Microsoft Excel data. The methods I've looked into: 1. Save data in CSV file and use csv.dictreader to create a dictionary. I actually couldn't figure this one out. 2. Read the CSV and extract the numerical data into an array of floats so I can manipulate the data. This is alright but then I lose the text for the header row. I have a fairly large biological data set, but I also really just want to practice with a good method. Any advice is much appreciated. Thanks! -Jahan Jahan Mohiuddin University of North Carolina at Chapel Hill, 2010 Cell: (516) 480-4825 |
|
From: João L. S. <js...@fc...> - 2010-10-07 17:05:05
|
On 10/07/2010 05:11 PM, Jahan Mohiuddin wrote: > Hi, > > I am an novice-intermediate user of python (I took a 1 semester course > in scientific programming with python). I wanted to know what the best > way is to manipulate, analyze, and plot Microsoft Excel data. The > methods I've looked into: > > 1. Save data in CSV file and use csv.dictreader to create a dictionary. > I actually couldn't figure this one out. > > 2. Read the CSV and extract the numerical data into an array of floats > so I can manipulate the data. This is alright but then I lose the text > for the header row. > > > I have a fairly large biological data set, but I also really just want > to practice with a good method. Any advice is much appreciated. Thanks! > > > -Jahan > You could use the python package xlrd, available at http://www.lexicon.net/sjmachin/xlrd.htm From what I remember the documentation wasn't all that great, but you should be able to access your data without too much trouble. Regards, João Silva |
|
From: Michael D. <md...@st...> - 2010-10-07 17:27:20
|
You may want to look at this as well: http://matplotlib.sourceforge.net/api/mlab_api.html?highlight=csv#matplotlib.mlab.csv2rec Mike On 10/07/2010 01:04 PM, João Luís Silva wrote: > On 10/07/2010 05:11 PM, Jahan Mohiuddin wrote: >> Hi, >> >> I am an novice-intermediate user of python (I took a 1 semester course >> in scientific programming with python). I wanted to know what the best >> way is to manipulate, analyze, and plot Microsoft Excel data. The >> methods I've looked into: >> >> 1. Save data in CSV file and use csv.dictreader to create a dictionary. >> I actually couldn't figure this one out. >> >> 2. Read the CSV and extract the numerical data into an array of floats >> so I can manipulate the data. This is alright but then I lose the text >> for the header row. >> >> >> I have a fairly large biological data set, but I also really just want >> to practice with a good method. Any advice is much appreciated. Thanks! >> >> >> -Jahan >> > You could use the python package xlrd, available at > http://www.lexicon.net/sjmachin/xlrd.htm > > From what I remember the documentation wasn't all that great, but you > should be able to access your data without too much trouble. > > Regards, > João Silva > > > ------------------------------------------------------------------------------ > Beautiful is writing same markup. Internet Explorer 9 supports > standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2& L3. > Spend less time writing and rewriting code and more time creating great > experiences on the web. Be a part of the beta today. > http://p.sf.net/sfu/beautyoftheweb > _______________________________________________ > Matplotlib-users mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-users |
|
From: John H. <jd...@gm...> - 2010-10-07 18:01:33
|
On Thu, Oct 7, 2010 at 12:27 PM, Michael Droettboom <md...@st...> wrote: > You may want to look at this as well: > > http://matplotlib.sourceforge.net/api/mlab_api.html?highlight=csv#matplotlib.mlab.csv2rec And these examples: http://matplotlib.sourceforge.net/search.html?q=codex+csv2rec JDH |
|
From: Alessio C. <via...@gm...> - 2010-10-10 07:04:09
|
Hi, a strong advice from someone who is using excel format with tons of data is to save them in csv and then import in Sqlite. Excel messes up the data types and gives a lot of troubles with numbers. Sqlite is fast and data are secure. The power of this system is that you can query your data and plot what you need for example. I can share with you my script to import from csv to sqlite if you want. Jahan Mohiuddin wrote: > > Hi, > > I am an novice-intermediate user of python (I took a 1 semester course > in scientific programming with python). I wanted to know what the > best way is to manipulate, analyze, and plot Microsoft Excel data. > The methods I've looked into: > > 1. Save data in CSV file and use csv.dictreader to create a > dictionary. I actually couldn't figure this one out. > > 2. Read the CSV and extract the numerical data into an array of > floats so I can manipulate the data. This is alright but then I lose > the text for the header row. > > > > I have a fairly large biological data set, but I also really just want > to practice with a good method. Any advice is much appreciated. > Thanks! > > > > -Jahan > > > Jahan Mohiuddin > University of North Carolina at Chapel Hill, 2010 > Cell: (516) 480-4825 > > > > > > ------------------------------------------------------------------------------ > Beautiful is writing same markup. Internet Explorer 9 supports > standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3. > Spend less time writing and rewriting code and more time creating great > experiences on the web. Be a part of the beta today. > http://p.sf.net/sfu/beautyoftheweb > _______________________________________________ > Matplotlib-users mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-users > > -- View this message in context: http://old.nabble.com/Best-way-to-use-Excel-Data-tp29908079p29926020.html Sent from the matplotlib - users mailing list archive at Nabble.com. |
|
From: Peter B. <bu...@gm...> - 2010-10-10 10:56:35
|
Hi, To load csv data, I use a modified version of csv2rec for which the data type of each column is specified explicitly in the data file. By removing the dtype guessing you get a speedup and you also avoid potential mess-ups. Alessio: sadly you right about it not being possible to trust Excel with data. Could you please give more details on the sqlite method you suggest ? -- >> by Alessio Civ Oct 10, 2010; 09:04am: Hi, a strong advice from someone who is using excel format with tons of data is to save them in csv and then import in Sqlite. Excel messes up the data types and gives a lot of troubles with numbers. Sqlite is fast and data are secure. The power of this system is that you can query your data and plot what you need for example. I can share with you my script to import from csv to sqlite if you want. -- thanks, peter butterworth |
|
From: Alessio C. <via...@gm...> - 2010-10-10 18:42:26
|
Hi Butterw, first of all, if you are working on data a lot, we could get in contact. I need to work better on my scripts and we could help each other. I've uploaded my script that imports from csv to sqlite. It's not a totally clean script, as I am working on something else now I couldn't clean what I did. Anyway, you should get an idea of what it does. Basically, the script cleans up the messed data from excel (take out the comas, puts point, convert numbers to floats). Then, the scripts writes the resulting tuple in an sqlite database. This script import over 200000 records in around 3 seconds. After this, you can access your data connecting to sqlite. Accessing sqlite data is extremely easy and efficient. I'm uploading a small script to plot from Sqlite and getting the values for a linear regression, it just works fine and smoothly. Let me know if are interested in helping each other. http://old.nabble.com/file/p29928938/SqlitePlot_GDF.py SqlitePlot_GDF.py http://old.nabble.com/file/p29928938/SqliteImportCsv.py SqliteImportCsv.py butterw wrote: > > Hi, > > > To load csv data, I use a modified version of csv2rec for which the > data type of each column is specified explicitly in the data file. > By removing the dtype guessing you get a speedup and you also avoid > potential mess-ups. > > > Alessio: sadly you right about it not being possible to trust Excel with > data. > Could you please give more details on the sqlite method you suggest ? > > > -- >>> by Alessio Civ Oct 10, 2010; 09:04am: > Hi, > > a strong advice from someone who is using excel format with tons of > data is to save them in csv and then import in Sqlite. > > Excel messes up the data types and gives a lot of troubles with > numbers. Sqlite is fast and data are secure. > The power of this system is that you can query your data and plot what > you need for example. > > I can share with you my script to import from csv to sqlite if you want. > > > -- > thanks, > peter butterworth > > ------------------------------------------------------------------------------ > Beautiful is writing same markup. Internet Explorer 9 supports > standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3. > Spend less time writing and rewriting code and more time creating great > experiences on the web. Be a part of the beta today. > http://p.sf.net/sfu/beautyoftheweb > _______________________________________________ > Matplotlib-users mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-users > > -- View this message in context: http://old.nabble.com/Best-way-to-use-Excel-Data-tp29908079p29928938.html Sent from the matplotlib - users mailing list archive at Nabble.com. |
|
From: Peter B. <bu...@gm...> - 2010-10-11 16:30:28
|
Hi Alessio, Thank you for the sqlite code example. What have been the key advantages of using a Database over a structured array for your applications ? http://docs.python.org/library/sqlite3.html : SQLite is a C library that provides a lightweight disk-based database that doesn’t require a separate server process and allows accessing the database using a nonstandard variant of the SQL query language. On Sun, Oct 10, 2010 at 12:56 PM, Peter Butterworth <bu...@gm...> wrote: > Hi, > > > To load csv data, I use a modified version of csv2rec for which the > data type of each column is specified explicitly in the data file. > By removing the dtype guessing you get a speedup and you also avoid > potential mess-ups. > > > Alessio: sadly you right about it not being possible to trust Excel with data. > Could you please give more details on the sqlite method you suggest ? > > > -- >>> by Alessio Civ Oct 10, 2010; 09:04am: > Hi, > > a strong advice from someone who is using excel format with tons of > data is to save them in csv and then import in Sqlite. > > Excel messes up the data types and gives a lot of troubles with > numbers. Sqlite is fast and data are secure. > The power of this system is that you can query your data and plot what > you need for example. > > I can share with you my script to import from csv to sqlite if you want. > > > -- > thanks, > peter butterworth > -- thanks, peter butterworth |
|
From: Alessio C. <via...@gm...> - 2010-10-15 04:52:58
|
Let' put things this way: if you have to work with many records, it is better if you have a database. With a database you can query what you need and only this is worthed the effort of using a DB. butterw wrote: > > Hi Alessio, > > Thank you for the sqlite code example. > > What have been the key advantages of using a Database over a > structured array for your applications ? > > http://docs.python.org/library/sqlite3.html : > SQLite is a C library that provides a lightweight disk-based database > that doesn’t require a separate server process and allows accessing > the database using a nonstandard variant of the SQL query language. > > On Sun, Oct 10, 2010 at 12:56 PM, Peter Butterworth <bu...@gm...> > wrote: >> Hi, >> >> >> To load csv data, I use a modified version of csv2rec for which the >> data type of each column is specified explicitly in the data file. >> By removing the dtype guessing you get a speedup and you also avoid >> potential mess-ups. >> >> >> Alessio: sadly you right about it not being possible to trust Excel with >> data. >> Could you please give more details on the sqlite method you suggest ? >> >> >> -- >>>> by Alessio Civ Oct 10, 2010; 09:04am: >> Hi, >> >> a strong advice from someone who is using excel format with tons of >> data is to save them in csv and then import in Sqlite. >> >> Excel messes up the data types and gives a lot of troubles with >> numbers. Sqlite is fast and data are secure. >> The power of this system is that you can query your data and plot what >> you need for example. >> >> I can share with you my script to import from csv to sqlite if you want. >> >> >> -- >> thanks, >> peter butterworth >> > > > > -- > thanks, > peter butterworth > > ------------------------------------------------------------------------------ > Beautiful is writing same markup. Internet Explorer 9 supports > standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3. > Spend less time writing and rewriting code and more time creating great > experiences on the web. Be a part of the beta today. > http://p.sf.net/sfu/beautyoftheweb > _______________________________________________ > Matplotlib-users mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-users > > -- View this message in context: http://old.nabble.com/Best-way-to-use-Excel-Data-tp29908079p29968676.html Sent from the matplotlib - users mailing list archive at Nabble.com. |
|
From: Christopher B. <Chr...@no...> - 2010-10-15 15:50:10
|
On 10/14/10 9:52 PM, Alessio Civ wrote: > Let' put things this way: if you have to work with many records, it is > better if you have a database. pyTables is worth a look, too" http://www.pytables.org/moin -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chr...@no... |
