Re: [Pytables-users] PyTables data files for a tutorial

[Andy W. <wil...@gm...>]

On Sun, Oct 21, 2012 at 10:41 AM, Francesc Alted <fa...@py...> wrote:

> Hi,
>
> I'm going to give a tutorial on PyTables next Thursday during the PyData
> conference in New York (http://nyc2012.pydata.org/) and I'd like to use
> some real life data files.  So, if you have some public repository with
> data generated with PyTables, please tell me.  I'm looking for files
> that are not very large (< 1GB), and that use the Table object
> significantly.  A small description of the data included will be more
> that welcome too!
>
> Thanks!
>
> --
> Francesc Alted



Hi Francesc.

I've been working on a library for accessing climatology data that
uses pytables to cache data from the USGS. It could easily be used to
create a sample dataset for some area of interest. File size is
determined by how much data gets queried.


The general layout is:

/usgs/sites
- the sites table contains information and metadata about a site


/usgs/values/<AGENCY>/<SITE_CODE>/<PARAMETER_CODE>
- a table containing all the timeseries data for each site and
parameter is created as data are queried
- parameter codes are a bit obscure but a dict with descriptive
metadata stashed at table.attrs.variable
- the datetime column has a CSIndex on it and stored as as a string
because some sites have data prior to the year 1901
- pretty inefficient in terms of disk space (lots of large-ish string
columns) because it handles a very general class of data types


Here's what the code would look like to download and create the hdf5
file for 10 random sites in New York:

import ulmo

# the default location for the hdf5 file is OS dependent, so provide
the path you want to use
hdf5_file_path = './usgs_data.h5'

# get list of sites in NY
ulmo.usgs.pytables.update_site_list(state_code='NY', path=hdf5_file_path)
sites = ulmo.usgs.pytables.get_sites(path=hdf5_file_path)

# download data for a few random sites
for site in sites.keys()[:10]:
    ulmo.usgs.pytables.update_site_data(site, path=hdf5_file_path)



The project is on github: https://github.com/swtools/ulmo
and the code that does all the pytables stuff (including the table
descriptions) is here:
https://github.com/swtools/ulmo/blob/master/ulmo/usgs/pytables.py

-andy