Menu ▾ ▴

SQLEnterDataIntoDatabase

Anonymous

SUPERCEDED

These instructions are superceded by the later instructions in HowToSetupPeriodAndVariableBasedDatabaseStructure

Introduction

The data for the graph pages are created from the CCDAS output by offline post-processing. When a fresh dataset is generated, it must be uploaded into the PostGreSQL database carbones_graph. This page explains how to do this.

It is also necessary to follow these steps when setting up the graph database on a machine for the first time, as described under DevQuickStart_GraphDatabase.

Details

The post-processing creates the data in text file format. You will need to obtain these files. Currently the latest files are on the CERC network here. 

P:\FM\FM862_CARBONES\Web site\20110506_20yearGraphData\
  • Copy the text files to a local directory.
  • It is advisable to avoid deeply nested directories or directory names containing spaces.
  • You will need to change the paths in the SQL scripts below from E:/PostgreSQL/Data/ to the correct path to the new data on your machine.
  • It is necessary to use forward slashes / as path delimiters even on Windows.

1. Enter data (in csv format):

This SQL will load the data.

  • On a Windows machine we have found it best not to execute all of these statements in one operation, but only two or three at a time. Particularly for the "raw" data which are very large.

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Babove.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Bbelow.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Btot.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fbbur.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fbbur_raw.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Ffos.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Ffos_raw.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fgpp.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fgppresp.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fgppresp_raw.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fgpp_raw.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fle.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fle_raw.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fnatural.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fnatural_raw.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fnee.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fnee_crop.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fnee_crop_raw.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fnee_for.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fnee_for_raw.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fnee_gras.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fnee_gras_raw.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fnee_raw.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Foce.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Foce_raw.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fresp.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Fresp_raw.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Ftot.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/Ftot_raw.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/LAI.txt'
    WITH DELIMITER ','
    CSV HEADER;

    COPY graphdata_new(startdate, "value", series_id, region_id)
    FROM 'E:/PostgreSQL/Data/SOC.txt'
    WITH DELIMITER ','
    CSV HEADER;

2. Run the ANALYZE command

The PostGreSQL documentation advises that you should run the ANALYZE command just after making major changes in the contents of a table, to improve the query performance. 

ANALYZE

This command can also be run from pgAdmin by right clicking on a table (or database) and choosing Maintenance.... The graphdata_new table will need to be analyzed.

3. That's all!

Hints

It takes a long time to import the data. You don't necessarily need to import it all to check that things are working properly.

You can start just with "Fnee", which is the daily/monthly/yearly data for "Net ecosystem flux". You won't be able to test any other variables or the 3-hourly data, but you can see whether the graphs are working on your local machine.

You can test that the stored procedure is running correctly by executing a SQL query like this select * from linegraph ('Fnee', 'day', 'mean', 'Global', '1990-05-01', '1990-05-10');

Background information

These SQL queries were created automatically. The code snippets for this have been saved here CodeSnippetCreateSQL. They may be useful if we need to generate new SQL for additional variables.

Related

Wiki: CodeSnippetCreateSQL
Wiki: DevQuickStartSidebar
Wiki: DevQuickStart_GraphDatabase
Wiki: HowToSetupPeriodAndVariableBasedDatabaseStructure

MongoDB Logo MongoDB