Dataspace application technical documentation

This document describes the technical and Python programming aspects of the Dataspace application. The documentation assumes that you have installed the application and that you are able to run it. If not, please refer to the install.txt file. Some familiarity with Python and the Flask web framework is beneficial.

This document reflects the state of the software as of June 2020.

Basic concepts

This software is a web application built for managing Comma Separated Values files for exploration and integration. It supports uploading such files (if you are the Admin user), editing their meta data, and exporting the data in the RDF format.

The directory structure of a running installation looks like this:

dataspace.py (the main file), classes.py (contains class definitions) and some auxiliary Python files.
\
directory templates: contains templates that assist with user communication with the main file
\
directory static: contains the uploaded CSV files and their ".meta" files. The meta files contain descriptions of the contents of the CSV files

Program flow example: The main page

When the user arrives on the main page of the application, database.py invokes the appmain function. This is because the application's route (URL) matches with the directive to call appmain.

After setting some global variables, appmain constructs an object of type MetaList. A MetaList represents the list of meta data information of CSV files managed by the application. When the MetaList constructor is executed, it reads the CVS meta information from the ".meta" files of the static directory (see above).

We now have the meta information of the CSV files, but it needs to shown to the user. For this end, the template "home.html" is called at the end of appmain by render_template. The MetaList variable is given as a parameter. Templates are found in the templates directory.

Since home.html is a front page template, it first calls another template (layout.html) that adds the menu bar and header elements on the page. After that, the main view of CSV files, their descriptions and their fields are built on the page by iterating over the elements of the MetaList variable.

Program flow example: Login

The login functionality is invoked by the URL /login calling the function login. This method is simple and it uses the WTF form library with

form = LoginForm()

if form.validate_on_submit():

..

The LoginForm is a class inside the forms.py containing only 3 lines: getting the username, password and building a submit button. The form's field are passed to the function login that verifies them. If the verification is successful, the username is added to the session. For functions that require that the user is logged in as the admin, the following check can be used: if 'username' in session and session['username'] == ADMIN_USERNAME ..

Program flow example: Edit file metadata

The CSV file metadata consists of the file's description, and meta data related to each of its fields.
The functionality is invoked by the URL /editmeta. Since the edit request is sent from the main page with the file name as a parameter, we read it to variable "myfile". A MetaInfo object (from classes.py) is constructed with myfile as parameter. The constructor does not read the data, so we read the existing metadata by mymeta.read_from_file. Next, we use the CSV file to read a sample line using getfieldsamples and get the field metadata using get_fieldlist.
This data is passed to template editmeta.html. The template contains a form with action {{ url_for('editmetasubmit') }}. Thus, when the form is submitted, the function editmetasubmit is called in dataspace.py. This function parses the input and saves it in JSON format in a file myfile.jmeta.

Program flow example: New CVS file

Function "new" allows the user to type (or copy/paste) CSV content into a textarea. This function utilizes the WTF form library so it is quite simple. If the form validation succeeds (we check that it is not empty), the content is stored into a file dataN.csv (N is a number) and the meta data editor (see above) is invoked for the meta data.