QuickStartDataVaultAndGrapher

Step 3: LabRAD Modules, Part I

Overview

Here, we'll finally add some real functionality to your LabRAD System by introducing two very powerful Modules that you can use to manage your data storage:

• The Data Vault Server Module provides a very flexible way to store datasets including meta-data like comments, time-stamps, experimental parameters, etc.
• The Grapher Client Module will monitor the Data Vault for the arrival of new datasets and plot them intelligently on the fly. It also allows you to browse all other datasets you have ever taken and add comments to them in the form of an instant messaging session associated with each dataset.

Running the Data Vault

Our current version of the Data Vault was written in Python. This would usually mean that you need to install the Python interpreter and required packages. But to keep things simple for the tutorial, we packaged everything up into a single Windows executable - DataVault-v1.0.1.exe - that you can run directly from your browser, just like the Manager and Explorer. The file is a self-extracting archive that contains the Data Vault and a couple of DLLs required to run it without a Python installation. Apart from extracting the files into a temporary directory, it will not install anything on your machine. It requires one external DLL to be able to run: MSVCRT71.DLL. Most systems have this installed already, but if yours doesn't you need to copy it into the System (not System32!) folder inside your Windows directory. Since this DLL is copyrighted by Microsoft, we can't just offer it for download here, but one place where you can find it is, if you have Microsoft Office installed, in the directory C:\Program Files (x86)\Microsoft Office\Office12\ADDINS.

Aside: Should you be using Linux or know already that you will have Python running, you can install Python 2.5.1, the Twisted Networking Package, and the pylabrad API and use them to run DataVault-v1.0.1.py directly in Python.
I am still working on porting the Data Vault to Delphi to have all programs in this tutorial behave the same way and to make it faster at handling large datasets. But I didn't want to keep you waiting for that since this version already performs really well.

Since this version of the Data Vault is not based on the Delphi API, it will prompt you slightly differently for the required information to connect to the Manager. When you run the executable, you will see a command shell type window with only text. You will be asked a couple of questions, like what the password for the manager connection is and where you would like to have your data stored (the latter, you can leave at the default for now by just hitting [Enter]). If everything goes as planned, you will eventually see a line of text that ends in "Data Vault now serving." and the Data Vault should show up in the Explorer, which you need to have running for the next steps. The Data Vault's window should end up looking somewhat like this:

Unfortunately, should any errors occur, the Data Vault will simply quit and disappear, making it hard to debug. Overall, though, getting the Data Vault running should be fairly identical to getting the Explorer running, so simply use the exact same settings and password that you used there. Maybe you just mistyped the password?

Exploring the Data Vault

Take a minute to look over the Data Vault's settings in the Explorer and see if you can understand how to use it simply from the help texts provided. Since this tutorial is all about instant gratification, though, we should probably get the Grapher going first before we try too hard to play with the Data Vault.

Running the Grapher

The Grapher, just like the Manager and Explorer, was written in Delphi and compiled into a stand-alone executable - Grapher-v1.0.0.exe. Follow your usual method of running it. The window it will great you with, depending on how much you've already played with the Data Vault, might be as empty as this:

Notice how the Grapher, just like the Explorer itself, does not show up in the Explorer's Servers list. This is because the Grapher and Explorer are LabRAD Client Modules that use the LabRAD System to provide functionality to the user rather than using hardware, etc. to provide functionality to the LabRAD System.

Creating and Navigating Directories

In the Explorer, create a new packet for the Data Vault containing a single record for the mkdir setting. As its content enter 'Test'. As soon as you send the packet, the new Test directory should appear in the Grapher's Browser panel. You can double-click the Test folder to enter it and you will find a .. entry that will allow you to exit the folder again.

Now, if you go back to the Explorer and change the content of your mkdir request packet to 'Test 2' and send it, you will notice that the new directory also got created in the root directory, i.e. next to Test. In other words mkdir does not change you into the directory it creates.

Also notice that the directory you are looking at with the Grapher is independent of the directory you are working in with the Explorer. Therefore, to add folders or datasets inside the Test directory, we first need to change into it inside the Explorer by sending a packet to setting cd with content 'Test'. The answer to the cd request will contain your new current working directory, in this case probably ['', 'Test'] (the '' entry (which is two apostrophes, i.e. an empty String, not one quotation mark) in the array stands for the root directory). To request the current working directory without changing it, simply send an empty record to cd. To move back out of a subdirectory, you can either send a w to cd containing the number of directories you want to move up or you can specify a new absolute path by sending something like ['', 'Test 2'].

Play with creating and navigating directories a bit more until you feel comfortable with the steps involved. After, change into the 'Test' directory in both, the Explorer and the Grapher.

Creating new Datasets and Adding Datapoints

In the Grapher's main menu, click on Plot Windows -> Show Live View. This will open an empty container window that will automatically display new datasets that you create.

Back in the Explorer, make a new packet for the Data Vault with a single record for the new setting with content 'Half-Life', [('x', 'in')], [('y','','in')] and send it. This will create a new dataset named 00001 - Half-Life containing a single trace of data with the independent axis labeled x [in] and the dependent axis labeled y [in]. The Grapher, provided it is showing the same current directory as the Explorer, will show the new dataset in the Browser as well as an empty plot for it in the right half of the Live View window.

You can now add data to the dataset by making a new packet in the Explorer for the Data Vault's add setting with the following content:

[[0.0,0.2],[1.0,0.2],[2.4,2.3],[3.3,0.0],[4.7,0.4],[4.5,1.2],[3.8,1.0],[2.3,4.8],[1.1,4.8],[1.1,4.1],[1.7,4.1],[2.0,3.4],[0.0,0.2]]

The Grapher should immediately update the plot in the Live View window with the new data points.

Annotating Datasets

The Data Vault offers one automated and two manual ways of annotating datasets.

Time Stamps

Apart from the description of the independent and dependent variables, the Data Vault also automatically stores the creation, last modification, and last access times with each dataset and folder. Currently, the only way to retrieve these time stamps is by looking at the *.ini files associated with the datasets. In the Delphi version of the Data Vault, they will be more easily accessible, most likely as part of the directory listing.

Parameters

You can manually add parameters to datasets. Parameters can usually be thought of as independent variables that are held constant over the dataset, like the sampling rate of some D/A converter or some constant bias. But parameters can also store non-numerical values, like the name of the person who took the dataset. To add parameters to a dataset, the Data Vault provides the conveniently named add parameter setting. Why don't you make a new packet for the Data Vault and add a record for add parameter with the content 'Data Source', 'LabRAD Tutorial'. If you look at the Browser in the Grapher's main window and double-click on the dataset you are currently working on, you should see the new parameter appear in the Parameters list. This list, like everything else, automatically updates with new information if you already have the dataset open.

Note: Parameters can only be added to the last new dataset created in a session. Usually you want to add all parameters immediately after creating the new dataset. As soon as you create or open another dataset or close the Manager connection that you used to create the dataset, you will no longer be able to add more parameters without manually updating the respective .ini files (discouraged!). In other words: Parameters are immutable. The same is true for data points.*

Comments

Comments can be added by anyone at any time to any dataset. The Grapher allows you to do this in the form of an instant messaging chat associated with each plot. In the left half of every plot window you will see a panel labeled Chat. This panel consists of two parts: The top shows comments that have already been added (of course auto-updating), while the bottom allows you to enter and send new comments.

Let's try this: Go into the text editor in the bottom left corner of the window containing your Half-Life plot and type Valve, Half-Life, and the Lambda logo are trademarks and/or registered trademarks of Valve Corporation and press [Ctrl]+[Enter].
The first time you are submitting a comment, the Grapher will ask you for your name with the following dialog:

The comment should appear immediately in the upper part of the Chat panel, leaving your window looking something like this:

Benefits

Now, if you combine this commenting model with the network distributed and modular nature of LabRAD, you get some neat new capabilities:

Usually, if you were to write a program like the Grapher without using LabRAD, you would include the functionality provided by the Data Vault into the program directly, for example as an imported package, unit, or sub-VI.

This has two implications:

• As soon as one copy of your graphing program has a file containing a dataset opened, another copy cannot fully access it any more (try opening the same spreadsheet with two copies of Excel to see what I mean).
• Your graphing program needs to access the files from the computer you are running it on. Thus, you need to somehow map a network drive with your data files if you would like to access them remotely.

The LabRAD model takes care of both of these implications for you:

• All dataset files are managed by a single copy of the Data Vault, which can now have full, exclusive access to the files and manage them with all the flexibility needed. (This centralized resource management becomes even more important if you are sharing hardware between different users. For example, in our lab we can have many people taking data on the same sample at the same time, since all hardware access is scheduled centrally by a single server which then simply time-slices the experimental runs; just like your operating systems time-slices your processor's capacity to run multiple programs at the same time.)
• Since the Grapher talks to the LabRAD System over the network, the multiple copies of the Grapher can be run anywhere in the world, provided you have a connection to the Internet on both sides.

Put all this together and you can host data-taking sessions with multiple collaborators located anywhere in the world, each looking at the live data-stream using their own copy of the Grapher. Each dataset immediately has a chat session associated with it that allows for live discussion of the incoming data. The discussion is automatically stored with each dataset and can later be reviewed easily.

This kind of distributed, independent, concurrent, but centrally managed resource access - like multiple Graphers accessing a single copy of the Data Vault that is in sole charge of the data files, or different client modules talking to different channels on the same GPIB device at the same time - is one of the main benefits of using LabRAD over conventional monolithic data-taking programs.

Oh, and before I forget to mention: The Grapher runs fully independently of your data taking scripts, so even if you connect to your lab via a really slow internet connection, as long as the actual data taking script is running on a fast connection, your data-rate is not impacted at all. You still get the full speed as if you were sitting right next to your experiment. Only, the Grapher might be updating the plots a little less smoothly as it otherwise would.

Next

 

~ Markus Ansmann, Feb 20th, 2008