Data Import

Data Import

In PlateEditor, importing data from files is managed through the import form. It allows the selection of files from the local hard-disk drive, the configuration of the parsing and a mapping of the selected column to a given role.
This page describes all the functionalities of the import form, currently used for the attachement of result and definition files to a layout.

Input selection

The left panel (1) indicates the files that are currently selected and will be processed for parsing and mapping during the next steps.
File ordering in the table and suppression of items can be achieved using the controls located in the toolbar (2). From left to right: Reset, remove all items from the table; Delete, remove the selected area; Up, move the selected area up; Down, move the selected area down; Top, move the selected area to the top; Bottom, move the selected area to the bottom;
The right panel allow selection of the input and offers two options (3): File selection or manual input.
Select the files needed then click Next (6) to access the parsing configuration.

File selection
In this mode, files can be retrieved using the Browse... button (4a), allowing selection of one or multiple files from any accessible hard-disk location.
Files can also be dragged and dropped directly from the file explorer to the browser window, as long as the files are dropped in the area indicated on the form (4a). Dropping files anywhere else will do nothing.
After files have been selected, don't forget to click the Add button (5) to insert them in the Table (left panel), otherwise, the selected files will not be processed.

Manual input
In this mode, the Input selection panel will change to allow the user to copy and paste text data in the text area control (4b).
The text can also be manually entered. In this case, tabulations to separate data in different columns can be inserted using the Insert Tab button (using the tabulation key will move the focus away from the text area).
The manually entered text can also be recovered as a text file and saved for later use, using the Get as txt button (4b).
A specific name can be given to the manual input, using the filed provided. If left blank, the first few characters entered in the text area will be used.
When the manual input is ready, click the Add button to insert it to the Table (left panel). This will reset the field to prepare for a new input.

Parsing

The left panel (1) summarizes the input selected for parsing. In case inputs are missing or to add/remove items, go back to the previous form using the Back button located at the bottom of the form (5).
The right panel allows tuning of the parsing configuration for each input. Select the desired input on the right (1) to display the corresponding configuration, along with a preview of the parsed data (4c).

Parser
The parser to use for the input can be adjusted using the drop-down list located on top of the Parsing fieldset (2). The parser is initially pre-selected based on the file extension:
- .txt and .csv files will be mapped to the TXT/CSV parser (Raw text)
- .xls files will be mapped to the XLS parser (Excel files, old generation)
- .xlsx files will be mapped to the XLSX parser (Recent Excel files)
In case of parsing errors, the reason for failure will be displayed in the preview box (4c). Other parser can be tried: if the problem is due to a wrong extension, it might still work. Note that attempting to read a binary file (like .xls or .xlsx) with the TXT/CSV parser will work, but deliver only unreadable characters.

Parsing Options
This fieldset regroup options used to configure the parsing of the file (3).
a. No headers: tick this option when the file does not contain header rows. Arbitrary column names (Col_1, Col_2...) will be automatically generated (4c).
b. First Row: indicates at which row index the parsing should start. Previous rows will be ignored. Note that the index numbering follows the natural numbering and starts from 1 (not 0).
c. First Column: indicates at which column index the parsing should start. Previous columns will be ignored. Note that the index numbering follows the natural numbering and starts from 1, as for the rows.
d. Single column: tick this option to parse only one column from the file. The selected column is the column with the index indicated by the First Column field just above.
e. Skip empty rows: tick this option to exclude rows that are empty and contains no data. Row exclusion is performed AFTER the data passed through the first row/first column/single column filters set above.
f. Additional options specific to the parser
For the TXT/CSV parser, the delimiter used to separate the column can be specified, in case the auto-detection fails to recognize the correct character. To insert a tabulation, use the TAB button (4f).
For the XLS and XLSX parsers, the specific sheet to import should be selected here, using the drop-down list provided. Note that only one sheet can be imported per input. If several sheets needs to be imported for the same excel file, the same file can be added as many time as required in the input Table, at the first step of the Import form.

Parsing preview
A preview of the parsing will be displayed in the preview box, along with information about the number of rows and columns in the file (4c):
- Total available Rows/Columns: this is the count of rows/columns that were seen in the entire file. It is not affected by the general parsing options (3a-e), but will still be dependent on the delimiter selected in case of TXT/CSV parser (3f).
- Selected Rows/Columns: this is the count of rows/columns that passed through the filtering imposed by the general parsing options (3a-e) and will be effectively accessible for this file within PlateEditor.

The preview box can be shown or hidden by ticking the corresponding option in the Preview fieldset (4a).
As per default, the preview is limited to the first 20 selected rows of the file. If more lines are selected, a message will indicate that the preview is restricted to a subset. To see more rows, use the drop-down list available (4b).
It is important to keep in mind that this preview is held in memory, so displaying too many lines may result in tab crashing or browser stalling, especially for big files. For safety, it is recommended not to change this setting.

Mapping

In PlateEditor, the process of assigning a given role to specific columns is referred to as Mapping. It is handled by the form shown above, listing the available files to be mapped on the left panel (1) and the corresponding mapping table on the right (2).
In the mapping table (2), the column names for the selected file are referred to as Parameters and listed as rows under the header Name, while the specific roles that can be assigned are indicated as column headers (3a, 3b).
To assign a specific column to the desired role, the user can simply click on the corresponding cell of the mapping table to highlight it in blue.
Optional roles can be unticked or left blank, but required roles (marked with *) must have at least one cell highlighted. Some roles can be assigned to more than one column.
Available roles and specific restrictions applied are based on the type of input being mapped:

Result file
Roles available are: (3a)
- Well ID (REQUIRED, ONLY ONE): specify the column that contains the well id information (ie. A1, A2, etc... The A01, A02 format is also accepted). This role will be guessed automatically by PlateEditor based on the parameter name: the last column seen containing the keyword "well" (case insensitive) will be selected.
- Plate ID (OPTIONAL, ONLY ONE): specify the column that contains the plate id information. This role will be guessed automatically by PlateEditor based on the parameter name: the last column seen containing the keywords "plate" or "barcode" (case insensitive) will be selected.
- Import (REQUIRED, MULTIPLE): specify the column containing data to visualize in PlateEditor. For more information, see the [Results] page.
- Numeric (OPTIONAL, MULTIPLE): specify whether the parameter should be interpreted as a text or as a number. PlateEditor will automatically guess the data type by reading the first selected row of data, but the type can be manually adjusted here. Numeric parameters will be displayed as heatmaps, while text parameters will be displayed textually, without heatmaps. See the [Results] page for more information.

Definition file
Roles available are: (3b)
- Definition (REQUIRED, ONLY ONE): specify the column that should be used to read the names for this definition file. See the [Definitions] page for more information.
- Well ID (OPTIONAL, ONLY ONE): specify the column that contains the well id information (ie. A1, A2, etc... The A01, A02 format is also accepted). This role will be guessed automatically by PlateEditor based on the parameter name: the last column seen containing the keyword "well" (case insensitive) will be selected.
- Plate ID (OPTIONAL, ONLY ONE): specify the column that contains the plate id information. This role will be guessed automatically by PlateEditor based on the parameter name: the last column seen containing the keywords "plate" or "barcode" (case insensitive) will be selected.

Note that mapping of the required roles for all the files listed in (1) will be required to complete the mapping step.