Sherpa
This release adds support for the NULISA Inflammation AQ assay. It also restores the changes in releases 17.4 and 17.4.1, which had been backed out in release 17.4.2, since the problematic communication with Basecamp turned out to be a Basecamp issue and not a Sherpa issue.
Sherpa
This release backs out changes from releases 17.4 and 17.4.1 that apparently made communication with Basecamp problematic.
Sherpa
This release fixes a bug in the previous release that prevented batch results documents from being uploaded.
Sherpa
In addition to incorporating internal changes that should not be apparent to the user, this release improves the performance of the Import Project page.
Sherpa
This release supports the NULISA CNS assay.i
Sherpa
This release fixes a problem where if using Safari the user was unable to select a project.
Sherpa
In this release, in the Select Project page, the PI of each listed project is shown. Both the Project column and the PI column are sortable (click on the little black triangles) and filterable (type a prefix in the text box and hit tab).
This release also includes support for the NULISA Inflammation assay.
Sherpa
This release fixes a problem wherein Sherpa could not successfully import projects from Basecamp with PIs that were hitherto unknown to SDM.
Sherpa
This release simply contains improvements to internal monitoring functionality that are invisible to users.
Sherpa
This release simply contains improvements to internal monitoring functionality that are invisible to users.
Sherpa
This release simply contains improvements to internal monitoring functionality that are invisible to users.
Sherpa
In this release, more selections from the left navigation bar result in the appearance of the "busy" modal dialog (a spinning wheel), indicating that Sherpa is actively processing the request. Other improvements should have no visible impact on functionality.
Sherpa
This release adds support for four new assays (Human Luminex 115plex-HC, Human Luminex 48plex-HC, Mouse Luminex 48plex-HC, and Mouse Luminex 64plex-HC) that are Luminex data processed by HIMConnect. In this release, only Human Luminex 115plex-HC is active. The other assays will be inactive until further notice.
Sherpa
A bug in Sherpa, causing the inability to upload files such as ‘Jessica Report H-80-8 Panel-2 01-23-2025.xlsm’ as assay ‘Human Luminex 62-63 plex’, has been fixed. There should be no other visible impact on functionality.
Sherpa
An obscure bug in Sherpa has been fixed. There should be no visible impact on functionality.
Sherpa
Sherpa has undergone some minor internal changes. There should be no visible impact on functionality.
Stanford Data Miner has been rehosted on a new hardware platform and a new software platform. We expect no functional differences from the previous releases of Sherpa and JasperServer, but have observed occasional issues with Edit Valid Values in Sherpa. Please report any problems you encounter to CytoAnalytics Technical Support (support@cytoanalytics.com).
Sherpa
Sherpa should now be able to communicate with Basecamp again, after their recent changes.
Sherpa
Sherpa should now be able to communicate with Basecamp again, after their recent changes. This release also contains various other minor changes that should have no impact on functionality.
Sherpa
Sherpa should now be able to communicate with Basecamp again, after their recent changes.
Sherpa
The deferred loader has been slightly rearchitected to remove its unnecessary dependence on an outdated third-party software component; there should be no observable difference in its behavior. The performance of editing valid values has been improved.
Sherpa
Sherpa should now be able to handle raw data files (zip files) prepared on Google Drive under a specific unusual set of circumstances. A side effect of this change is that files in zip files embedded within raw data files may not be available for mapping.
Sherpa
One (at least) side effect of dealing with the Basecamp issue addressed in Release 16.2.5 should now be addressed.
Sherpa
Sherpa should now be able to communicate with Basecamp again, after their recent changes.
Sherpa
Sherpa's XSD files have been updated to support the most recent failing WSP parse.
Sherpa
Sherpa now uses TLS 1.2 to communicate with Basecamp in the remaining contexts.
Sherpa
Sherpa now uses TLS 1.2 to communicate with Basecamp.
Sherpa
Rows with "Normalization" in the cell where an aliquot ID is expected are now ignored by the custom assay parser.
Sherpa
Invalid values of minRange and gain in some "linear" transforms in FlowJo workspaces prevented Sherpa from uploading those workspaces. Those invalid values are now ignored, and it should now be possible for Sherpa to upload those FlowJo workspaces.
Sherpa
Document comments may now be up to 300 characters in length. The previous limit was 200 characters.
Sherpa
The “Add New” custom assay page now shows only custom assays, not all assays.
A custom assay may no longer be edited once documents have been uploaded for that assay.
It should now be possible to upload Olink batch results documents by using a custom assay defined with Leading Rows = 6, Leading Columns = 1, Analyte Row = 4, and Aliquot Column = 1.
Sherpa
On the Add Analytes page, incoming analyte values that are longer than 60 characters no longer have a portion of their content replaced by an ellipsis.
Sherpa
This release mainly features enhancements to batch results upload.
BATCH RESULTS UPLOAD
There may only be one readout for each unique combination of aliquot ID, well, analyte, and units in any batch results document. Sherpa has never checked for compliance with this requirement, and as a result some documents violating it have been uploaded, and inconsistent data has resulted. Beginning with this release, Sherpa will reject attempts to upload such documents, instead issuing error messages of the form “There are replicate samples with aliquot ID ‘018-277-PMA_Iono.fcs’.”
This might happen, for example, due to any of the following:
In addition to the above requirement, for the pCyTOF phospho assays, there may only be one readout for each unique combination of sample name, stim, well, analyte, and units, where the sample name and stim are extracted from the aliquot ID. If Sherpa encounters documents violating this requirement, it issues error messages of the form “There is replicate data for sample name ‘018-277’ and stim ‘PMA_Iono’.”
This might happen, for example, due to changes in upstream equipment (see below).
The second page in the batch results upload workflow—the one with the “Here’s some of the data in the document” table—has been redesigned. That table is slightly smaller, but it no longer has “wasted space” on the right side, and the values in the Sample column (now called Aliquot ID) should be easier to read. Moreover, for the pCyTOF phospho assays, a second table is now below it, showing the sample names and stims extracted from each aliquot ID. Since upstream equipment may change the way in which sample name and stim are packaged into an aliquot ID unbeknownst to Sherpa, you are encouraged to review this table carefully.
In the final batch results upload confirmation dialog (the one asking “Do you really want to upload the file with these settings?”), in addition to the document name, the assay name is now displayed.
Some batch results upload error messages now contain more information, which should make them easier to understand.
OLD FILE CLEANUP
Though it should be invisible to the user, we want to document here the daily clean-up of old files. Every day, the following files are deleted:
Sherpa
A new item, Search All Documents, has been added to the left nav bar. It is available whether or not a project has been selected. The Search All Documents page is similar to the Batch Summary tab of the Summarize Project Data page, except that it spans all projects, allows clicking on buttons in the ID column to download, is paginated, and has filterable and sortable columns.
To avoid confusion between the new Search All Documents page and the existing Manage Documents page, Manage Documents (which is specific to project) has been renamed as Manage Project Documents.
The names "Lot" and "Batch Kit" have been changed to "Kit Batch" throughout.
The document-level "Stim" has been deprecated, and will eventually disappear. It has been removed from the Batch Summary tab of the Summarize Project Data page, and does not appear on the Search All Documents page.
A bug which prevented the display of documents with zero samples on the Batch Summary tab of the Summarize Project Data page has been fixed.
Known issues: in Search All Documents, filtering does not immediately update the scroll footer at the bottom of the table.
PARALLEL COORDINATES
This release features a major change in the Parallel Coordinates feature. Briefly, the SDM View menu no longer contains a “Parallel Coordinates” menu item; instead, the SDM OLAP View page contains an icon button depicting two vertical parallel lines and signifying “Show Parallel Coordinates”.
The Parallel Coordinates page itself has some differences. The most significant is that it no longer necessarily represents data from a single batch results document. (For this reason, it no longer even contains a choice box by which a data set can be selected, nor does it display the name of the selected data set.) Instead, when the page appears in response to clicking on the icon button in the OLAP View page, it represents the data summarized in the OLAP View page at that time, including the results of any filtering operations the user has performed. When the page appears in response to clicking on the “Parallel Coordinates” link in a Basecamp message or email received as a result of a batch results document upload, however, its behavior is as before.
The link allowing the user to “Click here to see the OLAP view.” has also been removed.
In addition to the aforementioned most significant difference in the feature, there are minor usability improvements. The most obvious are that (1) if the user reorders the axes by dragging, the table columns are now reordered to match; and (2) operating the Keep Filtered and Remove Filtered buttons (formerly called Keep Selected and Remove Selected) no longer causes the page to lose any axis reordering or attribute selection the user has performed. (But note that operating the Restrict Data Set button DOES cause this.)
Other minor bugs are fixed as well.
Known Issues:
The OLAP View pages for Assay Plus Person views include data that has not been mapped (batch results to sample to person). The corresponding Parallel Coordinates pages do not. Consequently, users may observe inconsistencies between the two pages for Assay Plus Person views.
After operating the Keep Filtered and Remove Filtered buttons, some attributes may no longer have any values represented in the (filtered) data set. (For example, if an assay is filtered out, attributes representing analytes unique to it may become moot.) Such attributes will continue to appear in the Select Attributes window, and their axes will continue to appear in the parallel coordinates display. This may be confusing.
The page size controls at the bottom of the Select Attributes window do not work. They are vestigial, and should be removed.
The dimming of paths in the parallel coordinates display when the user is setting a filter may be performed inconsistently.
NEW PHENO/PHOSPHO ASSAY PAIR
To support a new workflow for pCyTOF (whole blood), and the concomitant changes in aliquot ID (sample name) format and analyte aliases, this release adds a new assay pair “pCyTOF (whole blood) v2 pheno” and “pCyTOF (whole blood) v2 phospho”. Sherpa accepts batch results for this new assay pair in the form of FlowJo 10 workspaces, as it did (and still does) for the original pCyTOF (whole blood) assay pair.
As usual, this release includes a variety of minor changes that should cause no distress to the user. Significant changes are described below.
Sherpa
In editing or adding a custom assay, it is now possible to associate a project with the assay. If a project is associated with a custom assay, then that assay is unavailable during batch results upload and analyte mapping unless that project is the selected project. If no project is associated with a custom assay, then behavior is as before: that assay is available during batch results upload and analyte mapping regardless of the selected project.
The “Import Project” page now lists only active Basecamp projects, and not Basecamp projects whose status is “archived" or “on hold”.
In previous releases, when importing Basecamp projects into Sherpa, it was possible to leave the database in an inconsistent state by changing the case of a PI’s name (e.g. Smith to SMITH). It should no longer be possible to wreak havoc in this way. In general, error reporting for importing projects should be more robust.
As usual, this release includes a variety of minor changes that should cause no distress to the user. Significant changes are described below.
Sherpa
The assay “Mouse Luminex 38 plex” has been renamed as “Mouse Luminex 38-39 plex”. NOTE: this renaming may invalidate JasperServer views involving the assay under its old name.
In previous releases, all analytes were shown during analyte mapping. Now, only analytes associated with active assays are shown.
Six types of data are derived during batch results upload (age band, HAI Day 28/Day00, HAI response, arcsinh stim - unstim, stim/unstim, and Z.log2). In previous releases, messages regarding the number of rows of derived data were of inconsistent formats. Their formats are now consistent across all types of data:
| Number of Rows | Message Format |
|---|---|
| zero | WARNING! 0 rows of <type> data were derived.</type> |
| one | 1 row of <type> data was derived.</type> |
| more than one | <n> rows of <type> data were derived.</type></n> |
JasperServer
The meaning of “shared documents” has changed, to instead reflect “featured data”. The HIMC Shared folder in the repository has been replaced by the Featured Data folder, initially containing one view of the Immunobiology of Aging (RC4) project. This view is visible to all users, including a new guest user (User ID “guest”, Password “guest”). All previously shared documents, other than RC4 documents, have been unshared. As additional project documents are shared, they can be made visible as Featured Data. Please consult CytoAnalytics Technical Support for guidance on this.
The repository folder structure under “PI Data” has changed. There are no more “Analysis” folders; the two standard read-only views formerly under “Analysis/Basic” have been moved to be directly within the containing PI folder, and the “Analysis/Custom” folder has been moved there as well. A similar change has been made to the folder structure under “HIMC Admin”.
Custom views last modified prior to January 1, 2016 have been removed.
Sherpa
“Self-service download” is implemented. All JasperServer users can now log in to Sherpa with their JasperServer credentials (JasperServer User ID for Sherpa Username and JasperServer Password for Sherpa Password). If they do, they will then have the ability to select any project with which they are associated, and, via the Manage Documents page, download documents associated with the selected project, as well as those documents’ raw data and raw data mappings (if any).
The one exception to the above statement about JasperServer credentials is this: the Sherpa Password of a “staff” user (a Sherpa user created by traditional means) will not be changed, even if there should be, or come to be, a JasperServer user with the same name. This is expected to be a rare circumstance.
Changes to JasperServer users (adding one, removing one, changing one's password) will be reflected in Sherpa within an hour.
In previous releases, the nav bar (the list of items on the left side of every page) was fixed. But some items in the nav bar required prior selection of a project through the Select Project page; if the user operated any of them before selecting a project, Sherpa displayed an error message. Now, those items are not displayed at all until the user selects a project. A side effect of this change is that the order of items in the nav bar has been altered, so that all items dependent on selection of a project follow all items not dependent on selection of a project.
Sherpa
This release fixes a bug which could cause an incorrect list of aliquots to be presented for control designation during batch results upload.
Sherpa
On the Manage Documents page, a “Download Raw Data Mapping” button now appears for any batch results document for which raw data IDs have been mapped to raw data file names. (See the release notes for Release 12.1 for an explanation of this kind of mapping, which is called Map Batch Results to Raw Data.) Clicking on this button will cause a download of a CSV file containing the raw data mapping information for that document. The CSV file may be opened by spreadsheet-processing programs, such as Microsoft Excel, or by text-processing programs, such as MacOS TextEdit.
JasperServer
If JasperServer experiences a catastrophic error, it should now respond with a more helpful error page, instead of a page containing an ugly stack trace.
Previously, if more than two minutes had elapsed as JasperServer was working on bringing up a view, JasperServer would respond as if it had experienced a catastrophic error. It should no longer do that.
Certain JasperServer queries should now complete more quickly.
Sherpa
This release fixes a bug in the last release that prevented "arcsinh stim - unstim" data from being computed for the "pCyTOF (PBMC) phospho" assay.
Sherpa
The assay pair "pCyTOF pheno/phospho" has been renamed as "pCyTOF (PBMC) pheno/phospho". The two assays now expect batch results documents from FlowJo version 10, not FlowJo version 9.
Sherpa
Since Release 12.1, when you have chosen to associate a raw file with a batch results document that is a Luminex assay document or a FlowJo workspace document, Sherpa has given you the option to automatically map batch results raw data IDs to raw data file names. (See the release notes for Release 12.1 for more details.) With this release, Sherpa also gives you that option for gene expression documents. If you choose that option for a gene expression document, and the raw file you associate is a zip file, then Sherpa will look within that zip file for a file called targets.csv and containing columns labeled SampleName and FileName; it will then map based on the values in those columns, using SampleName as the raw data ID (the aliquot ID) and FileName as the raw data file name.
The Stanford Data Miner system now runs under a newer version of Java.
Sherpa
GENE EXPRESSION
There is a new assay group, "Gene expression", and a new assay, "Affy PrimeView". Analytes for this assay have been predefined based on a sample batch results document. (This was necessary because we deem it unreasonable to expect the user to use Sherpa to define tens of thousands of analytes.)
Batch results documents for Affy PrimeView must be comma-separated value (CSV) files. This is indicated by a new annotation on the initial Upload Batch Results page, a sequence of three asterisks (***), meaning "the batch results file must be a CSV file".
DEFERRED LOADING OF DATA FOR LARGE BATCH RESULTS DOCUMENTS
Some batch results documents are so large that it is not reasonable to expect the Sherpa user to wait until processing of a request to upload them has completed. (This may be particularly true, for example, in the case of the "Gene expression" assay group.) Beginning with this release, processing of a request to upload a batch results document containing more than a certain amount of data will not complete when requested--though the document will be uploaded to the server at that time, the remainder of the processing (chiefly loading of the uploaded document's data into the database) will be deferred until later in the night. To indicate that this has happened, Sherpa will display this message, instead of the usual message giving the user the option to upload a raw file:
Because of the amount of data in the file '...', its processing has been deferred.
An upload message will be posted to Basecamp when processing completes.
Any OLAP views you requested to be listed in the Basecamp message will not be listed.
The user will receive no notification when and if processing completes, other than a message being posted to Basecamp in the case of success. If this does not happen within 24 hours, please contact CytoAnalytics Technical Support for help in resolving the issue.
The threshold data size beyond which processing of requests to upload batch results documents will not complete when requested is initially set to be 10 megabytes. Note that the "amount of data" used in the threshold determination does not necessarily bear any relation to the size of the batch results document itself; it is actually the size of a temporary file called sample_fact.dat generated during the uploading process.
The upload message posted to Basecamp when processing completes differs in two respects from the upload message in the usual (non-deferred) case:
DEFERRED UNLOADING OF DATA FOR LARGE BATCH RESULTS DOCUMENTS
The situation with batch results document deletion is similar. When the Sherpa user requests the deletion of a batch results document containing more than a certain amount of data, unloading of its data from the database will be deferred. Depending on whether or not the user also requests the deletion of the matching document from Basecamp, the displayed message will be either
Because of the amount of data in the file '...', its processing has been deferred.
The matching document will be deleted from Basecamp when processing completes.
or
Because of the amount of data in the file '...', its processing has been deferred.
Deletion of the matching document from Basecamp was not requested.
In the case of deletion, the "amount of data" is the number of rows for the batch results document in the sample_fact table. This threshold data size is 500,000 rows.
OTHER
Processing of uploadings and deletions deferred during a day begins at 2 a.m. Pacific the next morning: deletions first, and then uploadings.
Analyte aliases are case-insensitive: that is, for example, TNFA and TNFa are equivalent. Though we have never seen a problem due to this, the fact remains that at some point Sherpa may encounter a single batch results document that contains conflicting data for two or more such different spellings of the same analyte alias. Sherpa has never checked for this in the past--it has not needed to--but beginning with this release it will, for safety, reject any such batch results document with the message
There was a problem parsing the document. (Analyte alias '...' has multiple spellings.)
Please contact CytoAnalytics Technical Support at support@cytoanalytics.com.
Several aspects of the batch results uploading process have been significantly reworked to improve speed and reliability, including, but not limited to, unmapped analyte alias detection and "Back" button handling. We believe users should experience no problems due to the reworking, but, as always, please contact us immediately if you do.
Values of the fields "Analyte Panel" and "Analyte Generic Name" no longer appear on the "Edit Valid Values" page. If you need to modify one of these values, please contact CytoAnalytics Technical Support.
Sherpa
MAP BATCH RESULTS TO RAW DATA
Map Batch Results to Raw Data (MBRtRD) is now implemented, and works as follows. After associating a raw file that is a zip file with a batch results document, you may proceed to the MBRtRD page, select the document in the list on the left, and map any unmapped batch results raw data IDs (see below) in the list in the middle to raw data file names (see below) in the list on the right.
What is a raw data ID? For all assays except Luminex assays, it is the aliquot ID that appears elsewhere in Sherpa. For Luminex assays, it is the aliquot ID followed by the well in parentheses.
What is a raw data file name? A raw data file name is the name of a file within the raw file (which is a zip file) associated with the document. Since zip files may contain other zip files, raw data file name determination is done in a recursive manner, using “ > ” to indicate the containment relationship. For example, if the file a.zip contains the file b.zip which contains the file c.fcs, the raw data file name for c.fcs is indicated as “a > b > c.fcs”.
Raw data mapping is done in the context of a single document, unlike, for example, sample mapping, which is done in the context of a project. This means that it is not possible to select more than one document at a time. Moreover, unlike in the other pages that provide mapping, in which operating the filter results in the display of documents that match the filter plus any selected documents, here operating the filter results in the display of only documents that match the filter, and not the selected document (unless, of course, it matches the filter).
If there are no documents in the project with associated raw files that are zip files, the left list will be empty. If no document is selected, or if all raw data IDs for the selected document have been mapped, the middle list will be empty. If no document is selected, or if no acceptable target files are found in the selected document’s associated zip file (an unlikely event), the right list will be empty. (Target files are not “acceptable” if their names contain certain strings, or, depending on the assay, if their names do NOT contain certain strings. For example, for Luminex assays, target files are acceptable only if their names end in “.csv”. The strings determining acceptability are stored in the database, so that in the future they can be adapted as desired without the necessity of a new Sherpa release.)
For certain kinds of batch results documents, Sherpa can attempt to auto-map (automatically map) batch results raw data IDs to raw data file names after you associate a raw file that is a zip file with one of them. Before you associate a raw file, it presents you with the option to “Automatically map batch results to raw data”. If you choose that option, and the raw file you associate is a zip file, then it auto-maps for Luminex assay documents and for FlowJo workspace documents. For Luminex assay documents, it maps based on the well: for example, it would map “KNVIP2-V1DX3 (B12)” to “5-15-2012 FLU STUDY 18 P-3_P1_B12.csv”. For FlowJo workspace documents, it maps based on the entire aliquot ID: for example, it would map “021216-Mike-Study 28-2014-plate 1-885d5 control_01.FCS” to “021216-Mike-Study 28-2014-plate 1-885d5 control_01.FCS”.
Batch results documents uploaded prior to this release are available for MBRtRD. But even if they are candidates for auto-mapping, they have not been retroactively auto-mapped.
OTHER
Previously, if you chose a filter value in one of the “Map” pages, that filter value would be remembered when you returned to that (or any other) “Map” page. Now, going to a “Map” page causes the filter value to be blanked out.
All mapping tables on the View Mappings page now present a paginated view rather than a complete view. This should improve page-loading time for those tables.
The logic supporting association of raw files with batch results documents has been considerably reworked. Users should not notice any difference, except that any errors that occur should be reported more consistently.
The Associate Uploaded Raw Data window has been completely redesigned, to make its operation easier to understand. As one example of the redesign, the “Number of days to search back for raw files” feature has been removed, as it provided negligible benefit and occasional frustration—all raw files previously associated with batch results documents of the given document’s assay are now displayed, in ascending alphabetical order of raw file name.
Other minor changes should not be apparent to the user, or, if apparent, should be greeted with joy.
Sherpa
For custom assays, behavior of the Add Analytes page has changed: on selection of an Incoming Analyte Value, Sherpa now populates the Specific Name and Generic Name fields with the selected value, as defaults. You may edit them before saving, or not, as you choose.
Issues recently observed with respect to the number of days one has to search back to find previously associated raw files may have been resolved; it is difficult to say, because we haven't been able to reproduce those issues, but handling of raw files has been moderately improved.
When batch results documents were deleted, their associated raw files (if any) were not deleted. Now they will be. "Abandoned" raw files (that is, raw files remaining because of the aforementioned lack of deletion) have been removed from the system.
The Analyte Mapper and Add Analytes pages have been redesigned so as to fit better on more screens.
"Map Batch Results to Raw Data" (formerly "Map Aliquots to Raw Data") is still unimplemented.
Batch results document deletion (on the Manage Documents page) is now much faster. In the initial confirmation window, instead of being told anything about matching Basecamp documents, you are simply asked "Also delete matching document from Basecamp?" After batch results document deletion, as part of the summary, you are told the status of Basecamp document deletion.
As usual, there are other minor changes which should be unnoticeable by the user, or nearly so.
Sherpa
In adding batch results data to the database, Sherpa has always silently truncated data that is too long for its database field: aliquot IDs, for example. Beginning with this release, Sherpa will reject batch results documents containing data that is too long, instead responding with the message "Sherpa currently can't store this data in the database. Please contact CytoAnalytics Technical Support at support@cytoanalytics.com." If you see this message, please email CytoAnalytics Technical Support the batch results document you are trying to upload, and we will discuss how to proceed. (Proceeding may involve our increasing the length of the database field.)
Also with this release, the left nav bar contains a new item, "Map Aliquots to Raw Data". Though the nav bar item is there, this feature is not yet implemented.
Sherpa
Previously, the first column in the tables on the Manage Documents page contained a table row number. This was not particularly useful, and in some cases could be misleading. Beginning with this release, the first column in those tables contains the ID number of the document described in the row--the unique number that SDM uses internally to identify the document.
Sherpa
Previously, when the user asked to associate uploaded raw data from the Manage Documents page, if Sherpa found no raw files with which to populate the lower table (the one under the text "Number of days to search back for raw files"), it displayed the erroneous and consequently misleading message "No raw files have been uploaded with batch results in the last - day(s).". Beginning with this release, it displays in that case the accurate message "No raw files are associated with batch results documents that have been uploaded in the last - day(s).".
Previously, when the user asked to associate uploaded raw data from the Manage Documents page, the contents of the lower table were raw files associated with batch results documents of the same assay as the subject document. Beginning with this release, the assay restriction is loosened in the following way: if the assay is a member of one of the pheno/phospho pairs, (currently, that is, either "pCyTOF pheno", "pCyTOF phospho", "pCyTOF (whole blood) pheno", or "pCyTOF (whole blood) phospho"), then raw files associated with batch results documents of the other member of the pair are shown as well. This allows the user to stage a raw file for such an assay one time, and then associate it with both the pheno document and the phospho document.
Sherpa
This release should allow uploading FlowJo version 10.1 workspaces as CyTOF ICS v2 documents.
Sherpa
This release fixes a bug that prevented sample IDs over 50 characters in length from being mapped.
Sherpa
Sherpa will now refuse to allow mapping of analytes if there are more than a configurable number of analytes defined for the assay group, to prevent unacceptably long page load times. The number is currently set to 2,000. This should have no visible effect until gene expression is supported.
This release also fixes a bug that prevented some Excel batch results documents from being correctly uploaded.
Sherpa
Sherpa now supports the 'TCR-seq' assay. Its assay group is 'Single cell genomics'. Known analytes for this assay have been bootstrapped.
There are other minor changes in this release, most of which should not be visible to the user.
Sherpa
A bug that prevented saving cubes with Kit Batch has been fixed.
Sherpa
Empty rows in the Assay Only cubes are now suppressed.
Sherpa
It should now be impossible for Jasper users to modify the contents of Basic folders, unless they are among the very few that have the most powerful Jasper role. It should still be possible for HIMC administrators to modify the contents of all Custom folders, and for each individual user to modify the contents of his or her PI's Custom folder.
Other minor changes in this release should not be visible to the user.
Sherpa
On the Add Analytes page, Generic Name and Panel are now swapped; they are displayed in the order "Panel, Generic Name". This agrees with the order in which they appear when analyte names are displayed elsewhere in Sherpa.
Release 9.6 introduced the change, "Analyte aliases are now scoped to the assay, not globally." Unfortunately, it neglected to take into account the fact that the Add Analytes process would now have to allow existing analytes to be "added" in the sense that they were simply being associated with a new assay under some alias. This release fixes that bug.
Sherpa
This release supports uploading batch results for the CyTOF assay (renamed as "CyTOF phenotyping") as either Excel workbooks or FlowJo 10 workspaces. (The marking on the batch results upload page indicates that FlowJo 10 workspaces are required; but Excel workbooks are still accepted.) It also changes the unit "Freq of Parent" to "% of Parent" everywhere.
This release removes the concept of "document promotion". You may think of this in the following way: until now, each uploaded batch results document has by default been promoted to a state in which its data was visible in OLAP views; but you had the option (by unchecking the "Promote This Document Now?" checkbox) to leave it in a reviewable state instead. If a batch results document was in the reviewable state, you had the choice to make its data visible later--or not. That option and choice will no longer exist; each uploaded batch results document's data will now unconditionally and immediately become visible.
The checkboxes "Promote This Document Now?" (for batch results documents) and "Promote Related Batch Results Documents Now?" (for experiment metadata documents) no longer exist.
Please note that, whereas prior to this release batch results documents had to be demoted before they could be deleted, with this release all batch results documents are immediately deletable, since promotion and demotion are no more. Be very careful.
Sherpa
This release makes minor improvements to user-invisible diagnostic messages.
JasperServer
This release adds two features to the Parallel Coordinates view:
The controls over how the analyte value axes are displayed appear below the graphs and toward the right-hand side. The following controls are provided:
When the number of analytes that belong to a data set is over 400, a filtered sample of the data set is returned. In these circumstances, the "Restrict Data Set" button next to the data set selection drop-down is enabled, allowing the user to set the sampling filter parameters to one of two types:
The default is for 50 analytes with highest SD to be returned.
Sherpa
Sherpa has always supported a maximum of 99 batch results documents having the same project, timepoint, assay, plate, and extension, and a maximum of 99 metadata documents having the same project and extension. This release removes those limits. There is now effectively no maximum number of either of these two types of documents.
Sherpa
The FlowJo 10 parser now supports database-configurable mapping of phospho-epitope names. In this release, the 'pCyTOF (whole blood) phospho' assay is configured to map the new Helios CyTOF platform names to the old names.
Sherpa
On the Manage Documents page, it is now possible to download raw data files through the new "Download Raw Data" button that appears for each document with a corresponding raw data file. If the raw data file exceeds 100 MB in size, a warning dialog appears first.
Sherpa
For batch results upload, assay names are now grouped by assay group in a tree widget.
The "CyTOF ICS" assay, with batch results parsed from an Excel workbook, is renamed to "CyTOF ICS v1".
A new "CyTOF ICS v2" assay, with batch results parsed from a FlowJo 10 workspace, is available.
Analyte aliases are now scoped to the assay, not globally.
Various other changes should not be noticed by users.
Sherpa
While parsing FlowJo workspaces, this release no longer erroneously complains about analytes intentionally discarded by the user.
Sherpa
This release extends the class of FlowJo 10 workspaces recognized by Sherpa, specifically for pCyTOF (whole blood). As is often the case in Sherpa releases, there are various other minor changes, which should not be noticed by the user.
Sherpa
Stanford Data Miner is now hosted on a new machine. For technical reasons, this has necessitated a new release of Sherpa, in which the wording of one error message has been improved.
Sherpa
This release fixes an error that prevented Sherpa from computing stim/unstim when appropriate, in some cases. In addition, on batch results upload, Sherpa now displays the number of rows of derived arcsinh stim - unstim, stim/unstim, and Z.log2 data, even if that number is zero.
Sherpa
This release fixes an error that prevented Sherpa from communicating with JasperServer when necessary (for example, when projects for new PIs are imported).
Sherpa
The 'Luminex/MSD' assay group has been renamed to 'Immunoassay'.
In all OLAP schemata, 'All Kits' has been changed to 'Kit Batch'.
Two new assays have been created: "pCyTOF (whole blood) pheno' and 'pCyTOF (whole blood) phospho'. The intent is that a FlowJo v10 workspace for pCyTOF (whole blood) may be parsed with respect to these two assays.
As is often the case in Sherpa releases, there are various other minor changes, which should not be visible to the user.
Sherpa
Differently formatted spreadsheet row headers, perhaps caused by a FlowJo change, are now allowed. (These row headers have a string of the form _ddd at the end, before the .fcs.)
Parsing of FlowJo v9 XML workspaces should now allow for more variability in the content of the workspaces.
As is often the case in Sherpa releases, there are various other minor changes, which should not be visible to the user.
Sherpa
This release adds an admin user with special privileges. It also deletes some spurious databases that should never have been there. The changes should be invisible to users.
Sherpa
The assay 'Lyoplate 1' has been deactivated.
Parsing of assay 'Lyoplate-HIPC' workbooks has been modified to adapt to the new plate layout.
Sherpa
Each section name in the Manage Documents page now includes the number of documents in the named section.
The "View All Mappings" page is now called "View Mappings", and requires that a project be selected before it can be displayed. It now has multiple tabs; the previous content is in the tab called "Metadata Values", but is now limited to mappings for the selected project. New tabs show Batch Results to Samples mappings, Samples to Persons mappings, Aliquots to Stims mappings, and Analyte Aliases. All except "Analyte Aliases" are limited to mappings for the selected project. Historical data is available for "Metadata Values" and "Analyte Aliases"; data for the other tabs will only be available going forward. Overall performance of the page has been improved. The page now also makes better use of available screen real estate, and allows for horizontal scrolling of content that is too big for its containing cell.
The "Share Documents" page now makes better use of available screen real estate, and it displays the spinning beachball icon in response to operation of the OK and Cancel buttons.
Performance of several pages has been somewhat improved.
It should now be impossible to enter a Comment or a Protocol on the batch results upload information page that is longer than the database allows. (This was already the case for Plate.)
As is often the case in Sherpa releases, there are various other minor changes, which should not be visible to the user.
Sherpa
This release fixes a bug exposed by the last release, which prevented the data in shared documents from being seen.
Sherpa
This release fixes two bugs. The first resulted in a stack trace being displayed for certain kinds of unexpected spreadsheet input. The second prevented "shared" documents from being truly shared across all PIs.
Sherpa
This release increases the transaction timeout from 20 minutes to 50 minutes.
Sherpa
This release fixes a bug introduced in the last release. The bug prevents, for example, the definition of new custom assays.
Sherpa
The "Edit Valid Values" option is restored to the left nav bar.
The custom assay parser (currently used for assays HAI, Sort results, Phenotyping-1, and Study 15 2007) has been changed to process ONLY the first worksheet in the workbook, and ignore all other worksheets.
Sherpa
This release simply adds pERK to the list of valid proteins (pSTAT1, pSTAT3, pSTAT5).
Sherpa
The "Edit Valid Values" option is removed from the left nav bar, pending fixes for issues with its functionality.
A bug that caused all custom assays to lose all their analytes when any custom assay was edited has been fixed.
An error that could result in the stack trace error page being displayed on uploading an invalid Luminex batch results document is now fixed.
Release notes for two earlier releases (Release 9.1 and Release 9.3) referred to data migration efforts that had not been undertaken. Those data migration efforts are now completed. Please refer to the release notes for those releases for more detail. (There is a similar reference in the release notes for Release 8.1; but that reference was rendered moot by subsequent design decisions.)
As is often the case in Sherpa releases, there are various other minor changes, which should not be visible to the user.
JasperServer
This release institutes a two-minute timeout on OLAP queries.
Sherpa
Release 9.3 incorrectly applied aliquot ID cleansing to the computation of outliers. Release 9.3.1 corrects that error.
Sherpa
A new "Download" button on the Manage Documents page allows downloading of batch results and experiment metadata documents.
This release adds to the checkered history of aliquot ID cleansing. Release 8.1 cleansed aliquot IDs on batch results document upload, instead of cleansing them "on the fly" in the two cases where that was needed: namely, (1) when they are displayed in the parallel coordinates representation and (2) when they are compared to compute post-upload metrics like stim/unstim. This was later deemed harmful, and Release 9. 1 rolled back that change. Unfortunately, Release 9.1 only restored the "on the fly" cleansing in the first of the two cases. Consequently, it broke post-upload metric computation. This release fixes post-upload metric computation. Note that the data migration effort to fix post-upload metrics that were computed in the interim has not been undertaken.
Other minor bug fixes and enhancements, in most cases not apparent to the user, have been included in this release.
JasperServer
The column headers of exported parallel coordinates data have been corrected. Further, on browsers that support the feature, a file name for the CSV file is suggested.
Sherpa
Access to Basecamp from the production server is unchanged; but access to Basecamp from test servers is now read-only, unless it occurs in the context of one of a configurable set of test projects.
Previously, if a project selected in Sherpa could no longer be found in Basecamp, results were unpredictable. Now, Sherpa is better-behaved if this occurs--in some (not necessarily all) cases.
The CyTOF parsing configuration has been changed to support the introduction of a new level (Cells) into the gating hierarchy.
All parsing configurations have been changed in response to the fixing of a long-standing bug having to do with validation of the content of workbook cells. There should be no change in visible behavior as a result of this.
Some diagnostic messages have been improved.
Sherpa
Release 9.2 introduced a bug: mappings invisible to the user because of selected-document filtering were in fact applied. This release fixes that bug.
Sherpa
A mapping bug, in which the presence of certain special characters might result in the exposure of a stack trace featuring a NullPointerException, has been fixed.
Sherpa
"Remove artifacts" now removes .txt if the string begins with a number followed by a colon. Previously it only removed _cells_found.txt in that case. It also now removes phospho artifacts (a trailing underscore, letter, and two digits).
In mappers where the selection of documents is allowed (in the left-hand column), any requested mapping is now applied only with respect to the selected documents, not all documents in the current project. A side effect of this change is that all project documents now appear in the left-hand column, not only those with unmapped aliquots.
Minor bugs in aliquot-to-stim mapping, sample-to-person mapping, and project data summary are fixed.
Mapping performance is improved.
Sherpa
The aliquot ID "cleansing" introduced in Release 8.1 had harmful side effects and has been rolled back. Aliquot IDs are now persisted without modification; they are only "cleansed" on the fly when needed, as when they are displayed in the parallel coordinates representation. Note that the data migration effort to "un-cleanse" aliquot IDs that were persisted between Release 8.1 and Release 9.1 has not been undertaken.
Sherpa
There is now an "edit custom assays" GUI.
There have been minor improvements to the "edit valid values" GUI.
Relevant ticket numbers are in parentheses at the beginning of each note.
Sherpa
(355) Empty spreadsheet cells encountered during metadata mapping have never presented a problem--the corresponding value has simply been left undefined. But it turns out that spreadsheet cells that are not empty, but that contain only spaces, have presented a problem. This release fixes that problem: such cells are now treated as empty cells have always been treated.
Relevant ticket numbers are in parentheses at the beginning of each note.
JasperServer
(333) JasperServer can now display an interactive parallel coordinates representation of the data in a batch results document. To access this representation, either select the Parallel Coordinates menu item from the View menu, or follow the indicated link in the posted Basecamp message.
Some JasperServer log messages that are considered unhelpful are now suppressed.
Sherpa
(330) Sherpa can now upload FlowJo version 3.0 workspaces for pCyTOF assays. (Version 3.0 workspaces are used by FlowJo version 9.7.2.) You must upload the pheno and phospho assay components separately, first with the assay choice "pCyTOF pheno" and second with the assay choice "pCyTOF phospho".
Sherpa now properly distinguishes between old-style and new-style (plate-based) phospho-flow workbooks.
Sherpa now includes a "Share Documents" page. You may share or unshare documents, individually or as a group. You may filter the presented list by Project, PI, shared/unshared status, and run date.
Other
To better support the parallel coordinates representation, Sherpa now "cleanses" aliquot IDs on BR document upload for many assays. Note that the data migration effort to cleanse existing aliquot IDs (that is, those from documents uploaded by earlier releases) has not been undertaken.
Due to a current limitation in the parallel coordinates representation, cleansed aliquot IDs that have the syntax of numbers are considered therein to be numeric attributes, not categorical attributes.
Various other minor bug fixes, cosmetic changes, and logging improvements are included.
Relevant ticket numbers are in parentheses at the beginning of each note.
Sherpa
(332) Calculation of Zlog2 for Luminex data upon upload. Values also calculated for previously uploaded data.
Fixed drag and drop during mapping using Firefox.
Edit valid values/Add new; project now defaults to currently selected project.
Relevant ticket numbers are in parentheses at the beginning of each note.
OLAP
(280) Automatic creation of OLAP users: Jasperserver/OLAP users are automatically created when a project is imported. OLAP users are created for all (non-HIMC) users associated with a project in Basecamp. User names are the part of the email address before the @ (SUNid for Stanford users). Default password is included in the email to users when data is uploaded. If users are added to a lab in Basecamp after the initial project is imported into Sherpa, OLAP users will be created for the newly added lab members when a new project is imported. Limitations: There is no auto-delete of OLAP users when they are deleted from Basecamp. There is no way other than project import to automatically create new OLAP users.
(328) Changes to age bands and ages: Age bands have been changed to be non-overlapping e.g. 60-64, 65-69. The last age band is "90 or older." The maximum age recorded in the sample table is "90 or older." This is for HIPAA compliance.
Sherpa
(329) Whole blood phospho-flow now supports a Non-BT worksheet.
Relevant ticket numbers are in parentheses at the beginning of each note.
OLAP
(258) Performance improvements in OLAP cubes: These improvements are based on a combination of index changes to the sample_fact table, and to the creation of PI-specific “source”/document tables in the PI schemas. Sherpa creates these latter tables on the creation of a new PI, and inserts into the appropriate table when a new document is loaded.
Sherpa General Workflow
(297) Performance improvements to Upload Batch Results workflow.
(301) Manage Documents: Sherpa deletes Basecamp message and file when deleting a batch results document. Limitations: (a) If multiple files exist with the same message title, Sherpa does not delete any of the messages. Normally, there will not be multiple files with the same message title. (b) This functionality does not apply to metadata documents.
(311) Upload Batch Results, Attach Raw File: A button for “Skip this step” has been added.
(317) Support for the staging of raw files: Following a specified workflow protocol, users can copy large files to the server using sftp. These files can be associated with an uploaded document from the Manage Documents page. Limitations: (a) Users must pre-register for this functionality with technical support staff. (b) The user must have access to a Unix/Linux shell (e.g. Mac OS X).
Sherpa “Administrative” Tasks
(298) Prototype generic assay: A “generic parser” has been written. This parser can be used to support new assays that match a “generic” workbook format. This generic format assumes that each row represents data collected on one aliquot, and that colums contain readouts for analytes. Consequently, there will be a leading column that contains aliquot IDs, and a header row that contains analyte names. Back-end scripts, executed by technical support staff, create the appropriate configuration data. When a user uploads a spreadsheet and selects the new assay, the user will be prompted to map analytes. This new parser is now used for HAI data. Limitations: (a) The current implementation supports only one unit of measure per assay. (b) Support for multiple sheets per workbook has not been tested.
(308) Derivation of HAI response, upon mapping batch results to samples: Computation of fold change (Day28/Day00) is performed only for samples uploaded on the same report. This behavior is based on the assumption that the Day28 and the Day00 sample for any particular person will be analyzed on the same plate, and that the plate will be uploaded as a batch results report.) Computation of “HAI response” (e.g. “2 strains with fold change >= 4”) is restricted to the last uploaded report for any particular person. This behavior is based on the assumption that if there is duplicate data for any particular person, the latest report is the best report.
(288) Changes in data promotion work flow:
The following changes are designed to support a default “auto-approve” work flow for both batch results and metadata. A workflow that requires explicit approval/promotion is also supported.
(a) Upload Experiment Metadata: Experiment metadata is initially loaded into the review database only. A “Promote Related Batch Results Documents Now” check box (defaults to checked/true) has been added. This changes the previous behavior, which would set all related batch results documents into an “under review” state. These documents would require promotion (one document at a time) for metadata changes to become client visible, which could be quite tedious. (If the “Promote Related Batch Results Document Now” is unchecked, this “old” behavior occurs.) Note that this “Promote…Now” can be a bit slow if many batch results files are already connected to uploaded metadata.
(b) Map Batch Results to Samples: Results of mapping operations automatically apply to both review and approved environments. (In an auto-approve workflow, this avoids the need to promote batch results documents to push metadata changes into the approved environment.) Note that the mapping of batch results to samples is the only mechanism by which person and sample metadata rows are promoted to the approved environment.