Data Processing Outline

directory structure

<compound> is a name of investigated compound
<dataset> is a name of directory for a single crystal specimen - it should be the same as the name of directory containing raw frames
<scan> is a name of a certain scan for a crystal, like 'laser01' or 'powerscan01'
<path> is a path to a given file

HDF5 files, i.e. processed files

...../WORK/APS-YYYY-MM/<dataset>/h5

SORTAV and text files

...../WORK/APS-YYYY-MM/<dataset>/ratio

monochromatic data in HDF5 format

    ...../WORK/APS-YYYY-MM/YYYY-MM_mono

'lambda_curve' file

    ...../WORK/APS-YYYY-MM

data processing

0. in specified directory, for each compound convert reference monochromatic data to hdf5 format:

$ lu_fcf2mono -o <scan>__XXXK.h5 -e '[(h,k,l),(-h,-k,-l),...]' <path>/<compound>.fcf

1. in new terminal switch to bash:

% bash

2. run laue_util initialization script:

$ source laue_util.sh

3. each time a new crystal is mounted, create a new <dataset> directory:

$ cd WORK/APS-2012-06/
$ mkdir <dataset>
$ cd <dataset>
$ mkdir h5
$ cd h5

4. process data with following commands in h5 subdirectory:

mccd2hdf

$ lu_mccd2hdf -o <scan>__mccd.h5 -p /data/coppens_1202/<dataset>/ /data/coppens_1202/<dataset>/<scan>.log

output: <scan>__mccd.h5

hdf2spots
a) dark

$ lu_hdf2spots -o <scan>__spots.h5 <scan>__mccd.h5

output: <scan>__spots.h5

b) laser

$ lu_hdf2spots -o <scan>__spots.h5 <scan>__mccd.h5 -m kruskal -S blocks

output: <scan>__spots.h5

spots2expt
a) default

$ lu_spots2expt -o <scan>__expt.h5 -c <beam center> -d 65.0 <scan>__spots.h5

output: <scan>__expt.h5

b) with refinement of detector distance & beam-center

$ lu_spots2expt -o <scan>__expt.h5 -c <beam center> -d 65.0 -r yes <scan>__spots.h5

output: <scan>__expt.h5

c) 'laser' dataset

$ lu_spots2expt -o <scan>__expt.h5 -c <beam center> -d 65.0 -r yes -l yes <scan>__spots.h5

output: <scan>__expt.h5

at the current setup, these three steps can be performed together with appropriate bash scripts:

dark

$ bash ../../processing_scripts/dark_integration.sh <dataset> <scan>

laser

$ bash ../../processing_scripts/laser_integration.sh <dataset> <scan>

multidelay

$ bash ../../processing_scripts/multi_integration.sh <dataset> <scan>

5. obtaining unindexed (RAW) ratios

laser

    $ lu_spots2ratios -o <scan>__rawR.h5 <scan>__spots.h5

    $ Lu_ratios2averaging -o <scan>__aveR.h5 <scan>__rawR.h5

5a. RAW correlation plot visualization

$ lu__view_correlation_ratios <path>/<scan>__aveR.h5 <path>/<scan>__aveR.h5

6. orientation matrix search

lu__match_pairs <scan>__expt.h5 ../../2012-02-mono/<compound>__<temp>K.h5

output: <scan>__initial.h5

parameters to be changed in 'short' dataset (<20 phi angles)

... or in 'long' dataset (>20 phi angles)

Expt Clusters Manager always has to be set so as the cluster number in console was in between 30 and 40

When this is done, click 'Calculate';

after the program completes calculation (i.e. unfreezes), click 'Cluster'

There is a list of possible orientations, the first, fitting the most reflections, being the best choice. In higher symmetry system, a few alternatives, equivalently fit, may present themselves. Choose the first one.

Save the output.

output: <scan>__initial.h5

7. orientation matrix refinement

$ lu__refine_orientmatrix <scan>__expt.h5 ../../2012-02-mono/<compound>__<temp>K.h5 ../../14IDB_10892.mda.0001 <scan>__initial.h5

output: <scan>__expt_rot.h5

output: <scan>__expt_full.h5 if full optimization was used

7a. cell parameters / Euler angle control

$ lu__view_matrix_parameters <scan>__expt_rot.h5
$ lu__view_matrix_parameters <scan>__expt_full.h5

8. hkl indices assignment

$ lu__hkl_assign_monolc <scan>__expt.h5 ../../2012-02-mono/<compound>__<temp>K.h5 ../../14IDB_10892.mda.0001 <scan>__expt_rot.h5 <OR <scan>__expt_full.h5> <-P delay>

output: <scan>__assign_IU.h5 - for unrefined global orientation matrix
OR <scan>__assign_CU.h5 - for frame by frame refined orientation matrix
OR <scan>__assign_FU.h5 - for full refinement

-P delay option is used for 'multi-delay' datasets

Alternative, new, presumably more precise algorithm of assignment, is executed as:

$ lu__hkl_assign_monolc_sph <scan>__expt.h5 ../../2012-02-mono/<compound>__<temp>K.h5 ../../14IDB_10892.mda.0001 <scan>__expt_rot.h5 <OR <scan>__expt_full.h5>

output: <scan>__assign_sph_IU.h5 - for unrefined global orientation matrix
OR <scan>__assign_sph_CU.h5 - for frame by frame refined orientation matrix
OR <scan>__assign_sph_FU.h5 - for full refinement

9. SORTAV processing - in ratio subdirectory

just ratios for SORTAV

$ lu_indexing2ratios_V2 -r ../h5/<scan>__ratios.h5 -R <scan>__ratios.txt -o <scan>__ratios.hkl -n 10 -s 0.5 ../h5/<scan>__assign_FU.h5

automatic SORTAV run: WINGX SORTAV version

    ../../processing_scripts/run_windows_sortav.sh <scan>__assign_FU <scan> 9 0.5

automatic SORTAV run LINUX

$ bash ../../processing_scripts/run_sortav.sh <scan>__assign_FU <scan> 9 0.5

10. correlation plot visualization

If hkl unmerged,

$ lu__view_correlation_merged_ratios_phi <path>/<scan>__ratios.h5 <path>/<scan>__ratios.h5

If sortav merging done,

$ lu__view_correlation_merged_ratios <path>/<scan>__sortav.hkl <path>/<scan>__sortav.hkl

11. Optional 'sensitive' hkl retrieval:

$ $PYTHON /space-md2/lab/_WORK/python-bin/hkl-h5.py <scan>__ratios.h5 ~/Desktop/RhPNP_significant_positive_experimental_eta.txt '(h,k,l), (-h,-k,-l), (h, -k, l), (-h, k, -l)'

output: selected_hkl.lst