reading mode

Reading modes

The AOFlagger has three reading modes:

• Direct reading (slowest)
• Indirect reading (fast, but requires disk space)
• Read in memory (even faster, but only possible for small sets)

In the direct reading moding, AOFlagger reads baselines by scanning the entire file for the data relevant for the currently required baseline. This mode can be rather slow. In the indirect reading mode, a measurement set is written to a temporary location in a different order. On large sets the increase in speed is on the order of several factors.

A third reading mode is implemented that reads the entire set into memory, if this is possible. This mode is referred to as the memory reader, and is the fastest mode when enough memory is available. The memory reader is automatically selected when no reading mode is specified and there is enough memory available to load the set in memory. Otherwise, the direct mode is selected. For very large sets, this can be slow, so then it is recommended to use the indirect reading mode.

The indirect reading mode

Because this approach uses a very large amount of disk space, it is not the default. The aoflagger will use the indirect baseline reader by specifying the "-indirect-read" parameter. In this case, the measurement set will be rewritten to a temporary location. Here is an example how to run the aoflagger in this mode:

offringa@lce032:/data/scratch/offringa/temp$ aoflagger -indirect-read SB4.MS

Please note that the current working directory will be used as a temporary storage location! Thus by running aoflagger like above, temporary files will be created in /data/scratch/offringa/temp, and these will take up a volume equal to the size of the measurement set. So, for LOFAR, do not run this in your home directory but always on the local hd's of the nodes. Using fast storage (e.g. SSD drives or RAID setups) as temporary location will speed up the aoflagger with indirect reading considerably.

If you specify "-v" on the command line, you will see when the file is reordered. Here is an example output of aoflagger during initializing, when using the indirect reading mode:

[..]
0% : +-+-+-For each baseline...
Estimate of memory each thread will use: 1 MB.
Will process 91 baselines.
0% : +-+-+-+-Initializing...
Initializing observation times...
Opening temporary files.
Pre-allocating 16 MB...
Pre-allocating 2 MB...
Reordering data set...
Done reordering data set of 18 MB in 0.917 s (19.9788 MB/s)
[..]

Rewriting 40 GB MS's takes in the order of 10 to 20 minutes.

If the AOFlagger were to crash, please be sure to remove the temporary files (and send me a bug report if it is a bug).