Recent changes to reading mode

reading mode modified by André Offringa

André Offringa — Fri, 19 May 2017 06:32:02 -0000

--- v4
+++ v5
@@ -1,8 +1,14 @@
 ### Reading modes

-The AOFlagger used to read baselines by scanning a file for the data relevant for the currently required baseline. The mode is called the direct baseline reader. Since AOFlagger used to be limited by IO seeking and not by cpu performance, a new approach was implemented in which a measurement set is written to a temporary location in a different order: the indirect baseline reader. The increase in speed on large sets is on the order of several factors, typically around 3 or 4 times.
+The AOFlagger has three reading modes:

-Later, a third reading mode was implemented that reads the entire set into memory, if this is possible. This mode is referred to as the memory reader, and is the fastest mode when enough memory is available. The memory reader is automatically selected when no reading mode is specified and there is enough memory available to load the set in memory.
+ * Direct reading (slowest)
+ * Indirect reading (fast, but requires disk space)
+ * Read in memory (even faster, but only possible for small sets)
+
+In the direct reading moding, AOFlagger reads baselines by scanning the entire file for the data relevant for the currently required baseline. This mode can be rather slow. In the indirect reading mode, a measurement set is written to a temporary location in a different order. On large sets the increase in speed is on the order of several factors.
+
+A third reading mode is implemented that reads the entire set into memory, if this is possible. This mode is referred to as the memory reader, and is the fastest mode when enough memory is available. The memory reader is automatically selected when no reading mode is specified **and** there is enough memory available to load the set in memory. Otherwise, the direct mode is selected. For very large sets, this can be slow, so then it is recommended to use the indirect reading mode.

 ### The indirect reading mode

reading mode modified by André Offringa

André Offringa — Thu, 18 May 2017 07:35:50 -0000

--- v3
+++ v4
@@ -10,7 +10,7 @@

     offringa@lce032:/data/scratch/offringa/temp$ aoflagger -indirect-read SB4.MS

-Please note that the current working directory will be used as a temporary storage location! Thus by running aoflagger like above, temporary files will be created in /data/scratch/offringa/temp that will take up an amount of space equal to the size of the measurement set. So, for LOFAR, do not run this in your home directory but always on the local hd's of the nodes. Using fast storage (e.g. SSD drives or RAID setups) as temporary location will speed up the aoflagger with indirect reading considerably.
+Please note that the current working directory will be used as a temporary storage location! Thus by running aoflagger like above, temporary files will be created in `/data/scratch/offringa/temp`, and these will take up a volume equal to the size of the measurement set. So, for LOFAR, do not run this in your home directory but always on the local hd's of the nodes. Using fast storage (e.g. SSD drives or RAID setups) as temporary location will speed up the aoflagger with indirect reading considerably.

 If you specify "-v" on the command line, you will see when the file is reordered. Here is an example output of aoflagger during initializing, when using the indirect reading mode:

reading mode modified by André Offringa

André Offringa — Fri, 28 Feb 2014 22:49:14 -0000

--- v2
+++ v3
@@ -8,7 +8,7 @@

 Because this approach uses a very large amount of disk space, it is not the default. The aoflagger will use the indirect baseline reader by specifying the "-indirect-read" parameter. In this case, the measurement set will be rewritten to a temporary location. Here is an example how to run the aoflagger in this mode:

-    offringa@lce032:/data/scratch/offringa/temp$ aoflagger -indirect-read -v -j 8 SB4.MS
+    offringa@lce032:/data/scratch/offringa/temp$ aoflagger -indirect-read SB4.MS

 Please note that the current working directory will be used as a temporary storage location! Thus by running aoflagger like above, temporary files will be created in /data/scratch/offringa/temp that will take up an amount of space equal to the size of the measurement set. So, for LOFAR, do not run this in your home directory but always on the local hd's of the nodes. Using fast storage (e.g. SSD drives or RAID setups) as temporary location will speed up the aoflagger with indirect reading considerably.

reading mode modified by André Offringa

André Offringa — Tue, 02 Apr 2013 14:16:11 -0000

--- v1
+++ v2
@@ -12,21 +12,21 @@

 Please note that the current working directory will be used as a temporary storage location! Thus by running aoflagger like above, temporary files will be created in /data/scratch/offringa/temp that will take up an amount of space equal to the size of the measurement set. So, for LOFAR, do not run this in your home directory but always on the local hd's of the nodes. Using fast storage (e.g. SSD drives or RAID setups) as temporary location will speed up the aoflagger with indirect reading considerably.

-Here is an example output of aoflagger during initializing, when using the indirect reading mode:
+If you specify "-v" on the command line, you will see when the file is reordered. Here is an example output of aoflagger during initializing, when using the indirect reading mode:

     [..]
-    Initializing ms cache data...
-    Unique time stamps: 379
-    Bands: 1
-    Number of parts: 1
-    0% : +-+-+-Strategy...
-    0% : +-+-+-+-strategy...
-    0% : +-+-+-+-+-For each baseline...
-    Estimate of memory each thread will use: 11 MB.
-    0% : +-+-+-+-+-+-Initializing...
-    Requesting 9 x 105 x 1109 x 4 x 256 bytes of data
-    RW
+    0% : +-+-+-For each baseline...
+    Estimate of memory each thread will use: 1 MB.
+    Will process 91 baselines.
+    0% : +-+-+-+-Initializing...
+    Initializing observation times...
+    Opening temporary files.
+    Pre-allocating 16 MB...
+    Pre-allocating 2 MB...
+    Reordering data set...
+    Done reordering data set of 18 MB in 0.917 s (19.9788 MB/s)
+    [..]

-aoflagger will keep reporting R's and W's as long as it is rewriting the set (it will only do this in verbose mode, i.e., with -v). Each R and W means 1 GB of data has been respectively read or written. Rewriting 40 GB MS's takes in the order of 10 to 20 minutes.
+Rewriting 40 GB MS's takes in the order of 10 to 20 minutes.

-In the case the AOFlagger might crash, please be sure to remove the temporary files (and send me a bug report if it is a bug).
+If the AOFlagger were to crash, please be sure to remove the temporary files (and send me a bug report if it is a bug).

WikiPage reading mode modified by André Offringa

André Offringa — Wed, 21 Nov 2012 23:18:37 -0000

### Reading modes The AOFlagger used to read baselines by scanning a file for the data relevant for the currently required baseline. The mode is called the direct baseline reader. Since AOFlagger used to be limited by IO seeking and not by cpu performance, a new approach was implemented in which a measurement set is written to a temporary location in a different order: the indirect baseline reader. The increase in speed on large sets is on the order of several factors, typically around 3 or 4 times. Later, a third reading mode was implemented that reads the entire set into memory, if this is possible. This mode is referred to as the memory reader, and is the fastest mode when enough memory is available. The memory reader is automatically selected when no reading mode is specified and there is enough memory available to load the set in memory. ### The indirect reading mode Because this approach uses a very large amount of disk space, it is not the default. The aoflagger will use the indirect baseline reader by specifying the "-indirect-read" parameter. In this case, the measurement set will be rewritten to a temporary location. Here is an example how to run the aoflagger in this mode: offringa@lce032:/data/scratch/offringa/temp$ aoflagger -indirect-read -v -j 8 SB4.MS Please note that the current working directory will be used as a temporary storage location! Thus by running aoflagger like above, temporary files will be created in /data/scratch/offringa/temp that will take up an amount of space equal to the size of the measurement set. So, for LOFAR, do not run this in your home directory but always on the local hd's of the nodes. Using fast storage (e.g. SSD drives or RAID setups) as temporary location will speed up the aoflagger with indirect reading considerably. Here is an example output of aoflagger during initializing, when using the indirect reading mode: [..] Initializing ms cache data... Unique time stamps: 379 Bands: 1 Number of parts: 1 0% : +-+-+-Strategy... 0% : +-+-+-+-strategy... 0% : +-+-+-+-+-For each baseline... Estimate of memory each thread will use: 11 MB. 0% : +-+-+-+-+-+-Initializing... Requesting 9 x 105 x 1109 x 4 x 256 bytes of data RW aoflagger will keep reporting R's and W's as long as it is rewriting the set (it will only do this in verbose mode, i.e., with -v). Each R and W means 1 GB of data has been respectively read or written. Rewriting 40 GB MS's takes in the order of 10 to 20 minutes. In the case the AOFlagger might crash, please be sure to remove the temporary files (and send me a bug report if it is a bug).