Recent changes to faq

faq modified by André Offringa

André Offringa — Thu, 02 Jun 2016 08:05:58 -0000

--- v35
+++ v36
@@ -10,7 +10,7 @@

 ### Does the AOFlagger work on data from any telescope?

-As far as I know, Yes, it does. The software has been tested and shown good results on LOFAR, MWA, WSRT, VLA, GMRT, ATCA and Parkes data. The only requirement is that the data is stored in the Casa Measurement Set format, or the SDFits format for single dish telescopes. If you have results for other telescopes, I'd love to hear about it.
+As far as I know, yes it does. The software has been tested and shown good results on [LOFAR](http://www.lofar.org/), [MWA](http://www.mwatelescope.org/), [WSRT](https://www.astron.nl/radio-observatory/public/public-0), [VLA](http://www.vla.nrao.edu/), [GMRT](http://gmrt.ncra.tifr.res.in/), [ATCA](https://www.narrabri.atnf.csiro.au/), [Parkes](http://www.parkes.atnf.csiro.au/) and [Arecibo 305m](https://www.naic.edu/) data. The only requirement is that the data is stored in the Casa Measurement Set format or the SDFits format for single dish telescopes. If you have results for other telescopes, I'd appreciate hearing about it!

 Note though, that the settings have been optimized for telescopes I have worked with, notably LOFAR and the MWA. Some of the settings might need to be changed to get good results. See the "How can I solve bad results?" question for more info.

faq modified by André Offringa

André Offringa — Thu, 30 Apr 2015 11:45:09 -0000

--- v34
+++ v35
@@ -16,7 +16,7 @@

 ### Can I build AOFlagger from the GIT repository?

-It's fine if you try (see the "[Code](https://sourceforge.net/p/aoflagger/code/)" tab on SourceForge), but I do not test the running development changes on different platforms until I publish a new release. That means that you might hit a compilation issue for your platform, or some bug because of insufficient testing. If compilation fails, please try the latest tarball before mailing me.
+You can try (see the "[Code](https://sourceforge.net/p/aoflagger/code/)" tab on SourceForge), but I do not test the running development changes on different platforms until I publish a new release. That means that you might hit a compilation issue for your platform, or some bug because of insufficient testing. If compilation fails, please try the latest tarball before mailing me.

 ### How can I solve bad results?

faq modified by André Offringa

André Offringa — Thu, 30 Apr 2015 08:01:54 -0000

--- v33
+++ v34
@@ -14,7 +14,7 @@

 Note though, that the settings have been optimized for telescopes I have worked with, notably LOFAR and the MWA. Some of the settings might need to be changed to get good results. See the "How can I solve bad results?" question for more info.

-### Can I build aoflagger from the GIT repository?
+### Can I build AOFlagger from the GIT repository?

 It's fine if you try (see the "[Code](https://sourceforge.net/p/aoflagger/code/)" tab on SourceForge), but I do not test the running development changes on different platforms until I publish a new release. That means that you might hit a compilation issue for your platform, or some bug because of insufficient testing. If compilation fails, please try the latest tarball before mailing me.

faq modified by André Offringa

André Offringa — Thu, 30 Apr 2015 08:01:34 -0000

--- v32
+++ v33
@@ -14,6 +14,9 @@

 Note though, that the settings have been optimized for telescopes I have worked with, notably LOFAR and the MWA. Some of the settings might need to be changed to get good results. See the "How can I solve bad results?" question for more info.

+### Can I build aoflagger from the GIT repository?
+
+It's fine if you try (see the "[Code](https://sourceforge.net/p/aoflagger/code/)" tab on SourceForge), but I do not test the running development changes on different platforms until I publish a new release. That means that you might hit a compilation issue for your platform, or some bug because of insufficient testing. If compilation fails, please try the latest tarball before mailing me.

 ### How can I solve bad results?

faq modified by André Offringa

André Offringa — Thu, 28 Aug 2014 15:23:02 -0000

--- v31
+++ v32
@@ -40,7 +40,7 @@

 Many artefacts with different causes look similar to RFI (i.e., stripes, noise). Often, the artefacts can be traced to calibration errors or unremoved strong off-axis sources. If you can't find RFI in your time-frequency images or spectral power plots (e.g. with rfigui or aoqplot), it is very unlikely that RFI is the cause of your artefacts.

-Of course, AOFlagger will only flag be effective on RFI that is detectable and not both broadband and continuous. Especially broadband in situ RFI (RFI generated by hardware near the telescope) can not be removed by flagging and might cause calibration failures and/or striping while imaging. Other techniques should be used in such cases.
+Of course, AOFlagger will only be effective on RFI that is detectable and not both broadband and continuous. Especially broadband in situ RFI (RFI generated by hardware near the telescope) can not be removed by flagging and might cause calibration failures and/or striping while imaging. Other techniques should be used in such cases.

 ### How do I export a strategy from RFI GUI to the 'aoflagger' console application?

faq modified by André Offringa

André Offringa — Thu, 28 Aug 2014 15:21:03 -0000

--- v30
+++ v31
@@ -56,34 +56,32 @@

 Since version 2.4, it will correctly flag each separate pointing and spectral window. Before that, it would treat the entire observation as a whole and not handle multiple fields very well. See the [changelog of version 2.4](changelog-2.4.0) for more info.

-### aoflagger uses too much memory
+### Why is aoflagger using so much memory?
 Here are some hints on memory usage of aoflagger:

-* First of all, the memory reported by top and similar *is not the actual memory usage*. It includes a so called "memory map", which is not actually used memory, but is hard disk space that is addressable as if it was memory. In most cases, this is the largest chunk reported by top. Therefore, as long as you don't get memory crashes or a lot of swapping, you should not worry about this huge number.
-
-* The casacore library uses internally a caching mechanism that should speed up reading of the data. I am not sure how the data is cached, but I noticed that the cache will slowly grow until all physical available memory is occupied. This looks like a memory leak, but memory usage should stop growing as soon as swapping is about to occur.
+* The memory reported by top and similar is not always accurate. It might include a "memory map", which is not actually used memory, but is hard disk space that is addressable as if it was memory. In most cases, this is the largest chunk reported by top. Therefore, as long as you don't get memory crashes or a lot of swapping, you should not worry about this huge number.

 * The minimal required memory is currently depending on three things: (1) the [reading mode] (direct, indirect or memory) ; (2) the number of threads being used (defaults to 4) ; and (3) the size of a single baseline. The system will approximately use <THREAD COUNT> x 4 x <BASELINE SIZE> bytes of memory in direct mode and <THREAD COUNT> x 2.5 x <BASELINE SIZE> bytes in indirect mode. The size of a single baseline can be calculated like 8 (bytes/complex float) x 4 (polarizations) x 256 (channels) x 6000 (time steps) (thus 50 MB per baseline in this case, 0.5 GB in total, a rather typical LOFAR example). Hence, if the system really runs out of memory / starts swapping, *decrease the number of threads* as explained in the command line help of the aoflagger executable and make sure to use the [indirect reader](reading mode).

 * The software has been carefully tested for memory leaks, and there should not be any serious ones in it. A typical desktop machine should be able to run the software without problems.

-* Don't run multiple aoflagger's on the same machine. Run them sequential, not in parallel.
+* Don't run multiple aoflagger's on the same machine in parallel: run them sequentially.

 ### rfigui shows flags from a different baseline
-RFIGui has two modi for running the flagger. If one wants to analyze how well the flagger is doing, it should be run without the for-each-baseline action and without adding a write flags action. This could be any strategy as the wizard would give you. In that case, it will only flag the strategy that are visible in the GUI. You would typically run it like this on a few baselines until you are satisfied with the results.
+RFIGui has two modi for running the flagger. If one wants to analyse how well the flagger is doing, it should be run without the for-each-baseline action and without adding a write flags action. This could be any strategy as the wizard would give you. In that case, it will only flag the strategy that are visible in the GUI. You would typically run it like this on a few baselines until you are satisfied with the results.

 If one adds a for-each-baseline and a write-flags action at the right places, the strategy is run on all baselines and the results are written to the measurement set. After all baselines are flagged, the flags of the last flagged baseline are shown on top of the currently loaded baseline (with yellow colour). It did flag the visible baseline too, but doesn't show those flags. You would only run it like this when you are satisfied with the flags. If you want to see the new flags, just reload any baseline and the flags will show up in purple.

-### 'aoflagger' does not seem to use CPU / does not progress
+### Why is 'aoflagger' not progressing / using any CPU?
 This probably means that reading the data takes a lot of time and the computing threads are awaiting new data. Reasons for slow IO can be:

-* You are trying to run multiple aoflagger's on the same machine. Run them sequential, not in parallel.
-* The machine is low on memory, causing the cache to be too small or swapping. See the question about memory usage to improve memory.
-* Too many users on the storage node (/hard disk users).
+* If you are running multiple aoflagger's on the same machine in parallel, the IO access becomes very inefficient. Therefore, always run them sequentially.
+* If your machine is low on memory, the cache can to be too small or swapping might occur. See the question about memory usage to improve memory.
+* If you are on a shared node, make sure the node is not over-used.

 Generally, the [indirect reader](reading mode) is much faster than the direct reading mode.

-### Crashes with a 'FLAG is readonly' message
+### How do I solve a 'FLAG is readonly' crash?
 The full message is:

     terminate called after throwing an instance of 'casa::TableInvOper'
@@ -100,7 +98,7 @@
 The mode called the indirect reading mode is described on the following page: [reading mode].

 ### What is the newest version of the RFI tools?
-The SourceForge git repository contains the latest version. This git repository should always compile, but have been tested less extensively than the tagged releases.
+The SourceForge git repository contains the latest version. This git repository should always compile, but its version has been tested less extensively than the tagged releases.

 Since the move to SourceForge, the LOFAR clusters do not contain the latest version anymore, so bug fixes will take time to get installed. If you need a fix, contact the LOFAR sysmans.

@@ -121,12 +119,12 @@
 * [A LOFAR RFI detection pipeline  and its first results](http://adsabs.harvard.edu/abs/2010arXiv1007.2089O), PoS, RFI2010, March 2010.
 * [A morphological algorithm for improving radio-frequency interference detection](http://adsabs.harvard.edu/abs/2012A%26A...539A..95O), A&A, Vol. 539, A95, March 2012.

-The following two paper show RFI results for LOFAR:
+The following two papers show RFI results for LOFAR:

 * [The LOFAR radio environment](http://www.aanda.org/articles/aa/abs/2013/01/aa20293-12/aa20293-12.html), A&A, Vol. 549, January 2013, A11.
 * [The brightness and spatial distributions of terrestrial radio sources](http://mnras.oxfordjournals.org/content/435/1/584), MNRAS, Vol. 435, 1, October 2013, pp. 584-596.

 If I come across a paper that compares/checks the aoflagger, I'll add it to the [list of comparisons](comparisons).

-### I still have questions / contact info
-I like receiving feedback in order to update this FAQ and get a grib on common problems. You can reach me on my e-mail address, <my-last-name>@gmail.com, and my last name is offringa.
+### How can I contact you if I have more questions or suggestions?
+I like receiving feedback in order to update this FAQ and get a grip on common problems. You can reach me on my e-mail address, <my-last-name>@gmail.com, and my last name is offringa.

faq modified by André Offringa

André Offringa — Thu, 28 Aug 2014 15:09:26 -0000

--- v29
+++ v30
@@ -36,17 +36,17 @@

 I have never seen an observation in which optimizing the settings did not work in the end, so don't give up. You can always mail me and hand over a small part of your observation to let me take a look at the problem.

+### How do I know if artefacts are RFI related?
+
+Many artefacts with different causes look similar to RFI (i.e., stripes, noise). Often, the artefacts can be traced to calibration errors or unremoved strong off-axis sources. If you can't find RFI in your time-frequency images or spectral power plots (e.g. with rfigui or aoqplot), it is very unlikely that RFI is the cause of your artefacts.
+
+Of course, AOFlagger will only flag be effective on RFI that is detectable and not both broadband and continuous. Especially broadband in situ RFI (RFI generated by hardware near the telescope) can not be removed by flagging and might cause calibration failures and/or striping while imaging. Other techniques should be used in such cases.
+
 ### How do I export a strategy from RFI GUI to the 'aoflagger' console application?

 Since AOFlagger 2.6, you can immediately save a strategy that works on a single baseline in the rfigui, and run it in the 'aoflagger' console application.

 For AOFlagger 2.5 and earlier: If you make a strategy in RFI gui, it will be a strategy that works on already-loaded data. To run the same strategy in the aoflagger console, you need to add tasks to load the data. Moreover, you will likely want to write the results back too, which a RFI gui strategy normally won't do either. So, you will need to add two tasks: i) a "save flags" action; and ii) a "load baseline" action. In your strategy, click the top node and click Add->Write flags task. After that, press the "FOB" button to encapsulate the whole strategy in a for-each-baseline action. Now, your strategy will load and process all data in an observation. You can run this either in the RFI Gui itself (which will now update the set) or save the strategy and run it from the 'aoflagger' console program.
-
-### I suspect a presence of RFI in my final reduced data
-
-That is sometimes possible, even after using aoflagger. However, note that many artefacts with different causes look like RFI in an image (i.e., stripes, noise). Many times the artefacts can be traced to calibration errors or unremoved strong off-axis sources. If you can't find RFI in your time-frequency images or spectral power plots (e.g. with rfigui or aoqplot), it is very unlikely that RFI is the cause of your artefacts.
-
-Keep in mind that the AOFlagger will only flag RFI that is (1) detectable and (2) not both broadband and continuous. Especially broadband in situ RFI (RFI generated by hardware near the telescope) can not be removed by flagging and might cause calibration failures and/or striping while imaging. Other techniques should be used in such cases.

 ### How does the aoflagger handle multiple bands in the same observation?

@@ -54,7 +54,7 @@

 ### How does the aoflagger handle multiple fields in the same observation?

-Since version 2.4, it will correctly flag each separate pointing and spectral window. Before that, it would treat the entire observation as a whole and not handle spectral windows very well. See the [changelog of version 2.4](changelog-2.4.0) for more info.
+Since version 2.4, it will correctly flag each separate pointing and spectral window. Before that, it would treat the entire observation as a whole and not handle multiple fields very well. See the [changelog of version 2.4](changelog-2.4.0) for more info.

 ### aoflagger uses too much memory
 Here are some hints on memory usage of aoflagger:

faq modified by André Offringa

André Offringa — Thu, 28 Aug 2014 15:03:07 -0000

--- v28
+++ v29
@@ -12,7 +12,7 @@

 As far as I know, Yes, it does. The software has been tested and shown good results on LOFAR, MWA, WSRT, VLA, GMRT, ATCA and Parkes data. The only requirement is that the data is stored in the Casa Measurement Set format, or the SDFits format for single dish telescopes. If you have results for other telescopes, I'd love to hear about it.

-Note though, that the settings have been optimized for telescopes I have worked with, notably LOFAR and the MWA. Some of the settings might need to be changed to get good results. See the "I get bad results, how te solve?" question for more info.
+Note though, that the settings have been optimized for telescopes I have worked with, notably LOFAR and the MWA. Some of the settings might need to be changed to get good results. See the "How can I solve bad results?" question for more info.

 ### How can I solve bad results?

faq modified by André Offringa

André Offringa — Thu, 26 Jun 2014 02:10:23 -0000

--- v27
+++ v28
@@ -126,7 +126,7 @@
 * [The LOFAR radio environment](http://www.aanda.org/articles/aa/abs/2013/01/aa20293-12/aa20293-12.html), A&A, Vol. 549, January 2013, A11.
 * [The brightness and spatial distributions of terrestrial radio sources](http://mnras.oxfordjournals.org/content/435/1/584), MNRAS, Vol. 435, 1, October 2013, pp. 584-596.

-If I come across a paper that compares/checks the aoflagger, I'll add it to the [list of comparisons](comparisons)
+If I come across a paper that compares/checks the aoflagger, I'll add it to the [list of comparisons](comparisons).

 ### I still have questions / contact info
 I like receiving feedback in order to update this FAQ and get a grib on common problems. You can reach me on my e-mail address, <my-last-name>@gmail.com, and my last name is offringa.

faq modified by André Offringa

André Offringa — Thu, 26 Jun 2014 02:10:02 -0000

--- v26
+++ v27
@@ -126,5 +126,7 @@
 * [The LOFAR radio environment](http://www.aanda.org/articles/aa/abs/2013/01/aa20293-12/aa20293-12.html), A&A, Vol. 549, January 2013, A11.
 * [The brightness and spatial distributions of terrestrial radio sources](http://mnras.oxfordjournals.org/content/435/1/584), MNRAS, Vol. 435, 1, October 2013, pp. 584-596.

+If I come across a paper that compares/checks the aoflagger, I'll add it to the [list of comparisons](comparisons)
+
 ### I still have questions / contact info
 I like receiving feedback in order to update this FAQ and get a grib on common problems. You can reach me on my e-mail address, <my-last-name>@gmail.com, and my last name is offringa.