Menu ▾ ▴

faq

André Offringa

Frequency Asked Questions

Where can I find documentation for these tools?

Browse around this Wiki for general info. If you are reducing LOFAR data, you should at least read the LOFAR imaging cookbook. The NDPPP chapter contains also information about the "quality" tools that are included in the AOFlagger package. A tutorial on the GUI can be found here: RFI gui tutorial, and an example to optimize the default strategy here: Strategy optimization example. Of course, scanning through this FAQ might solve some of your questions. See the "How does the AOFlagger recognise RFI" faq item for some links to scientific papers about the flagger.

The API documentation can be found here: Application Programming Interface

Does the AOFlagger work on data from any telescope?

As far as I know, yes it does. The software has been tested and shown good results on LOFAR, MWA, WSRT, VLA, GMRT, ATCA, Parkes and Arecibo 305m data. The only requirement is that the data is stored in the Casa Measurement Set format or the SDFits format for single dish telescopes. If you have results for other telescopes, I'd appreciate hearing about it!

Note though, that the settings have been optimized for telescopes I have worked with, notably LOFAR and the MWA. Some of the settings might need to be changed to get good results. See the "How can I solve bad results?" question for more info.

Can I build AOFlagger from the GIT repository?

You can try (see the "Code" tab on SourceForge), but I do not test the running development changes on different platforms until I publish a new release. That means that you might hit a compilation issue for your platform, or some bug because of insufficient testing. If compilation fails, please try the latest tarball before mailing me.

How can I solve bad results?

Strong sources and different telescope properties can cause the default settings to fail in certain cases. The gui can help you a lot when trying to solve problems. Be sure to read the strategy optimization example. If you think aoflagger/rfigui is doing a bad job, I would suggest to experiment with the GUI to optimize the following settings:

  • Horizontal resolution decrease factor before the sliding window fit action
    LOFAR has a high time resolution (currently 1 sec by default). If your telescope / observation has a longer integration time or a very strong source far from the phase centre, this should be set to 1 instead of 3).

  • Try removing the time and frequency selection actions.

  • Kernel width (=kernel time size) of the sliding window fitting action
    Once again, this depends on integration time and source field.

  • Kernel height (=kernel frequency size) of the sliding window fitting action
    This depends on the channel resolution and frequency behaviour. Should be fine in most cases, but if your band pass is very steep (such as with WSRT), you might need to make it smaller. Another (actually better) solution is to correct for the band pass before running the aoflagger.

  • Change the sensitivity of the SumThreshold action.

  • Enlarge the number of iterations and initial threshold level (slower, but more stable)

I have never seen an observation in which optimizing the settings did not work in the end, so don't give up. You can always mail me and hand over a small part of your observation to let me take a look at the problem.

Many artefacts with different causes look similar to RFI (i.e., stripes, noise). Often, the artefacts can be traced to calibration errors or unremoved strong off-axis sources. If you can't find RFI in your time-frequency images or spectral power plots (e.g. with rfigui or aoqplot), it is very unlikely that RFI is the cause of your artefacts.

Of course, AOFlagger will only be effective on RFI that is detectable and not both broadband and continuous. Especially broadband in situ RFI (RFI generated by hardware near the telescope) can not be removed by flagging and might cause calibration failures and/or striping while imaging. Other techniques should be used in such cases.

How do I export a strategy from RFI GUI to the 'aoflagger' console application?

Since AOFlagger 2.6, you can immediately save a strategy that works on a single baseline in the rfigui, and run it in the 'aoflagger' console application.

For AOFlagger 2.5 and earlier: If you make a strategy in RFI gui, it will be a strategy that works on already-loaded data. To run the same strategy in the aoflagger console, you need to add tasks to load the data. Moreover, you will likely want to write the results back too, which a RFI gui strategy normally won't do either. So, you will need to add two tasks: i) a "save flags" action; and ii) a "load baseline" action. In your strategy, click the top node and click Add->Write flags task. After that, press the "FOB" button to encapsulate the whole strategy in a for-each-baseline action. Now, your strategy will load and process all data in an observation. You can run this either in the RFI Gui itself (which will now update the set) or save the strategy and run it from the 'aoflagger' console program.

How does the aoflagger handle multiple bands in the same observation?

Since version 2.4, it will correctly flag each separate pointing and spectral window. See the changelog of version 2.4 for more info. Before that, different reading modes did different things; it was best to separate the bands into different measurement sets. LOFAR never stores multiple bands in one measurement set by default, but WSRT and the VLA do.

How does the aoflagger handle multiple fields in the same observation?

Since version 2.4, it will correctly flag each separate pointing and spectral window. Before that, it would treat the entire observation as a whole and not handle multiple fields very well. See the changelog of version 2.4 for more info.

Why is aoflagger using so much memory?

Here are some hints on memory usage of aoflagger:

  • The memory reported by top and similar is not always accurate. It might include a "memory map", which is not actually used memory, but is hard disk space that is addressable as if it was memory. In most cases, this is the largest chunk reported by top. Therefore, as long as you don't get memory crashes or a lot of swapping, you should not worry about this huge number.

  • The minimal required memory is currently depending on three things: (1) the [reading mode] (direct, indirect or memory) ; (2) the number of threads being used (defaults to 4) ; and (3) the size of a single baseline. The system will approximately use <THREAD COUNT> x 4 x <BASELINE SIZE> bytes of memory in direct mode and <THREAD COUNT> x 2.5 x <BASELINE SIZE> bytes in indirect mode. The size of a single baseline can be calculated like 8 (bytes/complex float) x 4 (polarizations) x 256 (channels) x 6000 (time steps) (thus 50 MB per baseline in this case, 0.5 GB in total, a rather typical LOFAR example). Hence, if the system really runs out of memory / starts swapping, decrease the number of threads as explained in the command line help of the aoflagger executable and make sure to use the indirect reader.

  • The software has been carefully tested for memory leaks, and there should not be any serious ones in it. A typical desktop machine should be able to run the software without problems.

  • Don't run multiple aoflagger's on the same machine in parallel: run them sequentially.

rfigui shows flags from a different baseline

RFIGui has two modi for running the flagger. If one wants to analyse how well the flagger is doing, it should be run without the for-each-baseline action and without adding a write flags action. This could be any strategy as the wizard would give you. In that case, it will only flag the strategy that are visible in the GUI. You would typically run it like this on a few baselines until you are satisfied with the results.

If one adds a for-each-baseline and a write-flags action at the right places, the strategy is run on all baselines and the results are written to the measurement set. After all baselines are flagged, the flags of the last flagged baseline are shown on top of the currently loaded baseline (with yellow colour). It did flag the visible baseline too, but doesn't show those flags. You would only run it like this when you are satisfied with the flags. If you want to see the new flags, just reload any baseline and the flags will show up in purple.

Why is 'aoflagger' not progressing / using any CPU?

This probably means that reading the data takes a lot of time and the computing threads are awaiting new data. Reasons for slow IO can be:

  • If you are running multiple aoflagger's on the same machine in parallel, the IO access becomes very inefficient. Therefore, always run them sequentially.
  • If your machine is low on memory, the cache can to be too small or swapping might occur. See the question about memory usage to improve memory.
  • If you are on a shared node, make sure the node is not over-used.

Generally, the indirect reader is much faster than the direct reading mode.

How do I solve a 'FLAG is readonly' crash?

The full message is:

terminate called after throwing an instance of 'casa::TableInvOper'
what():  Invalid Table operation: FLAG is readonly (use the ROxxxColumn class)
Abort

This means you are trying to flag raw LOFAR data in which the LofarStMan storage manager is used for the flag column. This storage manager is read only, and the aoflagger executable can not update these flags. The LOFAR imaging cookbook will tell you on changes, but the current way to solve this on the LOFAR cluster is:

makeFLAGwritable &lt;MyObservation.MS&gt;

This will take a few seconds to rewrite the flag column, after which you can run the aoflagger executable.

What are the reading modes, and how do they affect performance?

The mode called the indirect reading mode is described on the following page: [reading mode].

What is the newest version of the RFI tools?

The SourceForge git repository contains the latest version. This git repository should always compile, but its version has been tested less extensively than the tagged releases.

Since the move to SourceForge, the LOFAR clusters do not contain the latest version anymore, so bug fixes will take time to get installed. If you need a fix, contact the LOFAR sysmans.

How can I transfer old strategies to newer versions of the tool?

Newer versions can not always read strategies which are created by older tools. Sometimes, the changes are small and can be easily solved, but sometimes it is better to recreate the strategies. You can try changing the version of the strategy file with a text editor and load it with aoflagger (aoflagger -strategy old_strategy.rfis). It will probably reply with an exception about which action is causing the incompatibility. By comparing that part of the strategy with a newer strategy, you might be able to solve the problem.

Does the aoflagger executable store e.g. the statistics in a log?

The aoflagger parameter '-v' can be used to give more feedback during flagging, including some statistics. The standard unix program 'tee' can be used to save the output to file while at the same time monitoring the standard output. Have a look at the [aoqplot] tool to plot statistics.

How does the AOFlagger recognise RFI?

It estimates the astronomical signal and searches for statistical abnormalities that look like lines in the time/frequency plots. After initial detection, it applies a morphological algorithm to find further likely contaminated samples. It has several tricks to enhance the speed. See the next question for literature

Where can I find more literature?

The following refereed papers describe parts of the detection technique in detail:

The following two papers show RFI results for LOFAR:

If I come across a paper that compares/checks the aoflagger, I'll add it to the list of comparisons.

How can I contact you if I have more questions or suggestions?

I like receiving feedback in order to update this FAQ and get a grip on common problems. You can reach me on my e-mail address, <my-last-name>@gmail.com, and my last name is offringa.

Related

Wiki: Home
Wiki: changelog-2.4.0
Wiki: comparisons
Wiki: reading mode

Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.