#98 digifil not able to remove interchannel dispersion when fscrunching?

[Scott Ransom]

Hi All,

I was hoping based on the help info for digifil that you would be able to do incoherent dedispersion while subbanding/fscrunching (ala psrfits_subband). I'd like to do this for some filterbank files (and also output filterbank files). I can fscrunch just fine, but the "-K" option doesn't seem to remove the interchannel delays as I thought it would (if used in conjunction with "-D") from the help screen. Is that a bug? Or something that is not yet implemented?

If the latter, please consider this a feature request!

Thanks a bunch,

Scott

[Paul Demorest]

Hi Scott, from a quick look at the code I think dedispersion (-K option) should be applied before fscrunch (-f) if both options are used. I guess that is not what you are seeing?

[Scott Ransom]

Hi Paul,

It seems to run, but the resulting file is definitely not dedispersed properly (and, in fact, seems
to have extra artifacts etc).

My command lines were like:

digifil -D 71.02 -K -f 8 -o out.fil input.fil

Scott

On 4/5/22 1:38 PM, Paul Demorest wrote:

Hi Scott, from a quick look at the code I think dedispersion (-K option) should be applied before
fscrunch (-f) if both options are used. I guess that is not what you are seeing?

---

[bugs:#98] https://sourceforge.net/p/dspsr/bugs/98/ digifil not able to remove interchannel
dispersion when fscrunching?

Status: open
Group: version_1.0
Created: Tue Apr 05, 2022 04:49 PM UTC by Scott Ransom
Last Updated: Tue Apr 05, 2022 04:49 PM UTC
Owner: nobody

Hi All,

I was hoping based on the help info for digifil that you would be able to do incoherent dedispersion
while subbanding/fscrunching (ala psrfits_subband). I'd like to do this for some filterbank files
(and also output filterbank files). I can fscrunch just fine, but the "-K" option doesn't seem to
remove the interchannel delays as I thought it would (if used in conjunction with "-D") from the
help screen. Is that a bug? Or something that is not yet implemented?

If the latter, please consider this a feature request!

Thanks a bunch,

Scott

---

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/dspsr/bugs/98/
https://sourceforge.net/p/dspsr/bugs/98/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/
https://sourceforge.net/auth/subscriptions/

--
Scott M. Ransom Address: NRAO
Phone: (434) 296-0320 520 Edgemont Rd.
email: sransom@nrao.edu Charlottesville, VA 22903 USA
GPG Fingerprint: A40A 94F2 3F48 4136 3AC4 9598 92D5 25CB 22A6 7B65

[Paul Demorest]

Does -K work on its own in this case (ie, without also using -f)?

[Scott Ransom]

I'm not sure what should be expected, but I think so? It seems like the data now has a DM of 0.0.
However, there is a bunch of junk at the beginning of the output file (like the buffer had garbage
in it, or all zeros).

Maybe things are working with fscrunch and -K and I was just expecting something different. For
the subbanding that psrfits_subband does, the resulting file acts like a coherently dedispersed .fil
file with bigger channels. In other words, the subbands have not had their center-freq DM delays
removed. Only the inter-channel within-subband delays were removed.

I'll test to see if that is the case.

Scott

On 4/5/22 3:25 PM, Paul Demorest wrote:

Does -K work on its own in this case (ie, without also using -f)?

---

[bugs:#98] https://sourceforge.net/p/dspsr/bugs/98/ digifil not able to remove interchannel
dispersion when fscrunching?

Status: open
Group: version_1.0
Created: Tue Apr 05, 2022 04:49 PM UTC by Scott Ransom
Last Updated: Tue Apr 05, 2022 05:38 PM UTC
Owner: nobody

Hi All,

I was hoping based on the help info for digifil that you would be able to do incoherent dedispersion
while subbanding/fscrunching (ala psrfits_subband). I'd like to do this for some filterbank files
(and also output filterbank files). I can fscrunch just fine, but the "-K" option doesn't seem to
remove the interchannel delays as I thought it would (if used in conjunction with "-D") from the
help screen. Is that a bug? Or something that is not yet implemented?

If the latter, please consider this a feature request!

Thanks a bunch,

Scott

---

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/dspsr/bugs/98/
https://sourceforge.net/p/dspsr/bugs/98/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/
https://sourceforge.net/auth/subscriptions/

--
Scott M. Ransom Address: NRAO
Phone: (434) 296-0320 520 Edgemont Rd.
email: sransom@nrao.edu Charlottesville, VA 22903 USA
GPG Fingerprint: A40A 94F2 3F48 4136 3AC4 9598 92D5 25CB 22A6 7B65

[Scott Ransom]

And nope. With the fscrunching, it is even more messed up and definitely not correct.

S

On 4/5/22 3:38 PM, Scott Ransom wrote:

I'm not sure what should be expected, but I /think/ so? It seems like the data now has a DM of 0.0.
However, there is a bunch of junk at the beginning of the output file (like the buffer had garbage
in it, or all zeros).

Maybe things /are/ working with fscrunch and -K and I was just expecting something different. For
the subbanding that psrfits_subband does, the resulting file acts like a coherently dedispersed .fil
file with bigger channels. In other words, the subbands have not had their center-freq DM delays
removed. Only the inter-channel within-subband delays were removed.

I'll test to see if that is the case.

Scott

On 4/5/22 3:25 PM, Paul Demorest wrote:

Does -K work on its own in this case (ie, without also using -f)?

----------------------------------------------------------------------------------------------------

/[bugs:#98] <https://sourceforge.net/p/dspsr/bugs/98/> https://sourceforge.net/p/dspsr/bugs/98/
<https://sourceforge.net/p/dspsr/bugs/98/> digifil not able to remove interchannel
dispersion when fscrunching?/

/Status:/ open
/Group:/ version_1.0
/Created:/ Tue Apr 05, 2022 04:49 PM UTC by Scott Ransom
/Last Updated:/ Tue Apr 05, 2022 05:38 PM UTC
/Owner:/ nobody

Hi All,

I was hoping based on the help info for digifil that you would be able to do incoherent
dedispersion
while subbanding/fscrunching (ala psrfits_subband). I'd like to do this for some filterbank files
(and also output filterbank files). I can fscrunch just fine, but the "-K" option doesn't seem to
remove the interchannel delays as I thought it would (if used in conjunction with "-D") from the
help screen. Is that a bug? Or something that is not yet implemented?

If the latter, please consider this a feature request!

Thanks a bunch,

Scott

----------------------------------------------------------------------------------------------------

Sent from sourceforge.net because you indicated interest in
https://sourceforge.net/p/dspsr/bugs/98/ <https://sourceforge.net/p/dspsr/bugs/98/>
https://sourceforge.net/p/dspsr/bugs/98/ <https://sourceforge.net/p/dspsr/bugs/98/>

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/
<https://sourceforge.net/auth/subscriptions/>
https://sourceforge.net/auth/subscriptions/ <https://sourceforge.net/auth/subscriptions/>

--
Scott M. Ransom Address: NRAO
Phone: (434) 296-0320 520 Edgemont Rd.
email: sransom@nrao.edu Charlottesville, VA 22903 USA
GPG Fingerprint: A40A 94F2 3F48 4136 3AC4 9598 92D5 25CB 22A6 7B65

---

[bugs:#98] https://sourceforge.net/p/dspsr/bugs/98/ digifil not able to remove interchannel
dispersion when fscrunching?

Status: open
Group: version_1.0
Created: Tue Apr 05, 2022 04:49 PM UTC by Scott Ransom
Last Updated: Tue Apr 05, 2022 07:25 PM UTC
Owner: nobody

Hi All,

I was hoping based on the help info for digifil that you would be able to do incoherent dedispersion
while subbanding/fscrunching (ala psrfits_subband). I'd like to do this for some filterbank files
(and also output filterbank files). I can fscrunch just fine, but the "-K" option doesn't seem to
remove the interchannel delays as I thought it would (if used in conjunction with "-D") from the
help screen. Is that a bug? Or something that is not yet implemented?

If the latter, please consider this a feature request!

Thanks a bunch,

Scott

---

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/dspsr/bugs/98/
https://sourceforge.net/p/dspsr/bugs/98/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/
https://sourceforge.net/auth/subscriptions/

--
Scott M. Ransom Address: NRAO
Phone: (434) 296-0320 520 Edgemont Rd.
email: sransom@nrao.edu Charlottesville, VA 22903 USA
GPG Fingerprint: A40A 94F2 3F48 4136 3AC4 9598 92D5 25CB 22A6 7B65

[David McKenna]

Hi Scott & Paul,

I was looking into another issue I was having with DSPSR this week (ended up being due to the fact I was using an outdated version) and spotted this ticket. I thought I had this issue in late 2020, and isolated the cause to be some kind of error that occurs when dedispersing while data resampling/rescaling is enabled.

I tried to get a group of results together to show what I had seen previously, but have run into a large number of inconsistent results, where sometimes things break, but other times they don't, making me think there's some array indexing issues at play here.

Overall, the trends that I have noticed are that

• Rescaling with large resampling factors was the easiest way to trigger this issue, with nearly full control over the length of the block of noise coming from the rescaling interval (default is 10 seconds, and is controllable with -I for all but the last 2 broken commands)
• Dedispersion does cause this issue as well in limited circumstances, though it is typically only above a certain dispersion delay, and varies in a periodic with respect to the dispersion measure you are dedispersing at, never extending beyond the rescaling interval
• None of the commands with rescaling disabled caused the issue to appear

I've uploaded a zip file with a text file of all the commands I've mentioned here (too big for sourceforge: https://drive.google.com/file/d/1zzhpxC0f0ozflzoD100WxByqKhqp7PrC/view?usp=sharing ), alongside plots of the data. These outputs were generated with the current heads of the main branches of DSPSR (124a0bda) and PSRCHIVE (350d0ad) and some random data I observed this week. I've uploaded this data snippet here if you want to try and replicate what I've seen: https://drive.google.com/file/d/1t3Q_xuN3p0aQWV7JdIA4dJB4fPfs4qdP/view?usp=sharing

Cheers,
David

Apologies for the wall of commands here, but I've tested quite a few configurations and found a number that break. Here's a list of commands that I've found issues with:
~~~
digifil ./Crab_90s.fil -t 128 -b-32 -o ./Crab_90s_simult128.fil
digifil ./Crab_90s.fil -t 128 -D 56.75 -K -b-32 -o ./Crab_90s_simult128_dedisp56.75.fil
digifil ./Crab_90s.fil -t 128 -b-32 -f 2 -o ./Crab_90s_simult128_f2.fil
digifil ./Crab_90s_t128.fil -D 56.75 -K -b-32 -o ./Crab_90s_t128_dedisp56.75.fil <<different>>
digifil ./Crab_90s_t128.fil -D 56.75 -K -b-32 -t 2 -o ./Crab_90s_t128_t2_dedisp56.75.fil <<different>>
~~~</different></different>

And commands that appear to have run as expected:
~~~
digifil ./Crab_90s.fil -t 128 -b-32 -f 2 -I 0 -c -o ./Crab_90s_simult128_f2_nors.fil
digifil ./Crab_90s.fil -t 128 -b-32 -I 0 -c -o ./Crab_90s_simult128_nors.fil
digifil ./Crab_90s_t128.fil -D 56.75 -K -b-32 -t 2 -I 0 -c -o ./Crab_90s_t128_t2_nors_dedisp56.75.fil
digifil ./Crab_90s.fil -b-32 -I 0 -c -t 128 -o ./Crab_90s_t128.fil
digifil ./Crab_90s_t128.fil -b-32 -f 2 -o ./Crab_90s_t128_f2.fil
digifil ./Crab_90s_t128.fil -b-32 -f 2 -I 0 -c -o ./Crab_90s_t128_f2_nors.fil
digifil ./Crab_90s_t128.fil -b-32 -I 0 -c -o ./Crab_90s_t128_nors.fil
digifil ./Crab_90s.fil -t 128 -D 56.75 -K -b-32 -I 0 -c -o ./Crab_90s_simult128_nors_dedisp56.75.fil
digifil ./Crab_90s_t128.fil -b-32 -o ./Crab_90s_t128_rs.fil
digifil ./Crab_90s_t128.fil -b-32 -t 2 -o ./Crab_90s_t128_t2.fil
digifil ./Crab_90s_t128.fil -b-32 -t 2 -I 0 -c -o ./Crab_90s_t128_t2_nors.fil
digifil ./Crab_90s_t128.fil -D 56.75 -K -b-32 -t 2 -I 0 -c -o ./Crab_90s_t128_t2_nors_dedisp56.75.fil
~~~

I marked the last two commands as different as the length of the block of noise is shorter than the other outputs. In that case, something is happening during the dedispersion (which seems directly related to this ticket, should I open another ticket for the other resamping issues?), I can not see it happening for these commands
~~~
digifil ./Crab_90s_t128.fil -D 120 -K -b-32 -t 2 -o ./Crab_90s_t128_t2_dedisp120.fil
digifil ./Crab_90s_t128.fil -D 40 -K -b-32 -t 2 -o ./Crab_90s_t128_t2_dedisp40.fil
digifil ./Crab_90s_t128.fil -D 20 -K -b-32 -t 2 -o ./Crab_90s_t128_t2_dedisp20.fil
digifil ./Crab_90s_t128.fil -D 1 -K -b-32 -t 2 -o ./Crab_90s_t128_t2_dedisp1.fil
~~~

But all of these commands do cause it to happen:
~~~
digifil ./Crab_90s_t128.fil -D 45 -K -b-32 -t 2 -o ./Crab_90s_t128_t2_dedisp45.fil
digifil ./Crab_90s_t128.fil -D 50 -K -b-32 -t 2 -o ./Crab_90s_t128_t2_dedisp50.fil
digifil ./Crab_90s_t128.fil -D 55 -K -b-32 -t 2 -o ./Crab_90s_t128_t2_dedisp55.fil
digifil ./Crab_90s_t128.fil -D 60 -K -b-32 -t 2 -o ./Crab_90s_t128_t2_dedisp60.fil
digifil ./Crab_90s_t128.fil -D 65 -K -b-32 -t 2 -o ./Crab_90s_t128_t2_dedisp65.fil
digifil ./Crab_90s_t128.fil -D 70 -K -b-32 -t 2 -o ./Crab_90s_t128_t2_dedisp70.fil
digifil ./Crab_90s_t128.fil -D 75 -K -b-32 -t 2 -o ./Crab_90s_t128_t2_dedisp75.fil
digifil ./Crab_90s_t128.fil -D 80 -K -b-32 -t 2 -o ./Crab_90s_t128_t2_dedisp80.fil
digifil ./Crab_90s_t128.fil -D 85 -K -b-32 -t 2 -o ./Crab_90s_t128_t2_dedisp85.fil
~~~

it seems as though once the delay passes a certain threshold (the rescale interval?) it starts a cyclic pattern for the amount of data that is converted to noise in the output.

[Andrew Jameson]

Hi David,

Thanks for providing the example data set and lots of tests that you have run, that has been quite useful in determining the cause of the issue. I can indeed see a problem is being caused by the Recsale class and I believe I've got a fix that addresses many of these problem cases, perhaps the original one that Scott reported too. I hope to have a branch ready for you to test against soon.

Cheers,

Andrew

[Andrew Jameson]

David, would you be able to test against this branch and let me know how you find the results?

https://sourceforge.net/p/dspsr/code/ci/bug98-rescale-first-integration/tree/

Cheers,

Andrew

[David McKenna]

Hey Andrew,

Thanks! Indeed that seems to have fixed the bulk of the issues, though looking closely at the data I've spotted that the first two output timesamples (across all frequency channels) appear to be malformed (first sample is all 0, second is possibly a median value) for these commands:

digifil ./Crab_90s.fil -t 128 -b-32 -o ./Crab_90s_simult128.fil
digifil ./Crab_90s.fil -t 128 -b-32 -f 2 -o ./Crab_90s_simult128_f2.fil
digifil ./Crab_90s.fil -t 128 -D 56.75 -K -b-32 -o ./Crab_90s_simult128_dedisp56.75.fil

Looking at my old plots, this is was an issue in the previous master branch I tested with for the first and third command as well (second had a full block of 0s at the start), so it's likely another separate issue at play here.

Cheers,
David

[Andrew Jameson]

Hi David,

The issue of the first sample being zero and subsequent sample being close to the mean is a consequence of the default, small block that digifil uses (256 MB). Since your data set is quite large (many channels) and your Tscrunching quite substantial (128) , when digifil reads a block of data, it arises that all of those samples are scrunched into a single output time sample for the channel, and in this situation the rescaling algorithm can not determine the mean value - since it is trying to take a statistical mean of a single valued time series.

Once some subsequent blocks are processed, the estimate of the mean becomes sensible and so the algorithm functions correctly. As to the best solution for this situation, I'm not sure, but it should be handled separately to the bug that has been fixed in this branch.

Cheers,
Andrew

[Andrew Jameson]

Fixed in commit cba272