Re: [wsjt-devel] Using FFTW system 'wisdom' file

[Bill S. <g4...@cl...>]

On 11/08/2021 06:18, Phil Karn via wsjt-devel wrote:
> I'd like to make another suggestion. Somewhere near the top of
> wsprd.c/main(), insert the following code:
>
>      // FFTW plan files without threads are incompatible with plan files
> with threads (even 1 thread)
>      // To keep compatibility with system wisdom created by other
> programs that use FFTW with threads,
>      // initialize threaded mode with just one thread - KA9Q
>      fftwf_init_threads();
>      fftwf_plan_with_nthreads(1);
>      fftwf_import_system_wisdom();
>
> For debugging, check the return value of fftwf_import_system_wisdom().
> It returns 1 if successful.
>
>
> Explanation:
>
> The FFTW package supports "wisdom" files that allow a program to save
> for future use information on the fastest way to do a particular FFT on
> a particular system.
>
> wsprd already uses this mechanism, but only on a per-program basis by
> storing it in the local file 'wspr_wisdom.dat'. It doesn't share it with
> other copies of wsprd that might be running on the same system in
> different directories (which are necessary because wsprd doesn't
> coordinate writes to these files from multiple copies of the program.)
>
> Furthermore, wsprd creates the wisdom files at run time, as the "plans"
> are created. This limits how much time it can spend searching for the
> optimal algorithm since the program would otherwise take a very long
> time to run the first time (like an hour).
>
> I think a better way is to use FFTW's 'system wisdom'. This stores
> wisdom data in the file /etc/fftw/wisdomf (for single-precision floating
> point) for the use of all programs. It provides a standalone utility,
> fftwf_wisdom, for generating this file with a set of favorite transform
> sizes. With this mechanism you can really let your system crunch away
> finding the very best algorithms.
>
> As far as I can tell, the three transform sizes used by wsprd are:
>
> rof1474560 cob46080 cof512
>
> The first two (a real-to-complex forward transform with 1,474,560
> points, a complex-to-complex backward transform with 46,080 points) are
> used for decimating input .wav files. These are only done once per
> execution. The last one (a complex forward 512-point transform) is
> called many times during execution to find windowed transforms over 2
> symbol times stepped by a half symbol, so optimizing it is important.
>
> The code I suggest not only reads in the system wisdom file
> /etc/fftw/wisdomf, but it also sets up the FFTW package to operate in
> multi-threaded mode. This is necessary because (as I learned the hard
> way) the default wisdom file format is incompatible with the wisdom
> format for threaded operation, even when only 1 thread is used. I often
> use multithreaded FFTs in my other applications, so I use multi-threaded
> wisdom files as my system standard. (Multi-threaded wisdom files are
> compatible with each other regardless of the number of threads selected
> -- even 1).
>
> To initialize the system wisdom file, do this:
>
> mkdir /etc/fftw
>
> cd /etc/fftw
>
> fftwf-wisdom -o nwisdom -v -T 1 rof1474560 cob46080 cof512
>
> mv nwisdom wisdomf
>
> This will take a*long*  time to run, so be patient. The program isn't
> very fault-tolerant, so if you have to become root to create /etc/fftw,
> make sure you can write into the /etc/fftw directory before you start
> fftwf-wisdom. Use the temporary output file so the program won't
> overwrite any existing system wisdom before overwriting it.
>
> 73, Phil

Hi Phil,

another option that requires no code changes is as follows:

1) Create wisdom as above but without the '-T 1' option,
2) Rename the newly created wisdom to wspr_wisdom.dat in the expected 
directory,
3) Make the wspr_wisdom.dat file non-writable,
4) Symlink or hardlink the file into every location needed by multiple 
instances of wsprd.

The existing plans will be "upgraded" from their default of 
FFTW_ESTIMATE to FFTW_PATIENT for the specified FFT types and sizes, and 
there will be no file locking requirement as attempts by wsprd to update 
the wisdom file on exit will silently fail.

73
Bill
G4WJS.