SonasoundP - Real-time phonetics

Title: README for SonasoundP Author: Richard Lemieux Date: 2013-04-09 Last time reviewed: 2015-08-10 Copyright: 2013,1014 Richard Lemieux SonasoundP Copyright (C) 2009-1014 Richard Lemieux under GNU GPL Version 3 and higher versions. Sonasound Copyright (C) 2002-2003 Niklas Werner READY TO RUN PROGRAMS FOR WINDOWS sonasoundp.exe: The program runs well on my non-throttled Windows10 workstation. This build also work fine on an older LE1700 tabletPC featuring a Core2 dual CPU and an Intel video chipset. VBlank synchronization is not supported by OpenGL on that older platform but it works well in Manual sync mode by setting the bottom slider to 60 on the Panels window.. SonasoundP may segfault the first time it runs. That may be related to fact that the program creates and uses a small file to hold FFT3W parameters. Otherwise, SonasoundP looks quite stable. The parameter file is named "sonasound.fftw_wisdom3_threads". sonasound.exe (deprecated): This older edition ran on Windows8 and earlier versions of Windows. The Windows programs are compiled using a MingW setup. I didn't figure out yet how to make a ready-to-run program for the Linux version. PROLOGUE This software is provided as is and without any warranty of any kind and under the GNU copyright agreement. This software is a follow-up to Niklas Werner SonaSound that is distributed under the GNU copyright from the sourceforge.net site and includes material copyrighted by Niklas Werner. Differences between Niklas's original SonaSound and SonasoundP are listed in the initial entry of the ChangeLog file. SonasoundP uses a simple pipeline architecture that should provide a smooth user experience on many types of computers but this simple design results in glitches and stuttering of the sonogram display in some cases. The current design is more resilient than the earlier design was, and the hope is that the next design will improve on the current design still further. Sonasound allows a wide range of choices of the display speed, FFT size and sample rate and some combinations of parameters may give poor results depending on the computer and the operating system. The idea is that one can hopefully find a usable combination of parameters appropriate to the computer/operating system combination in use. ACKNOWLEDGEMENTS This project uses software and information from sources including. - PortAudio <www.portaudio.com/> This makes audio input very simple. - FLTK <www.fltk.org/> The portable lightweight windowing system. - FFTW <www.fftw.org/> A fantastic library of extremely efficient FFT functions. - MINGW <www.mingw.org/> The GNU develpment system on Windows. - GLFW <www.glfw.org/> The library of timing functions. - Microsoft <http://msdn.microsoft.com/> Excellent documentation on Windows API's. - Windows community forums such as <stackoverflow.com> and many others. - Linux documentation teams and community forums. and of course so many other people and projects. PURPOSE SonasoundP connects to (mono) sound input from the sound card or a USB microphone and displays a real-time sonogram on the display. The intent is to allow students of foreign languages to see features of the sound signal as they speak. I have been using SonasoundP while learning Chinese pronunciation. I have used it in four different ways: 1. Use the logarithmic FFT display to practice the tones, 2. Use the logarithmic FFT display to practice unvoiced consonants such as 'd', 3. Make sure to separate the syllabes with no liaison in between, and 4. Use the linear LPC display to visualize the formants and compare with native speakers. The formants appear as lines on the LPC display and represent resonances in the vocal tract. The resonances depend on the shape of the vocal tract as a result of the position of the tongue and other parts of the vocal tract. The formants do not depend on tones. In fact you can see some of the formants even when you don't voice speech if the microphone is close enough to the mouth. While vocal chords produce a sound of one frequency and harmonics, unvoiced sound is mostly noise and it includes a wide range of frequencies; the flow noises are not white noise and don't come from the focal chords location though. The vocal tract removes frequencies from the initial mix. So, move all parts of the mouth in all kinds of combinations to see which formants results. You can also see the formants on the FFT display, but then make your task easier by generating more harmonics by lowering the tone or try unvoiced speech. Unvoiced as well as voiced speech also includes spectrally colored sound created by a process called venturi collapse. This process occurs when air is accelerated at a stricture between the soft tongue and another surface such as the palate or the teeths and the pressure suddenly drops causing a deformation of the softer material and leading to a repetitive process. (See "https://en.wikipedia.org/wiki/Formant") The sonogram display lags a bit behind the sound and the delay depends mostly on the size of the FFT window and the audio packet delivered by PortAudio. The resulting lag is 1/10 s for a 16,000 sample/s signal and a 1024 FFT size (1/15 s from the FFT window and 1/30 from the PortAudio packet); this lag translates in 30 pixel columns on the display at 5x acceleration. So it is not that easy to relate the video to the sound when monitoring a fast speaker. HARDWARE REQUIREMENTS 1. A video card that provides hardware accelerated OpenGL. Some OpenGL implementations will allow synchronization between the program and the monitor and others won't. 2. A fast enough processor. SonasoundP uses nearly 20% of one CPU on my 2.4 GHz machine. So I guess the machine should be at least 500 MHZ. INSTALLATION OF SonasoundP See the INSTALL file for compilation instructions. The basic problem is to install all needed libraries and dependencies and edit the appropriate Makefile accordingly. Otherwise this distribution includes a binary compiled for Windows and ready to run. OPERATING INSTRUCTIONS 1. Start the program. Two windows will appear on the computer display: a Controls window and a Panels window. 2. In the Controls window, you need to 1. Click on button 'Audio device' to select the input audio device to connect to. If the microphone you want to connect to is not listed, it may be because some other program such as Skype is already using it; close other programs that may be using the microphone and restart SonasoundP. The audio device will likely be either the microphone jack of the sound card or the USB microphone attached to the webcam. Then check 'Sampling rate' just on the right. Click on the arrow head and select 16,000 or a number close to 16,000. 2. Click button 'Open/Close Device' to connect to the device you just selected. 3. Click button 'Start/Stop All' to start the sound processing and the drawing on the panel window. 3. Start talking and watch the spectrogram on the panels window. 1. Make sure the sound is loud enough to fill half of the top signal window. Adjust with the computer mixer program. In Windows click on 'Sound/Recording' and then on 'Microphone properties/Levels'. In Linux it depends on the distribution. I use 'alsamixer' in an xterm terminal to adjust the microphone level. 2. The panels in the Panels window can be resized by dragging the panel boundaries with the mouse. 3. Click on the left mouse button when the cursor is over a panel to freeze / unfreeze the display. 3. Possible problems. 1. The program seg faults when starting. This may occur once in a while, but the program is stable once started. This is probably caused by a variable that is not properly initialized at startup but which variable it is has escaped all code reviews up to now. 1. The animation stuttering/tearing can be removed by selecting VBlank using the bottom right button on the panelsWindow. The result may depend on the computer and the operating system. 2. The sonogram display appears to move too slowly or the time diplay shown when moving the mouse in the sonogram panel is too small. This is an indication that the video card is imposing VBlank synchronization and is not allowing applications to set/unset it under program control. Sonasound needs the capability to control the synchronization with the monitor refresh rate when using VBlank synchro mode because SonasoundP makes use of three GL contexts and just one of them needs to wait for the VBlank. Otherwise if the three panels are made to wait then the panels together run at one third of the expected rate. Under Linux with a NVIDIA video card I need to unset the 'Sync to VBlank' option using program 'nvidia-settings' under 'X Screen / OpenGL Settings'. If using an NVIDIA card under Windows, open the NVIDIA control panel and select 'Use the 3D application setting' under 'Manage 3D settings / Vertical sync'. I did not meet that issue with the Intel video chipset on my slate PC since the older OpenGL version used on that Intel Chipset does not provide any GL extension for VBlank control. MORE CONTROLS ON THE CONTROLS WINDOW. Sampling Rate: I use 16,000 samples per second for speech. This is good enough for language training. If you use a large number such as 44100 you will need to increase the 'FFT size' and 'LPC size' in proportion. The sampling rate is what determines the 'scale' between samples and seconds. The signal processing algorithms act on samples. The relation between those results and the world of sound is set by the sampling rate. A word or caution. In Windows the selection provided in the 'Sampling rate' menu reflects 'resampled' signals and may have no relation with the actual bandwidth of the microphone or the actual sample rate provided by the USB microphone. For example I use a USB camera microphone returning samples at a fixed 16,000 samples/s rate, but Windows offers 48,000 in the selection. FFT Size: 512 or 1024 is OK. You may change the size to your liking. This is the size of the sample window used to compute the FFT or the LPC parameters. Don't forget to increase the 'Sampling rate' if you want more resolution here. LPC size: Used when 'Spectrum Type' is LPC. A good starting value is 25 if 'Sampling rate' is 16,000 samples/second and the 'FFT size' is 1024. You may want to experiment a bit here. FFT window type: This has nothing to do with the computer monitor. Think of it as a curtain overlaid over each successive segment of the audio signal and attenuating the signal so it is zero on the edges an beyond. All choices except 'Rectangular' are good here. 'Rectangular' means that the audio signal is left unchanged in the window and is zero everywhere else; this choice results in many artifacts in the sonogram. The FFT is computed on the resulting windowed signal. The width of the window is the 'FFT size'. Spectrum Type: Select either FFT or LPC. FFT is OK. LPC with an appropriate value of 'LPC size' might give you a clearer display of the formants. When the pitch is high it may be hard to see some formants on the FFT display. However there as times when the FFT display is more informative though. Display Grid: 'logarithm' is the default, 'log_colored' gives a blue/red colored display. Ear response curve: Selects an additional attenuation (dB) to be added to the power spectrum to model the ear response. No correction: The attenuation is 0 dB at all frequencies. Human 1: The attenuation reflects the human ear response curve according to measurements published in: Masakazu Konishi, "How the owl tracks its prey.", 1973, American Scientist. Buffer Size: This is a read-only number. This tells how many new samples move in the signal window at successive sonogram computations. The same number of samples move out of the window at the other end. Draw staff lines: Whether or not you need the staff lines. High pass: Batch PS: This control has no effect on how the sound is processed. It controls the way the power spectra computations are scheduled. When the tick is marked, all the power spectra displayed in one paint cycle are computed in a single batch without exchanging signals with other threads. When the tick is unmarked, a signal is sent when a new power spectrum will be needed. Given the fact that Windows and Linux are not real-time operating systems it is better to limit the message traffic as much as possible and select the batch mode here. WHAT'S ON THE PANELS WINDOW. Top panel: Shows a segment of the signal but not necessarily of the same size as the FFT Size. Middle panel: This shows the power spectrum computed from the FFT of the latest signal window. Right slider: This adjusts the gain of the sonogram panel. Eventually, it will also adjust the gain of the FFT panel. Lower panel: This is the sonogram panel. This is a sequence of power spectras with the power values translated into color or gray level pixels. The horizontal axis shows the time and the vertical axis the frequency. Move the cursor in this window and see the values of time and frequency printed in the bar on the bottom of the window. Bottom left selector: This provides a choice of three ways to select the paint refresh rate. VBlank: Painting is done synchronized with the monitor VBlank signal if this works. Windows: The windowing system generally knows the refresh rate selected for the monitor. This is currently the value used at program startup but the default will eventually be VBlank. If windows can't tell the monitor frequency, this choice will revert to manual mode and a 60 Hz refresh rate. Manual: Use the slider on the right to select a refresh rate of your choice. The painting is then synchronized by the computer clock instead of the monitor. VB2: Experimental mode targeted for the LE1700 which has Intel video. This setup shows unexpected behavior which still needs investigation. Bottom slider: Sets the refresh rate. This number should be the monitor refresh rate but you can use other values. When Sonasound starts it tries to set this number to the value of the monitor refresh rate as known to Windows or X11. If Sonasound can't get the refresh rate that way, it uses 60 frames per second. Setting that slider to '0' activates 'VBlank sync' or tries to. Then Sonasound uses the measured value of the frame rate. Depending on the hardware combination 'VBlank sync' may not work with Sonasound. This is yet a tricky issue. Bottom right spinner: Sets the number of FFT's or LPC's to be painted at once in the video buffer (typically every monitor refresh cycle). The spinner allows a selection between 1 and 16. A typical LCD monitor will refresh every 1/60th second. The number here tells how many pixels to slide the sonogram panel towards the left. If the number is 4 then the panel is slided by 4 pixels on the left every 1/60th second, and four pixel columns are drawn one for each successive power spectrum. In this example power spectra need to be computed at the rate of 240 FFT/LPC per second (240 = 60 times 4). OTHER PRACTICAL POINTS * Some of the parameters can be set on the command line when starting the program. Try 'sonasound -h' to get the list of available parameters. Here is how I start sonasoundp, sonasound -D 2 -f 1024 -w 1 -g l -d 1 -l 25 In my case Device 2 is the webcam microphone, -f 1024 gives an FFT size of 2048, -w 1 for the Hanning asymmetric window, -g l for LPC processing -d 1 for the linear display -l 25 for a LPC size of 240. On Windows, if you use the already compiled program sonasound.exe, 'sonasound -h' won't print anything since the console window is disabled in this build (load option -mwindows). However you can still use a command line as above to start sonasound. For your convecience, the output of 'sonasound -h' is included at the end of this README. * Currently SonasoundP handles only Mono (as opposed to Stereo or multi-channel) sound. FUTURE DEVELOPMENT I publish updates when I learn new ways to improve the performance and functionality. The ultimate aim is to provide a practical tool for foreign language students, running on popular platforms. SUPPORT Support is throught the public forum on the SourceForge site. The focus at this point is to provide a stable program on main platforms. NOTES FFT Abbreviation for Fast Fourier Transform. This also means the Fourier transform of a window of the signal. In the extensive sense it often denotes the power spectrum of the FFT to contrast with the power spectrum of a LPC filtered signal. LPC Linear Predictive Coding. This denotes a simple signal filter model of the vocal tract. The parameters of that model are estimated for each window of the input signal. The LPC display shows the power spectrum of the output of that estimated filter when feeded with white noise. HOW TO START SONASOUNDP FROM A COMMAND LINE # sonasound.exe -h Usage: ./sonasoundp [-a Size] [-b Freq] [-d {0,1,2}] [-D {0,1,...}] [-e {0,1}] [-D {0,1,...}] [-f Size] [-g {f,l}] [-H Coeff] [-l Size] [-n {0,1}] [-s Number from list] [-v] [-w {0,...,5}] SonasoundP Copyright (C) 2009-2014 Richard Lemieux Sonasound Copyright (C) 2002-2003 Niklas Werner Please note that one needs to first run 'sonasound' and check the available options from the controlsWindow to find meaningful options for parameters D and s below. -a: FFT size. Any number up to 65535. Use "-f" instead unless you have specific needs. -b: Reference A frequency in [Hz] for the Staff-lines -d: Display grid: 0: logarithmic gray levels, 1: linear gray levels, 2: logarithmic colored -D: Input Audio device: First: 0, Second: 1, ... Disable: -1 -e: Ear response curve: 0: No correction, 1: Human 1 -f: FFT size: up to 65535 rounded to NextPowerOfTwo (512,1024,...). See also "a". -g: Spectrum type: f: FFT, l: LPC -H: HighPass-filter coefficient: a number in the range (0.001, 1.0) default: 0.5 -l: LPC size: up to 65535 though only up to 256 makes sense. -n: Draw staff-lines: 0: No, 1: Yes. -s: SamplingRate: a number available for the selected audio device (44100 or 48000 ?) -v: Verbosity. Prints lots of data used by the programmer when he tests the program. -w: FFT Window type: 0: Hamming, 1: Hanning, 2: Blackman, 3: Bartlett, 4: Kaiser, 5: rectangular -h: Print this help SOFTWARE REVISIONS SVN225. Single thread mode.