Basic_HRTF_NumCalc_Simulation

Authors:

The HRTF Simulation

Main simulation part of the Mesh2HRTF which is resource intensive and benefits from a computer with as large amount of memory (RAM) as possible.

------------------------------------------------------------	Part of the the Complete Beginner’s tutorial	------------------------------------------------------------
Previous <<< Project Export	\\\ Express tutorial ///	Next >>> Saving HRTF SOFA files

Video version of this tutorial:

The HRTF Simulation

Introduction to the NumCalc solver

NumCalc solver is the heart of Mesh2HRTF - it is the practical implementation of the algorithms that calculate sound propagation around the head and ears and find the numerical solutions that can be packaged into the HRTF .sofa files. NumCalc is a powerful audio simulation tool that can be used for more than just HRTFs.

Simulation pre-requisites

Everything for NumCalc is already installed as part of the Mesh2HRTF installation, but it is very common that the NumCalc simulations are executed on another computer (and another operating system) than the rest of Mesh2HRTF steps. Usual reasons are: because simulation can take many hours (or in worst cases even days) it is nice to avoid blocking the primary computer with simulation loads (note, it is possible to limit simulation resource usage). Plus distributing the simulation over additional computers is very effective way to speed up simulations. Therefore,

it is good to know which parts of the Mesh2HRTF installation are necessary for a computer that only executes NumCalc simulation:

The most important part is to install NumCalc.
- On Windows, it is usually done by just extracting the Mesh2HRTF tools.
- On Linux and Mac there is a very fast and simple compilation step using the source code from the Mesh2HRTF project.
To actually run the NumCalc it is usually necessary to use NumCalcManager.py (other scripts were used in the past as well).
- NumCalcManager.py is also available among Mesh2HRTF tools, but it is intended for Linux, Mac and Windows.
  - On Windows "NumCalcManager.py" and the folder "NumCalc_WindowsExe" usually must stay next to each other, but as a pair these files can be moved/copied anywhere, even to another computer that has Python 3 with psutil package installed.
  - On Linux/Mac you only need "NumCalcManager.py". The script will find where you installed NumCalc after the compilation.
- As a Python script, NumCalcManager.py requires Python 3.7+ and just the single package - pip install psutil.
There is No need to install "mesh2hrtf" Python package to run NumCalc simulations!

Run NumCalc simulation

This tutorial describes an approach using “NumCalcManager.py” which contains a number of very useful performance and convenience features for running NumCalc on any operating system. Otherwise, there are alternative ways to launch NumCalc, including the most primitive "just run the NumCalc.exe" approach. Advanced users are welcome to run NumCalc any way they need - "NumCalcManager.py" has no impact on the final "NumCalc" simulation results.

Typical steps to run NumCalcManager:

Following the Blender project export you should have 2 project folders (for left and right HRTF sides) which contain "info.txt" files along with other files and folders. A good place to have these folders is in your mesh2hrtf-tools folder, because then NumCalcManager.py is already located next to the 2 project folders and ready to execute.
Make sure “NumCalcManager.py” is located in a valid location:
1. The typical NumCalcManager.py location is "Next to project folders" - manager script will then scan all project folders next to it and start to simulate each folder sequentially one after another. This approach allows to simulate many HRTF projects one after another without manual launching of every folder. Any present finished project folders take only a few extra seconds to confirm their status and the manager script will continue executing just the instances that are not yet simulated.
2. The Alternative NumCalcManager.py location is "Inside one project folder" - this way only the selected project folder will be executed (not very useful, but supported mode). Note that the "NumCalc_WindowsExe" folder that is necessary on Windows (not used on Linux/Mac) should still be located one level above - next-to the project folder.
Launch the NumCalcManager.py.
- On Windows, if Python was installed Directly then it should be possible to just Double-click the “NumCalcManager.py” to launch the simulation. (If double-click does not work, try using right-mouse-click on the “NumCalcManager.py” and use "Open with" ==> "Python 3.x".)
- For Linux, Mac, Windows Anaconda installations or any other Python environment cases it is always possible to write python NumCalcManager.py(or possibly "python3 NumCalcManager.py") in the command line of the environment where "pip install psutil" was installed.
- Be careful on Windows: do NOT click inside the Python window. If you see a “white square cursor” – the “NumCalcManager.py” script will be paused and it will not do anything. To escape the problem: press “Esc” key until “white square cursor” disappears (you should then see some status messages appearing at regular intervals).
- Antivirus in Windows – if your antivirus raises a concern, it is recommended to treat “NumCalcManager.py” as Installer or Updater to prevent disturbances in repeatedly launching NumCalc.exe files. (anyone can see exactly what "NumCalcManager.py" does, so there is zero safety concern to use the unmodified script).
- If “NumCalcManager.py” script exits as soon as it is opened - most likely the issue is that "psutil" Python package is missing. Check the pre-requisites ("pip install psutil").

Done. Now wait for the simulation to complete :) Note that the simulation normally starts very slow and gets many times faster by the end.

Advanced NumCalc tips

Advanced knowledge around NumCalc (not necessary, but may be useful to know):

It is recommended to check the first NC*.out files when the first simulation instances are completed.
- Make sure that simulation printed out End time: line. In case of any issues, see Troubleshooting section.
- Note: You can ignore “Warning: There are some frequencies higher than the highest allowable frequency!”. This warning is only a rough indication that is valid for meshes that use uniform triangle edge size for the complete mesh. Properly optimized (graded) meshes almost always trigger this warning.
Two projects == two sources of errors. Note that any problem or crash in the simulation can happen independently in the LEFT or RIGHT ear projects – even if one ear is perfect, the simulation of the other side may still need checking.
Re-launch “NumCalcManager.py” after it is finished! To make sure that everything went well and all instances completed their tasks it is recommended to Re-launch “NumCalcManager.py” after the main simulation run is complete. If the project is fully simulated, the “NumCalcManager.py” will quickly report that everything is OK, alternatively it will automatically launch any instances that failed to produce output.
Memory usage (performance & system resources) – the main limitation to the number of instances you can run is system memory (RAM) usage. For reasonable computation time it is necessary to both use all available RAM & CPU resources and avoid RAM overload (which can absolutely slow down processing to a crawl). To manage the resources the simple solution is to use “NumCalcManager.py” which automates the process within one computer.
1. Multiply your simulation speed by running more NumCalc Instances. Note that each NumCalc instance can only utilize 1 CPU core (single threaded) and therefore maximum CPU usage of one NumCalc instance can stay well under 10% for your CPU. To utilize more cores, the only option is to run more NumCalc instances (which requires more free RAM).
2. More 3D mesh triangles == needs more RAM – Meshes with less than 15 thousand triangles require modest amounts of RAM/instance and can be calculated in a few hours on a PC with 32 GB of RAM. Meanwhile a mesh with 80k triangles can use over 10 GB of RAM/instance and hours of computation for each step. Note that with 16 GB of RAM available, a computer can easily be RAM limited to run only one instance at a time despite negligibly low CPU usage (one core). For some reference, in this paper Table 1 shows some estimates for computation time for a server with plenty of RAM / CPU-core (while most computers would be RAM-limited).
3. Higher sound frequency == needs more RAM – The RAM usage is highly dependent on the frequency that is computed in a given NumCalc instance. If computing 500 Hz takes just a couple of GB of RAM, the same mesh can use over 20GB when calculating 24 kHz. The amounts are approximate and highly dependent on exact mesh polygon count. To find the worst-case scenario early on, “NumCalcManager.py” runs the instances starting from the last one – last instances compute the highest frequency and need the most RAM resources.
  - This also means that in the beginning of the simulation the CPU will be hardly loaded because very few instances can fit into the available RAM, but as simulation progresses it will simulate lower and lower frequencies and eventually number of running NumCalc instances may even exceed number of available CPU cores and the simulation may be CPU performance limited (usually CPU limits aremuch less significant than RAM limits for the total simulation time).
  - In the beginning progress is slow because very few instances can fit into RAM (due to high-frequency), but the last instances are calculated with many running in parallel and the Simulation progress is dramatically faster in the end compared to the first instances.
Use multiple computers – Each simulation can easily be split into at least 2 parts (left and right side simulation project folders) and each project part can be simulated on any combination of Windows, Linux or Mac computers. When all simulations are complete the data will be merged in in the post-processing step.
Warning – It is dangerous to run “NumCalc.exe” without correct input arguments. If NumCalc executable (not the same as “NumCalcManager”) is launched directly in a folder that contains NC.inp file, NumCalc will delete all existing output data in that folder without warnings.
Do not run multiple “NumCalcManager.py” scripts on one computer. They will work, but the computer resources will be chaotically managed and the performance will be quite unpredictable. (If you need to schedule additional projects for simulation, it is better to kill the running “NumCalcManager.py” and restart it - you will loose progress in the currently running instances, but “NumCalcManager.py” will correctly scan all new and changed projects and schedule them for simulation.)
Interruptions – “NumCalcManager.py” guarantees that it is safe to kill any “NumCalc.exe” or “NumCalcManager.py” processes at any time and just re-start “NumCalcManager.py” to continue.
- “NumCalcManager.py” never re-runs instances that have completed their simulation. It properly checks the log files to detect which instances are complete. It means that unlike running raw NumCalc.exe, it is safe to launch “NumCalcManager.py” even when the whole project has been simulated to the end (note that if you manually re-start a NumCalc instance, it deletes its existing output data and starts over).
- It can happen that some NumCalc instances (out of the hundreds executed during the simulation) do not complete simulation because the computer was hibernated or any other process caused NumCalc instance to exit. The solution is very simple (see next tip):
Non-convergence – If the mesh is not smooth or has other defects the simulation can take 2-3 times longer to run. Normally the solution is found in less than 100 iterations, but in worst Non-convergence cases the simulation may run for 250 iterations before giving up. These extra iterations will prolong the total simulation time.

Limit simulation resource usage

Ideally the NumCalc should run when the computer is unused (over night?). It is always possible to kill NumCalcManager to immediately free up resources when needed and then just restart NumCalcManager to resume simulation.

In case you have to multi-task - continue using the computer that is also running "NumCalcManager.py" simulations, it is possible to adjust the setting in the manager script to leave more resources available:

Open/edit the "NumCalcManager.py" script as any text file.
Adjust the lines:
RAM_safetyFactor = - not an ideal control to ensure free memory but if you set safety factor very high (2.0 or more) then the manager script will always wait for much more free RAM than needed for the next NumCalc instance. If the simulation is RAM limited, this allows to keep some RAM free at most times.
Max_CPU_load_percent = - by setting maximum load limit to a lower percentage it is possible to ensure that there are always free CPU resources left in the system.
- Alternatively it is possible to set Auto_set_MaxInstances = False and force a particular Max_Instances = value for how many NumCalc instances can run simultaneously. (for estimates - one NumCalc instance can load 1 physical CPU core, no multi-threading)
Note that NumCalcManager always strives to run one NumCalc instance even if all other checks say that there is not enough RAM or CPU resources. Therefore if you run something resource intensive, even one NumCalc instance could result in RAM overload and reduced computer performance.

Troubleshooting

When using NumCalcManager, each instance of NumCalc simulation will generate:

NC*.out text log file (open it in any text editor). It contains very good readable printouts and statistics written by the NumCalc for diagnostics. Last lines may mention why and during which step NumCalc encountered problems.
NC*_log.txt text command line printout file. This information may compliment information from NC*.out file, but it usually is not necessary.

These files are located in ../yourProjectFolder/NumCalc/source_1/ and there is one file for every NumCalc instance (normally as many instances as there are frequency steps in the project).

Healthy NumCalc simulation

A sign of a successfully simulated NumCalc instance is a NC*.out text log file that ends with a line End time:. If end time line is missing, then the simulation is either not yet complete, has been interrupted or stopped due to an error. Here are EXAMPLES of:

healthy "NC .out" and the corresponding
healthy "NC _log.txt"

Note that there may be some non-fatal warnings among the printouts. One exception is non-convergence issue - it is critical, but will still produce end time line

In some cases it may also be useful to check the example of a heathy "info.txt" file (generated by the Blender project export).

Immediate crashes of NumCalc

In the most extreme cases, if the first launched NumCalc instance crashes directly, NumCalcManager.py will explicitly stop and warn the user. Such crashes are often due to major defects in the input mesh or if NumCalc executable is not working correctly.

Don't forget that sometimes it may be enough to just re-try the simulation (by starting NumCalcManager again).
To check that NumCalc executable itself is working, on Windows, check that /NumCalc_WindowsExe/folder is next to the NumCalcManager and then double-click the "- run_NumCalc_instance.bat" inside this folder. If everything is working correctly:
- Among the printouts there will be No input file found! - it means that NumCalc works.
- Also in the same folder NumCalc will create new folder "be.out", and file "NC1-1.out".
- If this is not the case - NumCalc.exe itself is not working on a given computer.
If "NC .out" files do not give any hints it may be a bug that should be reported as an Issue.

“3D analysis” not passed

The simulation crashes almost immediately and the “NC*.out” file contains just a few lines after the line (see example file with this issue):

3D analysis

This means that input mesh is obviously not suitable for simulation. For example the mesh is not air-tight, has some loose vertices, overlapping or internal surfaces or is just corrupt due to file format errors - these problems should be quite easy to identify by examining 3D mesh in detail.

“Non-Convergence issue”

The simulation takes much longer time and the “NC*.out” file contains lines (see example file with this issue):

Warning: Maximum number of iterations is reached!
CGS solver: number of iterations =

This information effectively says that the simulation did not find a solution (did not converge) and the result of this instance, for this frequency step is corrupt (garbage).

<= 24000 Hz – If this issue happens at frequencies within useful range (under 24kHz) the complete simulation results will NOT be usable! (the remaining usable data will not allow to produce HRTF for the common 48kHz sampling rate).
>24000 Hz – If this issue is encountered at simulation frequencies above 24000 Hz, the HRTF can still be used (assuming the frequency step is 150 or 75 Hz) because finalize_HRTF_simulation.py script will automatically drop all simulation data above the problematic frequency and still produce HRTF for the common 44.1 and 48kHz sampling rates. Higher than 24kHz frequencies are not fully tested and may require finer 3D than suggested in the recommended “hrtf_mesh_grading” (to keep closer to the rule of edge length must be at least 6times smaller than the shortest wavelength [[Source]](https://doi.org/10.1142/S0218396X02001401)).

This problem usually affects only the highest frequencies, but unfortunately it does not always show up in all frequencies (for example simulation may converge at 20000 and 19900Hz, but fail to converge at 19800Hz). Mesh2HRTF post-processing has built in detection, reporting and work-arounds of this issue, therefore if this issue happens at high-enough frequencies, the simulation results can still be usable. The possible workarounds are:

Make sure that the 3D meshes were processed using “hrtf_mesh_grading” – other methods for re-meshing may cause less optimal final meshes that can cause non-convergence.
Only in case of using Point source simulations: make sure that the point source is at least ~0.3 mm distance away from the closest tringle. If Point Source is too close, it may get treated as it is inside of the mesh.
The main cause of this issue is significant errors in the mesh. Check if you can find and fix issues in the 3D mesh, such as:
- Overlapping or folded-over triangles
- Unnaturally sharp edges or deep holes.
- Very large individual triangles.
- Some face normals do not point outwards (try to "recalculate normals").
- Any other surface disturbance in areas that should have a smooth skin surface.
Last resort: try this workaround: you need to "manually influence the cluster size" (the grouping of elements for the ML-FMM method). It is done by changing a value inside NC.inp file (located in the ../your_project/NumCalc/source_1/ folder). On the line 2. Main Parameters II, set the fourth number to 0.05 (everything else stays as it is):” for example like this: “ 0 1 0 0.05 0 0 0 ”. Then try to continue simulation using NumCalcManager.

“Gaussean points limit issue”

If simulation crashes and the “NC*.out” file contains lines (see example file with this issue):

Too many integral points in the theta-direction!
Number of integration points =

This issue usually only appears on complex (or broken) meshes at high frequencies. The possible workarounds are:

This issue has not been observed when 3D mesh is processed using “hrtf_mesh_grading” – this is the main fix (assuming the 3D mesh itself is good).
Check if you can find issues in the 3D mesh:
- Overlapping or folded-over triangles
- Unnaturally sharp edges or deep holes.
- Very large individual triangles.
- Some face normals do not point outwards (try to "recalculate normals").
- Any other surface disturbance in areas that should have a smooth skin surface.

“MSBE issue”

If all simulation instances crash and the “NC*.out” file ends with line "nsbe must be < MSBE(= 110)" (see example file with this issue):

coordinates of the source point: 
nsbe must be < MSBE(= 110)
number of the element =

According to Dr. Wolfgang Kreuzer this issue is usually caused by:

there is an overlapping element - a tringle that overlaps with another triangle.
the Point Source is too close to the surface of the mesh - repositioning the Point Source at least 0.3 mm away from the surface should fix this case.

Overlapping elements can happen in rare cases even when using “hrtf_mesh_grading”. To detect and fix this issue:

Open for inspection the mesh that is causing the problem. (It can be the graded mesh saved by hrtf_mesh_grading or the same mesh exported out of Blender file from which the Mesh2HRTF Project was exported.)
If the graded mesh has a problem, it will show up in the "Air-tight mesh check" as described in How to make sure that 3D mesh is ready.
- The most elegant fix for overlapping triangle issue is using Meshlab command: “Filters ==> Cleaning and Repairing ==> Remove Duplicate Faces”. If the command prints out "Successfully deleted 1 duplicated faces", then the problem is most likely resolved. The mesh will need to be re-imported into Blender and project re-exported.

Next tutorial step >>> Saving HRTF SOFA files

Wiki: Basic_HRTF_post_processing
Wiki: Basic_HRTF_tutorial
Wiki: Basic_Project_export
Wiki: Express_Basic_HRTF_tutorial
Wiki: Installation_2
Wiki: Tutorials
Wiki: Unix Tutorials

Mesh2HRTF Wiki