|
From: Xavier D. <xav...@gm...> - 2025-01-12 19:40:41
|
Hello Eric, Best solution I can think of is using module cache. It is a bit slower but it should not be noticeable. If it does, I would suggest to look at the performance of the underlying storage system. If there is no issue with the storage system, I would be happy to get some debugging output in --timer mode. With large number of non-modulefiles in modulepaths, load is slower on newer version of Modules than 3.2 due to collecting all module symbols that applied to loading module. Even if load is efficient with this kind of setup on version 3.2, bad performance should be observed on "module avail". Regards, Xavier Le ven. 10 janv. 2025 à 16:59, Miller, Eric via Modules-interest <mod...@li...> a écrit : > > [AMD Official Use Only - AMD Internal Distribution Only] > > > My question: Is there a way to optimize the modulefile search algorithm for cases where the directory contains many files that are NOT modulefiles? > > > > Background: > > > > Given a directory structure like: > > > > ModulePath/MyModule/1.0/ > > ModulePath/MyModule/1.1/ > > … > > > > And given that each version directory contains one file named “modulefile” at the root of the directory and also contains a large number of other subdirectories and files. > > > > The command: “module load MyModule/1.0” will be very slow. > > > > The root cause appears to be that the modulefile search algorithm stats and reads every file in each of the ModulePath/MyModule/* directories. > > Going back to modulecmd 3.2.10 we do not see this issue. > > > > I’ve tried setting MODULES_MCOOKIE_CHECK=eval, but that caused errors when the loader encountered non-modulefile files. > > > > I’ve also tried using the cachebuild command, and while it helps, it is still slower than 3.2.10. > > The cachefile also appears to be tied to that specific version of modulecmd due to the header: #%Module5.5. > > > > > > Thanks! > > Eric Miller > > _______________________________________________ > Modules-interest mailing list > Mod...@li... > https://lists.sourceforge.net/lists/listinfo/modules-interest |
