|
From: Paul M. <pau...@gm...> - 2025-01-13 17:13:23
|
First Make sure you read this (if you haven't already): https://modules.readthedocs.io/en/stable/modulefile.html#locating-modulefiles An option you might try is to make a module tree that is simply populated with links to the actual module files. And point MODULEPATH to that tree. If that tree were structured exactly the same as your current tree (but only containing the modulefiles); the modulefiles would not have to be rewritten. I suspect, if modules could be told to restrict modulefiles to a particular file extension (what is the extension used for the windows version of modules?), the processing would speed up considerably (as it would not have to open every file to check if it is a module file). I should point out this is NOT a current feature - just asking if such an OPTIONAL feature would even be useful. Assuming that extension were .modfl (It should probably be different), You would not have to change the name of the modulefile, just add the extension to the name (the extension wouldn't be needed when referencing the file - as it would be assumed). So the module file would be: /release_path/package_name/version_directory/modulefile.modfl On 2025-01-13 10:47 AM, Miller, Eric via Modules-interest wrote: > [AMD Official Use Only - AMD Internal Distribution Only] > > Hi Xavier- > > Thank you for the quick reply. > > You are correct, we do see poor performance when running "module avail" under 3.2.10. > However, we never use that command. Perhaps we would if the performance was better. > > It is not a hardware issue, but rather a scaling issue. We have thousands of software packages with millions of non-modulefile files across the version directories. > All our packages are structured as: /release_path/package_name/version_directory/modulefile > > For now we are specifying the full path just as you recommended in the second email, but this approach is just a short term fix. > > Another workaround we have tried is isolating the modulefiles from the release repository into a standalone module_path. However, this requires us to rewrite all our existing modulefiles (~10k). > I'll continue to do experiments in this area. Perhaps we can automate the conversion and add it to our release process. > > The cache currently has 2 issues that prevent us from using it with our current setup: > 1. Creating the cache is slow and uses lots of memory. I killed cachebuild after running for 2.5hrs and using 81GB of memory on modern enterprise hardware. > 2. Even if the cache is created, using it is still somewhat slow. Removing the module-invalid commands in the cache file helps, but is still slower than the search behavior in 3.2.10. > > I would ask for the following enhancements to bring the search performance back to parity with version 3 (or even improve on it): > 1. Grant the ability to tune the modulefile search so it does not inspect all files in a version directory. > 2. If there is an exact path match to the search specifier, use it without searching or inspecting the modulefiles. > 3. Improve the cache file search performance and memory usage (i.e. do not load the entire cache into memory). This can be tricky. > > Regards, > Eric Miller > > -----Original Message----- > From: Xavier Delaruelle <xav...@gm...> > Sent: Sunday, January 12, 2025 11:58 PM > To: Environment Modules usage and discussion. <mod...@li...> > Subject: Re: [Modules] Prevent modulefile search from looking at all files in directory > > Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding. > > > I just remind of an alternative way which is to refer to the module by its full path name: > > module load /path/to/ModulePath/MyModule/1.0/modulefile > > This way, no other files will be scanned/evaluated. > > This solution may help you to find the speed you expect. > > Regards, > Xavier > > Le dim. 12 janv. 2025 à 20:40, Xavier Delaruelle <xav...@gm...> a écrit : >> >> Hello Eric, >> >> Best solution I can think of is using module cache. >> >> It is a bit slower but it should not be noticeable. If it does, I >> would suggest to look at the performance of the underlying storage >> system. If there is no issue with the storage system, I would be happy >> to get some debugging output in --timer mode. >> >> With large number of non-modulefiles in modulepaths, load is slower on >> newer version of Modules than 3.2 due to collecting all module symbols >> that applied to loading module. Even if load is efficient with this >> kind of setup on version 3.2, bad performance should be observed on >> "module avail". >> >> Regards, >> Xavier >> >> Le ven. 10 janv. 2025 à 16:59, Miller, Eric via Modules-interest >> <mod...@li...> a écrit : >>> >>> [AMD Official Use Only - AMD Internal Distribution Only] >>> >>> >>> My question: Is there a way to optimize the modulefile search algorithm for cases where the directory contains many files that are NOT modulefiles? >>> >>> >>> >>> Background: >>> >>> >>> >>> Given a directory structure like: >>> >>> >>> >>> ModulePath/MyModule/1.0/ >>> >>> ModulePath/MyModule/1.1/ >>> >>> … >>> >>> >>> >>> And given that each version directory contains one file named “modulefile” at the root of the directory and also contains a large number of other subdirectories and files. >>> >>> >>> >>> The command: “module load MyModule/1.0” will be very slow. >>> >>> >>> >>> The root cause appears to be that the modulefile search algorithm stats and reads every file in each of the ModulePath/MyModule/* directories. >>> >>> Going back to modulecmd 3.2.10 we do not see this issue. >>> >>> >>> >>> I’ve tried setting MODULES_MCOOKIE_CHECK=eval, but that caused errors when the loader encountered non-modulefile files. >>> >>> >>> >>> I’ve also tried using the cachebuild command, and while it helps, it is still slower than 3.2.10. >>> >>> The cachefile also appears to be tied to that specific version of modulecmd due to the header: #%Module5.5. >>> >>> >>> >>> >>> >>> Thanks! >>> >>> Eric Miller >>> >>> _______________________________________________ >>> Modules-interest mailing list >>> Mod...@li... >>> https://lists.sourceforge.net/lists/listinfo/modules-interest > > > _______________________________________________ > Modules-interest mailing list > Mod...@li... > https://lists.sourceforge.net/lists/listinfo/modules-interest > > _______________________________________________ > Modules-interest mailing list > Mod...@li... > https://lists.sourceforge.net/lists/listinfo/modules-interest > -- -------------------------------------------------------- The views and opinions expressed above are strictly those of the author(s). The content of this message has not been reviewed nor approved by any entity whatsoever. -------------------------------------------------------- Paul FM Info: http://paulfm.com/~paulfm/ -------------------------------------------------------- |
