Re: [Modules] Prevent modulefile search from looking at all files in directory

[Xavier D. <xav...@gm...>]

I just remind of an alternative way which is to refer to the module by
its full path name:

module load /path/to/ModulePath/MyModule/1.0/modulefile

This way, no other files will be scanned/evaluated.

This solution may help you to find the speed you expect.

Regards,
Xavier

Le dim. 12 janv. 2025 à 20:40, Xavier Delaruelle
<xav...@gm...> a écrit :
>
> Hello Eric,
>
> Best solution I can think of is using module cache.
>
> It is a bit slower but it should not be noticeable. If it does, I
> would suggest to look at the performance of the underlying storage
> system. If there is no issue with the storage system, I would be happy
> to get some debugging output in --timer mode.
>
> With large number of non-modulefiles in modulepaths, load is slower on
> newer version of Modules than 3.2 due to collecting all module symbols
> that applied to loading module. Even if load is efficient with this
> kind of setup on version 3.2, bad performance should be observed on
> "module avail".
>
> Regards,
> Xavier
>
> Le ven. 10 janv. 2025 à 16:59, Miller, Eric via Modules-interest
> <mod...@li...> a écrit :
> >
> > [AMD Official Use Only - AMD Internal Distribution Only]
> >
> >
> > My question:  Is there a way to optimize the modulefile search algorithm for cases where the directory contains many files that are NOT modulefiles?
> >
> >
> >
> > Background:
> >
> >
> >
> > Given a directory structure like:
> >
> >
> >
> > ModulePath/MyModule/1.0/
> >
> > ModulePath/MyModule/1.1/
> >
> > …
> >
> >
> >
> > And given that each version directory contains one file named “modulefile” at the root of the directory and also contains a large number of other subdirectories and files.
> >
> >
> >
> > The command: “module load MyModule/1.0” will be very slow.
> >
> >
> >
> > The root cause appears to be that the modulefile search algorithm stats and reads every file in each of the ModulePath/MyModule/* directories.
> >
> > Going back to modulecmd 3.2.10 we do not see this issue.
> >
> >
> >
> > I’ve tried setting MODULES_MCOOKIE_CHECK=eval, but that caused errors when the loader encountered non-modulefile files.
> >
> >
> >
> > I’ve also tried using the cachebuild command, and while it helps, it is still slower than 3.2.10.
> >
> > The cachefile also appears to be tied to that specific version of modulecmd due to the header: #%Module5.5.
> >
> >
> >
> >
> >
> > Thanks!
> >
> > Eric Miller
> >
> > _______________________________________________
> > Modules-interest mailing list
> > Mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/modules-interest