[Lmod-users] Building packages and Lmod's Spider cache revisited

[Robert M. <mc...@ta...>]

I would like to revisit the spider cache file and building.   Previously,
we discussed the possibility of some how rebuilding the Spider cache
incrementally.  It became clear that that approach is possible but it gets
very complicated when there are multiple builders on a cluster sharing a
single file system.

We here at TACC have a somewhat similar situation.  We have a build machine
where we build rpm packages.  I have come up with a simpler solution that
may work for others.

As I said we use rpm as our package system.  So what I have done is created
a shell script called rpm (in /usr/local/bin)  All it does is check the
first argument of rpm to see if it is modifying the file system. If it does
then it updates the time stamp file that tells Lmod that the  Spider Cache
is out-of-date.  Then every 10 minutes (or 5 or 2mins),  we have a cron
jobs that updates the spider cache if it is out-of-date.

There may be other sites that can use this strategy.    I have been
thinking about whether this would work for case of EasyBuild on a cluster
with a shared file system.  I have placed in lmod/contrib/TACC the two
scripts we are using: (rpm and updateLocalCache.sh) to show what we are
doing.

R.


-- 
Robert McLay, Ph.D.
TACC
Manager, HPC Software Tools
(512) 232-8104