Steel Bank Common Lisp

On 15 September 2010 06:11, Peter Keller <psilord@...> wrote:

> How can I find the absolute paths of all the loaded shared libraries
> in a running lisp image? I'm writing an application which, before it
> calls save-lisp-and-die, needs to copy all loaded shared libraries to
> the current working directory of the lisp image. This is so the shared
> objects it needs can be moved along with the executable to another
> machine for execution. A constraint is that I'm not going to know a
> priori the set of loaded shared objects until just before I call
> save-lisp-and-die and those libraries may have been loaded via cffi
> or some other means.

This is going to be ugly, and is technically unsupported right now.
Sorry about that.

SB-SYS:*SHARED-OBJECTS* holds a list of all shared objects loaded.
This includes stuff loaded bu CFFI.

SB-ALIEN::SHARED-OBJECT-PATHNAME is the result of calling (PATHNAME
...) on the argument to LOAD-SHARED-OBJECT.

SB-ALIEN::SHARED-OBJECT-NAMESTRING is the native namestring that
dlopen() was called with to load that so. (If you want the lisp
pathname, use NATIVE-PATHNAME to parse it.)

SHARED-OBJECT-PATHNAME is only informational. S-O-NAMESTRING is what
is actually used by dlopen() when an image is reinitialized.

That's not too bad. Now comes the nasty bit.

To get the actual absolute pathname of the .so you have to duplicate
the searching behaviour of dlopen(), which is somewhat platform
dependent, and also depends on the current working directory at the
time the library was loaded.

For 90% of cases just searching LD_LIBRARY_PATH likely sufficient and
correct -- if that works for your case, good. If not, read the dlopen
docs carefully, and see what other environment variables or hardcoded
locations are involved.

If you find you have something loaded using current working directory,
you're really out of luck right now -- either you have to alter SBCL
to save the that information when the .so is loaded, or perhaps
preferably, alter the code that does so to do something else (eg. use
an absolute path.)

> Also, how do I adjust the lisp image to find said libraries (suppose
> in the same path as the executable) when the executable restarts? I
> can use LD_LIBRARY_PATH, but I'd prefer if I could somehow tell the
> image just before it dies a new set of paths to search whenever a
> request to load a library shows up.

Something like this, (untested):

(defun fix-shared-object (obj new-pathname)
  (declare (pathname new-pathname))
  (setf (sb-alien::shared-object-pathname obj) new-pathname)
  (setf (sb-alien::shared-object-namestring obj)
	(native-namestring (translate-logical-pathname new-pathname)
			   :as-file t))
  obj)

Cheers,

 -- Nikodemus

Peter Keller <psilord@...> writes:

> How can I find the absolute paths of all the loaded shared libraries
> in a running lisp image?
>
> ...
>
> How would I reliably do these things?  I'm happy to live with an SBCL
> only solution.

if a linux-only solution is acceptable, you can just parse
/proc/<PID>/maps to get the precise info, no need to second-guess
dlopen() and friends.

cheers,
--m

Michael Livshin <gmane@...> writes:

> Peter Keller <psilord@...> writes:
>
>> How can I find the absolute paths of all the loaded shared libraries
>> in a running lisp image?
>>
>> ...
>>
>> How would I reliably do these things?  I'm happy to live with an SBCL
>> only solution.
>
> if a linux-only solution is acceptable, you can just parse
> /proc/<PID>/maps to get the precise info, no need to second-guess
> dlopen() and friends.

Likewise, on Solaris, the pldd(1) command can be used to list the paths
of all shared objects loaded into a running process.

-- 
				Jim Wise
				jwise@...

On Wed, Sep 15, 2010 at 11:16:56AM +0200, Michael Livshin wrote:
> Peter Keller <psilord@...> writes:
> 
> > How can I find the absolute paths of all the loaded shared libraries
> > in a running lisp image?
> >
> > ...
> >
> > How would I reliably do these things?  I'm happy to live with an SBCL
> > only solution.
> 
> if a linux-only solution is acceptable, you can just parse
> /proc/<PID>/maps to get the precise info, no need to second-guess
> dlopen() and friends.

There is somewhat of an impedance mismatch between the information found in 
/proc/PID/maps, and that known by SBCL.

Suppose we take librt.so as an example.

SBCL (via sb-sys:*shared-objects*) knows the name of that library
as "librt.so" and that's how it will try to load it. However, the
/proc/PID/maps entry for it is "/usr/lib/librt-2.5.so". So while I'd be
able to get the actual contents of the file by searching /proc/PID/maps, I
wouldn't get the logical symlink name for it, only the physical filename.
Due to this, if I copy /usr/lib/librt-2.5.so to the current working directory
it won't matter since the lisp image will look for librt.so.

Thank you.

-pete

On 15 September 2010 18:02, Peter Keller <psilord@...> wrote:

> There is somewhat of an impedance mismatch between the information found in
> /proc/PID/maps, and that known by SBCL.
>
> Suppose we take librt.so as an example.
>
> SBCL (via sb-sys:*shared-objects*) knows the name of that library
> as "librt.so" and that's how it will try to load it. However, the
> /proc/PID/maps entry for it is "/usr/lib/librt-2.5.so". So while I'd be
> able to get the actual contents of the file by searching /proc/PID/maps, I
> wouldn't get the logical symlink name for it, only the physical filename.
> Due to this, if I copy /usr/lib/librt-2.5.so to the current working directory
> it won't matter since the lisp image will look for librt.so.

It's whack-a-mole.

I think you can get by by considering only those objects whose map
names have a subsequence matching the pathname-name.pathname-type of
stuff in sb-sys:*shared-objects*, and passing the resolved name (in
the new location) to fix-shared-object.

Of couse, if you have libfoo.so resolving to libbar-1.0.so, then the
maps approach is pretty useless for your purposes.

Cheers,

 -- Nikodemus

On Wed, Sep 15, 2010 at 11:19:16AM +0300, Nikodemus Siivola wrote:
> On 15 September 2010 06:11, Peter Keller <psilord@...> wrote:
> 
> > How can I find the absolute paths of all the loaded shared libraries
> > in a running lisp image? I'm writing an application which, before it
> > calls save-lisp-and-die, needs to copy all loaded shared libraries to
> > the current working directory of the lisp image. This is so the shared
> > objects it needs can be moved along with the executable to another
> > machine for execution. A constraint is that I'm not going to know a
> > priori the set of loaded shared objects until just before I call
> > save-lisp-and-die and those libraries may have been loaded via cffi
> > or some other means.
> 
> This is going to be ugly, and is technically unsupported right now.
> Sorry about that.

Thank you for the help! I appreciate it very much. The purpose of my
query is that I've written a master/slave library meant for distributed
computing and I want to make the production of the application binary
(and associated libraries) as simple and complete as possible for the
developer.

> To get the actual absolute pathname of the .so you have to duplicate
> the searching behaviour of dlopen(), which is somewhat platform
> dependent, and also depends on the current working directory at the
> time the library was loaded.

> For 90% of cases just searching LD_LIBRARY_PATH likely sufficient and
> correct -- if that works for your case, good. If not, read the dlopen
> docs carefully, and see what other environment variables or hardcoded
> locations are involved.

This is the hard part if only because there isn't a premade library
to parse ELF executables and I'd have to use (which I presume exists
though I don't see it in the SBCL manual) a native interface for reading
environment variables in SBCL.

I could write enough of an ELF file format parser to get what I need
out of the SBCL executable itself. I'm not too worried about that. The
ELF portion I really have to pay attention to is DT_RPATH, since that is
a pretty commonly used feature.

Hmm... loading the /etc/ld.so.cache file would probably go pretty far in
giving me the information I need concerning mapping bare library names
to pathnames. I will explore this one first since it seems so promising.
Google seems strangely silent on the file format for that file....

> If you find you have something loaded using current working directory,
> you're really out of luck right now -- either you have to alter SBCL
> to save the that information when the .so is loaded, or perhaps
> preferably, alter the code that does so to do something else (eg. use
> an absolute path.)

I'm going to bail on this problem since it doesn't come up that much in
practice. Once the executable has been saved, it is very unlikely in my
use cases that save-lisp-and-die will be called in the binary.

> Something like this, (untested):
> 
> (defun fix-shared-object (obj new-pathname)
>   (declare (pathname new-pathname))
>   (setf (sb-alien::shared-object-pathname obj) new-pathname)
>   (setf (sb-alien::shared-object-namestring obj)
> 	(native-namestring (translate-logical-pathname new-pathname)
> 			   :as-file t))
>   obj)

I was doing something similar, but not as correct. :)

Thanks!

-pete

On 15 September 2010 19:08, Peter Keller <psilord@...> wrote:

> This is the hard part if only because there isn't a premade library
> to parse ELF executables and I'd have to use (which I presume exists
> though I don't see it in the SBCL manual) a native interface for
> reading environment variables in SBCL.

Binary-types comes with an example ELF parser, actually. Not sure how
complete. See:

  http://common-lisp.net/cgi-bin/viewcvs.cgi/binary-types/?root=movitz

To access environment variables, use SB-EXT:POSIX-GETENV.

Cheers,

 -- Nikodemus

On Wed, Sep 15, 2010 at 07:22:18PM +0300, Nikodemus Siivola wrote:
> On 15 September 2010 19:08, Peter Keller <psilord@...> wrote:
> 
> > This is the hard part if only because there isn't a premade library
> > to parse ELF executables and I'd have to use (which I presume exists
> > though I don't see it in the SBCL manual) a native interface for
> > reading environment variables in SBCL.
> 
> Binary-types comes with an example ELF parser, actually. Not sure how
> complete. See:
> 
>   http://common-lisp.net/cgi-bin/viewcvs.cgi/binary-types/?root=movitz
> 
> To access environment variables, use SB-EXT:POSIX-GETENV.

I have decent documentation on the ELF format for i386 and x86_64 so
I can add more to that example if needed. Binary-types is kind of a
cool library...

Thanks for the knowledge!

-pete

On Wed, Sep 15, 2010 at 11:19:16AM +0300, Nikodemus Siivola wrote:
> This is going to be ugly, and is technically unsupported right now.
> Sorry about that.

I have a minor question:

Is there an API in SBCL that I can use to determine if the executable that
is my SBCL lisp image was compiled 32-bit or 64-bit?

Thank you.

-pete

On Wed, Sep 15, 2010 at 11:19:16AM +0300, Nikodemus Siivola wrote:
[snip]
> To get the actual absolute pathname of the .so you have to duplicate
> the searching behaviour of dlopen(), which is somewhat platform
> dependent, and also depends on the current working directory at the
> time the library was loaded.

[snip]

> (defun fix-shared-object (obj new-pathname)
>   (declare (pathname new-pathname))
>   (setf (sb-alien::shared-object-pathname obj) new-pathname)
>   (setf (sb-alien::shared-object-namestring obj)
> 	(native-namestring (translate-logical-pathname new-pathname)
> 			   :as-file t))
>   obj)

The above ideas coupled with parsing the output of /sbin/ldconfig -p
has provided a reliable enough method to find and preserve the shared
libraries used by sb-sys:*shared-objects* for linux on x86 or x86-64
architectures.

Thank you *very* much for your suggestions!

Later,
-pete