Re: [TCLCORE] GSoC ShlibMemLoad - Question about the RTLD

[Daniel H. <da...@gm...>]

Hello John,

I would not want you to think I am pushy, but have you already
returned? We tried to contact Alexander Kabaev but we have not receive
any response from him. We would be grateful if you could take a look
at our code and make some suggestions of what may be wrong there.

Greetings,
Daniel

2008/7/12 Daniel Hans <da...@gm...>:
> Hello,
>
> I am working on a Google Summer of Code projects for the Tcl
> community. The main goal of this project is to allow loading dynamic
> libraries not only from files on disk but also from memory images. For
> this we want to use the RTLD library. Since the RTLD is like a part of
> the FreeBSD kernel, I made some changes there (for example I commented
> out some code) in order to compile it as an application on Linux.
> Anyway, it works for loading very simple libraries, but it does not
> work for other libraries (which are also simple).
>
> Since none of us is an expert in this area we asked for help John
> Polstra. However, he is going to be traveling for the next week and we
> are getting out of time (GSoC ends in a month), so he has given us
> your email as you have worked with this code as well.
>
> More details are described below. Could you be so kind to take a look
> at our code and make a guess what may be wrong?
>
> Here is a link to a project repository:
> http://svn.assembla.com/svn/gsoc2008-tcl_dynamic_libraries
> You can find rtld code in /trunk/rtld or in archive RTLD.zip in /trunk
>
> I will be grateful for any kind of help.
>
> Regards,
> Daniel
>
> 2008/7/12 John Polstra <jd...@po...>:
>> I'm glad you followed up, because I didn't receive Daniel's mail.  More
>> about that later.
>>
>> I'd be happy to help with this, but I am going to be traveling for about a
>> week and can't look at it until I return.  If you can wait that long, fine;
>> if not, please contact one of the more recent maintainers of the FreeBSD
>> dynamic linker.  The person who has worked on it the most in the past 5
>> years is Alexander Kabaev <ka...@fr...>.  He no doubt knows the code
>> better than I do, because I haven't looked at it for many years.
>>
>> Daniel, it looks like my mail system discarded your mail because it had an
>> attachment containing a zip file.  I accept .tar.gz files, but not zip
>> files.  Sorry for the inconvenience.  In my world, zip files are practically
>> always Windows viruses.
>>
>> John
>>
>> Andreas Kupries wrote:
>>>
>>> I hope nobody gets upset for me butting in here for a bit.
>>>
>>>> Hello,
>>>>
>>>> I was given your email address from Arnulf Wiedemann who is a member
>>>> of the Tcl community.
>>>>
>>>> I am currently working on an Google Summer of Code project and
>>>> writing to you
>>>> because you are an author of the RTLD for FreeBSD.
>>>>
>>>> The main object of my project is to allow loading libraries
>>>> dynamically from memory. Normally if we want to load a library from a
>>>> file we can just use dlopen(), but I have to load libraries which are
>>>> mapped in memory.
>>>
>>> For a bit more context, Tcl is (of course) just fine with loading shared
>>> libraries (containing binary Tcl packages) from disk, using dlopen().
>>>
>>> Tcl however also has a feature called 'virtual filesystems' and uses this
>>> feature to enable the generation of single-file executables.
>>>
>>> Which is in essence a Tcl interpreter with a virtual filesystem image of
>>> some type attached to it. When such an executable is run the interpreter
>>> pulls the files it needs out of this embedded filesystem. At that point
>>> the
>>> files exist as blobs in memory. Tcl scripts can then be executed directly
>>> out of such blobs. Our GSoC project is about making this possible for
>>> shared
>>> libraries as well. The only platform there this is currently possible is
>>> OS
>>> X, which has direct OS support for this type of thing. On all other
>>> platforms the blob currently is (has to be) written to a temporary file on
>>> disk which is then handed to dlopen() to make it usable.
>>>
>>> Regarding all the people in the recipient list
>>>
>>>        Daniel H. is the student tasked with implementing this feature.
>>>
>>>        I (Andreas) am the primary mentor with advice on the Tcl side of
>>> things.
>>>
>>>        Daniel S. is the person who implemented the above-mentioned
>>> load-from-memory feature for OS X and is my backup mentor.
>>>
>>>        Tomasz is a mentor as well, his task is to watch for and help us in
>>> case of
>>> language/translation trouble cropping up.
>>>        He is Polish as is Daniel H.
>>>
>>>        Tcl-Core @ SF is the primary mailing list for development on the
>>> Tcl core.
>>>
>>>
>>>> My first thought was to modify some functions from glibc, but glibc is
>>>> GPL and this license is not appropriate for my project. Then I found
>>>> the RLTD library and thought of making some changes in its code. I am
>>>> not working with the latest version. The one I use can be found here
>>>> http://opengrok.creo.hu/dragonfly/xref/src/libexec/rtld-elf/.
>>>
>>> While Daniel attached the RTLD sources he used, the whole project can also
>>> be retrieved from a svn repository, its location is
>>>
>>>        andreask@gila:~/workbench/shmemload/head> svn info
>>>        Path: .
>>>        URL: http://svn.assembla.com/svn/gsoc2008-tcl_dynamic_libraries
>>>        Repository UUID: d7ba3245-3e7b-42d0-9cd4-356c8b94b330
>>>        [...]
>>>
>>>
>>>> I have modified some functions to make this RTLD compilable as an
>>>> application on Linux. Now I just want the RTLD to let me load
>>>> libraries from regular files. The problem is that it is not always
>>>> working. I mean it works for some simple libraries, but it does not
>>>> work for other which are also very simple. For example if my function
>>>> I call by 'your' dlsym calls printf or some other functions from
>>>> stdio.h its execution causes segmentation faults.
>>>
>>>
>>> Some relevant mails already exchanged regarding the problem are in the
>>> TclCore archives.
>>>
>>> It is known that the memory-RTLD (*) currently loads a second copy of
>>> libc.so. It is not yet understood where and how this leads to the crash.
>>>
>>> Now we are hoping that we can gain your assistance and pick your brain
>>> regarding RTLD's internals.
>>>
>>>
>>>> For the last few days I have tried to find what is wrong, but I have
>>>> not came up with any solution. I think there may be some problems with
>>>> initializations. In your code there is a function '_rtld' which I do
>>>> not call. The only initialization work I do is to initialize obj_main
>>>> with some random data.
>>>
>>>> I would be extremely grateful if you could just take a look at my code
>>>> and tell me what may be not ok. This is mainly your code. I just
>>>> commented out some functions and added some log messages. Maybe I
>>>> commented something which is crucial or I missed some
>>>> initializations... To make the RTLD work is very important,
>>>> because now I am
>>>> stacked and cannot go on with the project, but I would really want to
>>>> complete it.
>>>> I would appreciate any kind of advice from you.
>>>>
>>>> I am looking forward to hearing from you.
>>>>
>>>> Regards,
>>>> Daniel
>>>>
>>>> Note1. I also added defs.h file which contains defines which you use
>>>> in your code. They are architecture depended and work on 32-bit
>>>> Intels.
>>>> Note2. If managed to compile this code by yourself you could test it
>>>> ./rtld /full_path_to_library_you_want_to_load
>>>
>>> --
>>>        Andreas Kupries <andreask@ActiveState.com>
>>>        Developer @ http://www.ActiveState.com
>>>        Tel: +1 778-786-1122
>>>
>>
>