[Bacula-bugs] [bacula 0000403]: Statically linked agent crashes on second invocation

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

A BUGNOTE has been added to this bug.
======================================================================
http://bugs.bacula.org/bug_view_advanced_page.php?bug_id=0000403
======================================================================
Reported By:                psjoberg
Assigned To:                
======================================================================
Project:                    bacula
Bug ID:                     403
Category:                   File Daemon
Reproducibility:            always
Severity:                   crash
Priority:                   normal
Status:                     feedback
======================================================================
Date Submitted:             08-16-2005 13:47 PDT
Last Modified:              09-18-2005 09:35 PDT
======================================================================
Summary:                    Statically linked agent crashes on second invocation
Description: 
On one client that I run a statically linked bacula-fd it always crashes
when it's about to start the second job.
The client is running "Mandrake Linux release 10.0 (Official) for i586"
with all available patches installed.

The statically linking is done on "Mandrakelinux release 10.1 (Official)
for
i586" with all available patches installed.

To reproduce the problem I start up the client fd, run a job, when it's
done
I start another one.

The same thing also happens on a system that runs mdk 10.2 so a mdk 10.1
statically linked bacula-fd crashes on both a mdk 10.0 and a 10.2.

======================================================================

----------------------------------------------------------------------
 kern - 08-18-2005 08:03 PDT 
----------------------------------------------------------------------
To do anything about it, I will need a traceback of the FD that crashes
(yar-fd).  You might be able to get one my fixing the installation as it
appears that /opt/bacula/sbin/btraceback doesn't exist. However, IMO, the
simplest way to get a traceback is to manually run the debugger as
described in the Kaboom chapter of the manual, then attach the traceback
output to this bug report as a .txt file.

----------------------------------------------------------------------
 kern - 09-04-2005 10:19 PDT 
----------------------------------------------------------------------
Thank you for your output, but what I need is the output from the client
program run under the debugger as documented in the Kaboom chapter of the
manual.

----------------------------------------------------------------------
 psjoberg - 09-05-2005 15:17 PDT 
----------------------------------------------------------------------
I installed gdb and tried to generate a backtrace but I guess since it's
statically linked it doesn't work.
The traceback mail

(no debugging symbols found)
Using host libthread_db library "/lib/tls/libthread_db.so.1".
Using host libthread_db library "/lib/tls/libthread_db.so.1".
0x081e1eaa in ?? ()
Using host libthread_db library "/lib/tls/libthread_db.so.1".
/opt/bacula/btraceback.gdb:1: Error in sourced command file:
No symbol table is loaded.  Use the "file" command.

running interactive gdb
(gdb) run -s -f -c /opt/bacula/etc/bacula-fd-troy.conf
Starting program: /opt/bacula/sbin/bacula-fd -s -f -c
/opt/bacula/etc/bacula-fd-troy.conf
warning: shared library handler failed to enable breakpoint

Program received signal SIG32, Real-time event 32.
0x08113333 in ?? ()
(gdb)

dir "c" a few times and it ended up with

Program terminated with signal SIGKILL, Killed.
The program no longer exists.
(gdb) thread apply all bt
(gdb)

I tried to recompile the latest static agent but now compiling static
fails
...
==>Entering directory /home/peters/src/pubcvs/bacula/bacula/src/filed
make[1]: Entering directory
`/usr/local/src/peters/pubcvs/bacula/bacula/src/filed'
/usr/bin/g++   -static -L../lib -L../findlib -o static-bacula-fd filed.o
authenticate.o acl.o backup.o chksum.o estimate.o filed_conf.o heartbeat.o
job.o pythonfd.o restore.o status.o verify.o verify_vol.o \
    -lacl -lz              -lfind -lbac -lm -L/usr/lib/python2.3/config
-lpython2.3 -lutil -lrt  -lpthread -ldl  -lwrap -lintl \
     -lssl -lcrypto
/usr/lib/python2.3/config/libpython2.3.a(dynload_shlib.o)(.text+0x189): In
function `_PyImport_GetDynLoadFunc':
Python/dynload_shlib.c:129: warning: Using 'dlopen' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
/usr/lib/gcc/i586-mandrake-linux-gnu/3.4.1/../../../libc.a(dcigettext.o)(.text+0x6c0):
In function `_nl_find_msg':
: multiple definition of `_nl_find_msg'
....
....
....
/home/buchan/rpm/BUILD/acl-2.2.23/libacl/__acl_to_any_text.c:338: warning:
Using 'getgrgid' in statically linked applications requires at runtime the
shared libraries from the glibc version used for linking
/usr/lib/gcc/i586-mandrake-linux-gnu/3.4.1/../../../libacl.a(acl_from_text.o)(.text+0x62e):
In function `acl_from_text':
/home/buchan/rpm/BUILD/acl-2.2.23/libacl/acl_from_text.c:171: warning:
Using 'getgrnam' in statically linked applications requires at runtime the
shared libraries from the glibc version used for linking
/usr/lib/gcc/i586-mandrake-linux-gnu/3.4.1/../../../libwrap.a(options.o)(.text+0x495):
In function `group_option':
: warning: Using 'endgrent' in statically linked applications requires at
runtime the shared libraries from the glibc version used for linking
/usr/lib/gcc/i586-mandrake-linux-gnu/3.4.1/../../../libacl.a(acl_from_text.o)(.text+0x52f):
In function `acl_from_text':
/home/buchan/rpm/BUILD/acl-2.2.23/libacl/acl_from_text.c:155: warning:
Using 'getpwnam' in statically linked applications requires at runtime the
shared libraries from the glibc version used for linking
/usr/lib/gcc/i586-mandrake-linux-gnu/3.4.1/../../../libacl.a(__acl_to_any_text.o)(.text+0x6a5):
In function `__acl_to_any_text':
/home/buchan/rpm/BUILD/acl-2.2.23/libacl/__acl_to_any_text.c:326: warning:
Using 'getpwuid' in statically linked applications requires at runtime the
shared libraries from the glibc version used for linking
/usr/lib/gcc/i586-mandrake-linux-gnu/3.4.1/../../../libwrap.a(options.o)(.text+0x53d):
In function `user_option':
: warning: Using 'endpwent' in statically linked applications requires at
runtime the shared libraries from the glibc version used for linking
/usr/lib/gcc/i586-mandrake-linux-gnu/3.4.1/../../../libwrap.a(hosts_access.o)(.text+0x993):
In function `string_match':
: warning: Using 'getaddrinfo' in statically linked applications requires
at runtime the shared libraries from the glibc version used for linking
/usr/lib/gcc/i586-mandrake-linux-gnu/3.4.1/../../../libwrap.a(socket.o)(.text+0x43):
In function `gethostbyname_dot':
: warning: Using 'gethostbyname' in statically linked applications
requires at runtime the shared libraries from the glibc version used for
linking
../lib/libbac.a(bnet.o)(.text+0x6ff): In function `resolv_host':
/home/peters/src/pubcvs/bacula/bacula/src/lib/bnet.c:652: warning: Using
'gethostbyname2' in statically linked applications requires at runtime the
shared libraries from the glibc version used for linking
../lib/libbac.a(address_conf.o)(.text+0x924): In function `add_address':
/home/peters/src/pubcvs/bacula/bacula/src/lib/address_conf.c:303: warning:
Using 'getservbyname' in statically linked applications requires at runtime
the shared libraries from the glibc version used for linking
collect2: ld returned 1 exit status
make[1]: *** [static-bacula-fd] Error 1

edited on: 09-05-05 15:17

----------------------------------------------------------------------
 kern - 09-06-2005 08:31 PDT 
----------------------------------------------------------------------
Hmmm. Sorry to see you having so many problems. 

It looks like the debugging symbols were stripped from the binary, which
makes debugging pretty much impossible. The normal Bacula build process
does not strip debug symbols, so could you please look closely at how it
was built and ensure that the binary is not stripped.

Concerning the linking problems: I suspect that you have too many options
turned on in your ./configure and not all the necessary static libraries
loaded. Try simplifying your ./configure down to the bare essentials -- in
particular, make sure that TLS (openssl) and Python are both turned off
(default behavior).

----------------------------------------------------------------------
 psjoberg - 09-09-2005 07:07 PDT 
----------------------------------------------------------------------
Just a quick update, to show that I'm working on this. 
The build procedure I originally used can be found at
http://www.techwiz.ca/~peters/bacula/build  DoConfigure-static-all.sh with
the configure output in DoConfigure-static-all-i686.log  and the make
output in DoConfigure-static-make-i686.log 
The build script includes
   CFLAGS="-g -O2 -Wall -march=pentium"  ./configure ...
so I added the symboles and I don't do a "strip" anywhere in my script.

I'm still trying to build a static version but whatever I done so far it
always fails.
Latest issue is 

/home/peters/src/pubcvs/bacula/bacula/src/lib/address_conf.c:303: warning:
Using 'getservbyname' in statically linked applications requires at runtime
the shared libraries from the glibc version used for linking
/usr/lib/gcc/i586-mandrake-linux-gnu/3.4.1/../../../libc.a(dcigettext.o)(.text+0x6c0):
In function `_nl_find_msg':
: multiple definition of `_nl_find_msg'
/usr/lib/gcc/i586-mandrake-linux-gnu/3.4.1/../../../libintl.a(dcigettext.o)(.text+0x210):
first defined here
/usr/bin/ld: Warning: size of symbol `_nl_find_msg' changed from 1482 in
/usr/lib/gcc/i586-mandrake-linux-gnu/3.4.1/../../../libintl.a(dcigettext.o)
to 1239 in
/usr/lib/gcc/i586-mandrake-linux-gnu/3.4.1/../../../libintl.a(dcigettext.o)
collect2: ld returned 1 exit status
make[1]: *** [static-bacula-fd] Error 1
make[1]: Leaving directory
`/usr/local/src/peters/pubcvs/bacula/bacula/src/filed'

----------------------------------------------------------------------
 psjoberg - 09-18-2005 09:35 PDT 
----------------------------------------------------------------------
got it to link at last:
1: src/filed/Makefile.in contaings "strip" for static-bacula-fd:
2: In order for it to link I had to remove " -lintl" from LIBS, at least
for static-bacula-fd

After that I ran a incremental job twice and the second run gave the
following output. It's a little more but I don't know if it helps, it
contains some "No symbol table info available", how do I get that in?

Using host libthread_db library "/lib/tls/libthread_db.so.1".
Using host libthread_db library "/lib/tls/libthread_db.so.1".
0x080c739a in select ()
Using host libthread_db library "/lib/tls/libthread_db.so.1".
$1 = "troy-fd", '\0' <repeats 22 times>
$2 = 0x816c378 "static-bacula-fd"
$3 = 0x816c3a8 "/opt/bacula/sbin/static-bacula-fd"
$4 = "Internal"
$5 = 0x80ff780 "1.37.39 (17 September 2005)"
$6 = 0x80f7f54 "i686-pc-linux-gnu"
$7 = 0x80f7f4b "mandrake"
$8 = 0x80f7f47 "for"
http://bugs.bacula.org/bug_view_advanced_page.php?bug_id=0000000  0x080c739a in select ()
http://bugs.bacula.org/bug_view_advanced_page.php?bug_id=0000001  0x00000005 in ?? ()
http://bugs.bacula.org/bug_view_advanced_page.php?bug_id=0000002  0x00000000 in ?? ()
http://bugs.bacula.org/bug_view_advanced_page.php?bug_id=0000003  0x08129c0c in seen_objects ()
http://bugs.bacula.org/bug_view_advanced_page.php?bug_id=0000004  0x00000000 in ?? ()
http://bugs.bacula.org/bug_view_advanced_page.php?bug_id=0000005  0x0806522e in bnet_thread_server (addrs=0x816d1b0, max_clients=20, 
    client_wq=0x8123020, 
    handle_client_request=0x804c2a0 <handle_client_request(void*)>)
    at bnet_server.c:148
http://bugs.bacula.org/bug_view_advanced_page.php?bug_id=0000006  0x080487be in main (argc=135713336, argv=0x0) at filed.c:240
http://bugs.bacula.org/bug_view_advanced_page.php?bug_id=0000000  0x080c739a in select ()
No symbol table info available.
http://bugs.bacula.org/bug_view_advanced_page.php?bug_id=0000001  0x00000005 in ?? ()
No symbol table info available.
http://bugs.bacula.org/bug_view_advanced_page.php?bug_id=0000002  0x00000000 in ?? ()
No symbol table info available.
http://bugs.bacula.org/bug_view_advanced_page.php?bug_id=0000003  0x08129c0c in seen_objects ()
No symbol table info available.
http://bugs.bacula.org/bug_view_advanced_page.php?bug_id=0000004  0x00000000 in ?? ()
No symbol table info available.
http://bugs.bacula.org/bug_view_advanced_page.php?bug_id=0000005  0x0806522e in bnet_thread_server (addrs=0x816d1b0, max_clients=20, 
    client_wq=0x8123020, 
    handle_client_request=0x804c2a0 <handle_client_request(void*)>)
    at bnet_server.c:148
148           if ((stat = select(maxfd + 1, &sockset, NULL, NULL, NULL)) <
0) {
Current language:  auto; currently c++
be = {<SMARTALLOC> = {<No data fields>}, buf_ = 0x0, berrno_ = 0}
clilen = 16
turnon = 1
p = (IPADDR *) 0x0
buf =
"66.11.169.226\000\022\b8Ò\026\b\203k\017\b8úÿ¿´¼\a\bt:\022\b\000\000\000\0008Ò\026\b8Ò\026\b\000\000\000\000èÄ\026\bhúÿ¿ñ\032\006\bd:\022\b
:\022\b\000\000\000\000 \017\022\b\203k\017\bhúÿ¿» \a\b
\017\022\bèÄ\026\b8Ò\026\b\210úÿ¿ØK\a\bd:\022\bèÄ\026\b\230úÿ¿WR\a\b"
allbuf = '\0' <repeats 676 times>,
"µâ\017\bà]\020\b\n\000\000\0008øÿ¿Ïz\t\bLøÿ¿µâ\017\b\006\000\000\000µâ\017\bôóÿ¿\000\000\000\000\000\000\000\000\216\002\020\b`_\020\b\020\000\000\000høÿ¿¨â\017\bà]\020\b\n\000\000\000xøÿ¿Ïz\t\b\214øÿ¿¨â\017\b\004\000\000\000¨â\017\b4ôÿ¿",
'\0' <repeats 54 times>, "d ", '\0' <repeats 48 times>,
"\216#\000\000\000\000\000\000\000\000\000\000\002\000\000\000\003\000\000\000¨ùÿ¿$øÿ¿\000\000\000\000â\017\b(\000\000\000ÿÿÿÿ\000\000\000\0009\002\020\bÔôÿ¿\000\000
000\000\000\000\000\000"...
stat = 6
tlog = 0
fd_ptr = (s_sockfd *) 0x0
sockfds = {<SMARTALLOC> = {<No data fields>}, head = 0xbfffee20, 
  tail = 0xbfffee20, loffset = 0, num_items = 1}
newsockfd = 6
cli_addr = {sa_family = 2, sa_data =
"ÁÞB\v©â\000\000\000\000\000\000\000"}
http://bugs.bacula.org/bug_view_advanced_page.php?bug_id=0000006  0x080487be in main (argc=135713336, argv=0x0) at filed.c:240
240           bnet_thread_server(me->FDaddrs, me->MaxConcurrentJobs,
&dir_workq, handle_client_request);
p = (IPADDR *) 0x0
http://bugs.bacula.org/bug_view_advanced_page.php?bug_id=0000000  0x00000000 in ?? ()
No symbol table info available.

Bug History
Date Modified  Username       Field                    Change              
======================================================================
08-16-05 13:47 psjoberg       New Bug                                      
08-16-05 13:47 psjoberg       File Added: AgentCrash.tar.bz2                    
08-18-05 08:03 kern           Bugnote Added: 0001112                       
08-18-05 08:03 kern           Status                   new => feedback     
09-04-05 10:19 kern           Bugnote Added: 0001139                       
09-05-05 15:00 psjoberg       Bugnote Added: 0001141                       
09-05-05 15:17 psjoberg       Bugnote Edited: 0001141                      
09-06-05 08:31 kern           Bugnote Added: 0001144                       
09-09-05 07:07 psjoberg       Bugnote Added: 0001153                       
09-18-05 09:35 psjoberg       Bugnote Added: 0001172                       
======================================================================