From: Mantis B. T. <no...@bu...> - 2012-06-25 09:58:14
|
A NOTE has been added to this issue. ====================================================================== http://bugs.bacula.org/view.php?id=1897 ====================================================================== Reported By: peckel Assigned To: marcovw ====================================================================== Project: bacula Issue ID: 1897 Category: Director Reproducibility: always Severity: block Priority: urgent Status: assigned ====================================================================== Date Submitted: 2012-06-24 21:00 BST Last Modified: 2012-06-25 10:58 BST ====================================================================== Summary: Bacula Release 5.2.9 breaks IPv6 connectivity Description: When I tried to connect to my bacula-dir with bconsole this morning, I noticed that the connection could not be established, session hangs for a long time. All my bacula instances are running IPv6 only. I recently upgraded via yum from 5.2.6 to 5.2.9, at which time the error must have been introduced. Not only the connection from bconsole to bacula-dir is affected, but the bacula-dir also cannot access bacula-fd running on 5.2.9 - same problem, connection cannot be etsablished. Every time the bacula-dir is restarted, an error message is sent via mail and to the error log: 24-Jun 20:25 orcus-dir: Warning: Cannot bind port 9101: ERR=Address already in use: Retrying ... The listener port is set up correctly and a connection can be established, the daemon (bacula-dir and bacula-fd) just doesn't answer. Steps to Reproduce: - Install bacula 5.2.9 (complete bacula-postgresql package, fd, sd, dir and bconsole from the CentOS 5 repo at Dass-IT) - Configure the bacula-dir and bconsole to use IPv6 only: bacula-dir.conf: Director { Name = orcus-dir DirAddresses = { ipv6 = { addr = 2001:16e0:101:c00::b00:0; port = 9101 } } QueryFile = "/usr/lib64/bacula/query.sql" WorkingDirectory = "/var/lib/bacula" PidDirectory = "/var/run" Maximum Concurrent Jobs = 1 Password = "xxxx" Messages = daemon } ... bconsole.conf: Director { Name = orcus-dir DIRport = 9101 address = orcus.hindenburgring.com Password = "=MkJeUo02O.QNY_W" } ... Try to connect. Double-checked: - With 5.2.6 and the same configuration files, connection works without a problem. - Reconfiguration to use IPv4 also 'fixes' the problem. Additional Information: The problem could be reproduced with bacula-fd as well, the 5.2.9 FDs fail, 5.2.6 works fine. Some of the FDs are running on CentOS 6.2, Ubuntu 10.04 LTS and Mac OS X 10.7, Director and Storage Daemon on CentOS 5.8. The problem only occurs on 5.2.9 with IPv6. A pointer to bacula-postgresql RPMs for Version 5.2.6 would be extremely welcome. At the moment, my backup system is down. ====================================================================== ---------------------------------------------------------------------- (0006409) kern (administrator) - 2012-06-25 07:39 http://bugs.bacula.org/view.php?id=1897#c6409 ---------------------------------------------------------------------- I am sorry to hear you are having problems. There were changes in the code resolver code for IP addresses between 5.2.6 and 5.2.9, and I have asked the author to take a look at it. We will see. Sometime this later this week we will be releasing version 5.2.10, and either we will have a solution or we will back out the changes. The major problem for us is that we don't run IPv6 so we don't test it. If you have some pointers on how to install it and make it co-exist with IPv4, please let me know as it may avoid future problems. Concerning your installation, possible solutions: 1. Wait until we put out a new version and find rpms on the net, as it is unlikely CentOS will build a new version in the near future. 2. See with CentOS about installing older rpms. 3. Find older rpms on the net for your OS. 4. Download the 5.2.6 source from Source Forge and build the rpms yourself (using either the community furnished spec files or the spec files from 5.2.9) 5. Build new binaries from 5.2.6 source. 6. Get a Bacula Systems subscription, where we don't generally have these problems, and will supply new binaries in the case that we do -- of course this is a paying solution ... ---------------------------------------------------------------------- (0006410) marcovw (developer) - 2012-06-25 09:41 http://bugs.bacula.org/view.php?id=1897#c6410 ---------------------------------------------------------------------- The 24-Jun 20:25 orcus-dir: Warning: Cannot bind port 9101: ERR=Address already in use: Retrying ... Seems to indicate that you have duplicate addresses in your config of something is resolving to duplicate addresses. When the cannot bind happens it keeps on retrying and the daemon never gets ready for service so that might explain why you cannot connect. The fist steps would be the obvious ones: - output from bacula-dir -d 100 to have some debugging on what it wants to bind too etc. - output of bconsole -d 100 to see if the bconsole tries to connect to the correct instance. - netstat -an | grep 9101 to see if it binds at all and at what address. We can take it from there. ---------------------------------------------------------------------- (0006411) peckel (reporter) - 2012-06-25 10:58 http://bugs.bacula.org/view.php?id=1897#c6411 ---------------------------------------------------------------------- There is no duplicate address involved, though the message might indicate so. After a downgrade to 5.2.6 (I did this after opening the change, as you suggested later on :-)), everything works fine without further changes. There seems to be a problem in the IP socket allocation code for IPv6 that was introduced between 5.2.6 and 5.2.9. What's interesting as well is that the bacula-dir (and the bacula-fd, as far as I can see) open the listener port correctly as indicated by netstat, but don't send out any packages through it. I can connect via telnet to port 9101, but there is never any answer. I am using the repositories linked here <http://www.dass-it.de/news/download/dassit-bacula-rpm-repositories>, specifically <http://download.opensuse.org/repositories/home:dassit:bacula/CentOS_CentOS-5/home:dassit:bacula.repo> and <http://download.opensuse.org/repositories/home:dassit:bacula/CentOS_CentOS-6/home:dassit:bacula.repo> As for enabling IPv6, there is nothing much to be done on a recent operating system. For example, CentOS 5/6 already have IPv6 enabled by default, as you can see in ifconfig: eth0 Link encap:Ethernet HWaddr 00:16:35:06:16:BD inet addr:192.168.42.11 Bcast:192.168.42.255 Mask:255.255.255.0 inet6 addr: fe80::216:35ff:fe06:16bd/64 Scope:Link inet6 addr: 2001:16e0:101:c00::b00:0/64 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:407412022 errors:0 dropped:0 overruns:0 frame:0 TX packets:277820712 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:473186707500 (440.6 GiB) TX bytes:463697439671 (431.8 GiB) If you did not enable IPv6 on the routers inside your network, you will probably not see a 'Scope:Global' address. However, a link-local address (starting with FE80::, marked 'Scope:Link') will always be available, and that should be enough for testing. Just edit the bacula-dir config so that the 'DIRaddresses' instance matches mine (using your link local address instead of my global one, of course), and modify the bconsole-Setup's 'Director' instance to look like this: Director { Name = your-dir DIRport = 9101 address = fe80::216:35ff:fe06:16bd # change to your link-local address! Password = "your-password" } After restarting the director, you should see the link-local address as the listener address for the director with 'netstat -anTtup', like that: tcp 0 0 fe80::216:35ff:fe06:16bd:9101 :::* LISTEN 27685/bacula-dir I did not get a debug level 100 log, but I have a higher level one ... I'll upload it in a second. Issue History Date Modified Username Field Change ====================================================================== 2012-06-24 21:00 peckel New Issue 2012-06-25 07:39 kern Note Added: 0006409 2012-06-25 07:39 kern Status new => acknowledged 2012-06-25 09:41 marcovw Note Added: 0006410 2012-06-25 10:24 marcovw Assigned To => marcovw 2012-06-25 10:24 marcovw Status acknowledged => assigned 2012-06-25 10:58 peckel Note Added: 0006411 ====================================================================== |