Linux Containers

On Fri, Aug 24, 2012 at 7:00 AM, Dan Kegel <dank@...> wrote:
> Hi folks,
> has anyone written a  /usr/lib/lxc/templates/lxc-rhel yet?

AFAIK no. But see :
- http://www.mail-archive.com/lxc-users@...>
and http://www.mail-archive.com/lxc-users@...>
- http://wiki.1tux.org/wiki/Lxc/Installation/Guest/Centos/6

>
> Would it be reasonable to base one on rinse, as lxc-ubuntu is based on
> debootstrap?

IMHO "yum --installroot" is enough.

-- 
Fajar

On Thu, Aug 23, 2012 at 9:44 PM, Fajar A. Nugraha <list@...> wrote:
>> has anyone written a  /usr/lib/lxc/templates/lxc-rhel yet?
>
> AFAIK no. But see :
> - http://www.mail-archive.com/lxc-users@...>
> and http://www.mail-archive.com/lxc-users@...>
> - http://wiki.1tux.org/wiki/Lxc/Installation/Guest/Centos/6

I tried the one from Johannes in
http://yum.phx.singlehop.com/centos/5/os/x86_64/CentOS/centos-release-5-8.el5.centos.x86_64.rpm
Had to update it a bit, as URLs have changed.
It seems to create the image, but can't start it; lxc-start complains

lxc-start: failed to rename vethMR2TXx->eth0 : File exists
lxc-start: failed to setup netdev
lxc-start: failed to setup the network for 'demo.centos6'
lxc-start: failed to setup the container
lxc-start: invalid sequence number 1. expected 2
lxc-start: failed to spawn 'demo.centos6'

Did I miss something?

On Sat, Aug 25, 2012 at 4:26 AM, Dan Kegel <dank@...> wrote:
> lxc-start: failed to rename vethMR2TXx->eth0 : File exists

What does your config file looks like?
It looks like you're telling lxc to label the container's network
device in the HOST (lxc.network.veth.pair) as eth0. Which (obviously)
won't work, since the host already has an eth0.

-- 
Fajar

On Fri, Aug 24, 2012 at 6:23 PM, Fajar A. Nugraha <list@...> wrote:
> On Sat, Aug 25, 2012 at 4:26 AM, Dan Kegel <dank@...> wrote:
>> lxc-start: failed to rename vethMR2TXx->eth0 : File exists
>
> What does your config file looks like?
> It looks like you're telling lxc to label the container's network
> device in the HOST (lxc.network.veth.pair) as eth0. Which (obviously)
> won't work, since the host already has an eth0.

I'm not using any specific config file.
/etc/lxc/lxc.conf contains

lxc.network.type=veth
lxc.network.link=lxcbr0
lxc.network.flags=up

and I don't see any reference to eth0 there.
Aren't the default templates supposed to work with the default config file?

For what it's worth, the end of the log from
$ sudo lxc-create -n demo.centos6 -t centos -- -R 6
is:
Download complete.
Copy /var/cache/lxc/centos/i686/6/rootfs to
/var/lib/lxc/demo.centos6/rootfs ...
Copying rootfs to /var/lib/lxc/demo.centos6/rootfs ...setting root
passwd to rooter
container rootfs and config created
container is configured for lxc.network.type=veth and
lxc.network.link=virbr0 (which is default if you have libvirt runnig)
'centos' template installed
'demo.centos6' created

$ sudo lxc-start -n demo.centos6
then says
lxc-start: failed to rename vethA8XI49->eth0 : File exists

Uninstalling libvirt with "dpkg -r libvirt0 python-libvirt
libvirt-bin", and then repeating, didn't seem to change anything.
- Dan

p.s. Earlier, I'd been testing on a 64 bit system.
On my 32 bit system, I had to change the script's treatment of $arch,
see attachment.

On Sat, Aug 25, 2012 at 8:48 AM, Dan Kegel <dank@...> wrote:
> On Fri, Aug 24, 2012 at 6:23 PM, Fajar A. Nugraha <list@...> wrote:
>> On Sat, Aug 25, 2012 at 4:26 AM, Dan Kegel <dank@...> wrote:
>>> lxc-start: failed to rename vethMR2TXx->eth0 : File exists
>>
>> What does your config file looks like?
>> It looks like you're telling lxc to label the container's network
>> device in the HOST (lxc.network.veth.pair) as eth0. Which (obviously)
>> won't work, since the host already has an eth0.
>
> I'm not using any specific config file.
> /etc/lxc/lxc.conf contains
>
> lxc.network.type=veth
> lxc.network.link=lxcbr0
> lxc.network.flags=up
>
> and I don't see any reference to eth0 there.
> Aren't the default templates supposed to work with the default config file?

It should. Which is why I thought you edit the config file. Try:
- adding lxc.network.veth.pair=test-e0
- run "ip link show" on the host, and for every interface that starts
with a "veth", do "ip link del" on that interface. That is, assuming
you don't currently have any containers or anything that uses veth
running.
- try the second link, which create the rootfs and config file by hand
instead of using a script.

-- 
Fajar

On Fri, Aug 24, 2012 at 7:00 PM, Fajar A. Nugraha <list@...> wrote:
>> Aren't the default templates supposed to work with the default config file?
>
> It should. Which is why I thought you edit the config file. Try:
> - adding lxc.network.veth.pair=test-e0

No change.

> - run "ip link show" on the host, and for every interface that starts
> with a "veth", do "ip link del" on that interface. That is, assuming
> you don't currently have any containers or anything that uses veth
> running.

There are no veth's:

$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP qlen 1000
    link/ether 00:25:22:2f:b9:40 brd ff:ff:ff:ff:ff:ff
4: lxcbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
state DOWN
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
52: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
state DOWN
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff

> - try the second link, which create the rootfs and config file by hand
> instead of using a script.

Well, foo.  I need to end up with a debugged script, and I was hoping
that since lxc-create -t centos had worked for someone recently
(and lxc-create -t fedora works for me now), success wouldn't be far off.

I guess I'll try to understand lxc-start more.
- Dan

On Sat, Aug 25, 2012 at 9:22 AM, Dan Kegel <dank@...> wrote:
> On Fri, Aug 24, 2012 at 7:00 PM, Fajar A. Nugraha <list@...> wrote:
>>> Aren't the default templates supposed to work with the default config file?
>>
>> It should. Which is why I thought you edit the config file. Try:
>> - adding lxc.network.veth.pair=test-e0
>
> No change.

I think I figure out what's wrong. Short version is upstream template
(even the fedora one) needs modification. At least it does on Ubuntu
:)


>> - try the second link, which create the rootfs and config file by hand
>> instead of using a script.
>
> Well, foo.  I need to end up with a debugged script,

Had you tried creating it manually, you'd have a working centos
container AND understand more deeply how it works.
In case you haven't figure it out already, I wrote that wiki page :)

> and I was hoping
> that since lxc-create -t centos had worked for someone recently
> (and lxc-create -t fedora works for me now), success wouldn't be far off.

Not sure how Johannes got his template working and what his
environment is, but I've written mine. See
https://github.com/fajarnugraha/lxc/blob/centos-template/templates/lxc-centos.in

Tested on Ubuntu 12.04 host to install centos guest container (centos
6 only for now). Based on lxc-fedora, it mostly automates the manual
process on my wiki page, plus rpmdb fix to allow it to work on
non-centos host.

If you're not familiar with github, "raw" button downloads that file
only, while "history" button shows changes to that file.

-- 
Fajar

On Sat, Aug 25, 2012 at 3:34 AM, Fajar A. Nugraha <list@...> wrote:
> Had you tried creating it manually, you'd have a working centos
> container AND understand more deeply how it works.
> In case you haven't figure it out already, I wrote that wiki page :)

I didn't know that your procedure was complete and debugged.  (After
all, you pointed to it last :-)

> https://github.com/fajarnugraha/lxc/blob/centos-template/templates/lxc-centos.in
> Tested on Ubuntu 12.04 host to install centos guest container (centos
> 6 only for now). Based on lxc-fedora, it mostly automates the manual
> process on my wiki page, plus rpmdb fix to allow it to work on
> non-centos host.

I copied yours to /usr/lib/lxc/templates/lxc-centos (your .in file
didn't seem to have any @KEYWORDS@ that needed expanding by configure),
and verified that it creates a startable centos 6 container.

There's no networking (beyond being able to ping localhost).
Uncommenting the example network section of
/var/lib/lxc/demo.centos6/config
led to the same failure as before,
lxc-start: failed to rename vethbEIwaC->eth0 : File exists

But then, the default templates on this machine don't seem to
have working networks, either.  I guess I have some experimenting
and learning to do.
- Dan

On Sat, Aug 25, 2012 at 11:18 AM, Dan Kegel <dank@...> wrote:
>> https://github.com/fajarnugraha/lxc/blob/centos-template/templates/lxc-centos.in
>
> There's no networking (beyond being able to ping localhost).

https://help.ubuntu.com/12.04/serverguide/lxc.html says
"The simplest container creation command would look like:
sudo lxc-create -t ubuntu -n CN
This tells lxc-create to use the ubuntu template (-t ubuntu) and to
call the container CN (-n CN). Since no configuration file was
specified (which would have been done with `-f file'), it will use the
default configuration file under /etc/lxc/lxc.conf. This gives the
container a single veth network interface attached to the lxcbr0
bridge."

So I tried that on a clean ubuntu 12.04 machine, and
    lxc-create -t ubuntu -n democn
    lxc-start -n democn
did indeed result in a session with a working network.
(Whew.)

Trying the same thing with your centos template:
   cd /usr/lib/lxc/templates/
   wget https://raw.github.com/fajarnugraha/lxc/centos-template/templates/lxc-centos.in
   mv lxc-centos.in lxc-centos
   lxc-create -t centos -n democ6
   lxc-start -n democ6
resulted in a session without working network.
According to ifconfig inside the Centos session, eth0
didn't have an ip address.   I dimly recall that the network
doesn't start by default on Centos desktops, so I did
  /etc/init.d/network start
inside the centos session.  That at least got eth0 an ip address,
but dns still wasn't working.  And it looks like part of the machinery
is missing:
# ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 29 Mar 12 12:45 /etc/resolv.conf ->
../run/resolvconf/resolv.conf
Adding a real file there made dns work, and I was able to do simple networking.

How far do you think your script is from being ready to go into lxc?
Does it need to start the network automatically, and how should
resolv.conf be handled?

Thanks,
Dan

On Sun, Aug 26, 2012 at 3:06 AM, Dan Kegel <dank@...> wrote:
> Trying the same thing with your centos template:
>    cd /usr/lib/lxc/templates/
>    wget https://raw.github.com/fajarnugraha/lxc/centos-template/templates/lxc-centos.in
>    mv lxc-centos.in lxc-centos
>    lxc-create -t centos -n democ6
>    lxc-start -n democ6
> resulted in a session without working network.

What host are you using?

An ubuntu host will automatically add a networking section like this,
which works

lxc.network.type=veth
lxc.network.link=lxcbr0
lxc.network.flags=up

The default networking config section from lxc-fedora (upstream
version), when used in Ubuntu host would add an additional interface
in the container (because it's basically a separate, additional
networking section) and will result in "failed to rename
vethXXXXXX->eth0 : File exists" error, so I commented it out. Which is
why I'm surprised if you say lxc-fedora works for you, because its
default networking section should be the same (i.e. should also cause
"eth0: File exists" error).

> According to ifconfig inside the Centos session, eth0
> didn't have an ip address.   I dimly recall that the network
> doesn't start by default on Centos desktops, so I did
>   /etc/init.d/network start
> inside the centos session.  That at least got eth0 an ip address,

Weird. I tested it on my ubuntu host, and guest container networking
is up automatically.

Try replacing your container config networking section (lxc.network.*)
with the ones from your working ubuntu container, or the one I pasted
earlier (i.e. you should NOT have any line that says "lxc.network.name
= eth0", only those three lines above for lxc.network.*).

> but dns still wasn't working.  And it looks like part of the machinery
> is missing:
> # ls -l /etc/resolv.conf
> lrwxrwxrwx 1 root root 29 Mar 12 12:45 /etc/resolv.conf ->
> ../run/resolvconf/resolv.conf
> Adding a real file there made dns work, and I was able to do simple networking.

This one is partly my fault.

Ones a template is used, the resulting rootfs is cached on
/var/lib/lxc, and in the case of centos/fedora, it will be updated
(yum update) every time you use it to install new containers with that
template. Doing "yum --installroot" for update is bad, since the
host's yum version may do "bad stuff" (e.g. uses incompatible rpmdb
version), so I changed it to "chroot ... yum". However for that to
work, I need a working resolv.conf inside the rootfs, so I simply do a
"cp -a" from the host's resolv.conf, which works for me because I
uninstalled resolvconf. In your case it resulted in a non-working
resolv.conf :)

Since you already have a working /etc/resolv.conf (i.e. NOT a symlink)
inside the container, can you try rebooting the container? If it works
(i.e. got networking on boot), I can push a simple fix (which
basically would just change "cp -a" to just "cp").

-- 
Fajar

On Sat, Aug 25, 2012 at 3:53 PM, Fajar A. Nugraha <list@...> wrote:
> What host are you using?
$ uname -a
Linux i5lap 3.2.0-29-generic-pae #46-Ubuntu SMP Fri Jul 27 17:25:43
UTC 2012 i686 i686 i386 GNU/Linux
$ cat /etc/issue
Ubuntu 12.04.1 LTS \n \l
$ dpkg-query -W lxc
lxc	0.7.5-3ubuntu62

> An ubuntu host will automatically add a networking section like this,
> which works
>
> lxc.network.type=veth
> lxc.network.link=lxcbr0
> lxc.network.flags=up

Right, except it doesn't work with your template somehow.

> The default networking config section from lxc-fedora (upstream
> version), when used in Ubuntu host would add an additional interface
> in the container (because it's basically a separate, additional
> networking section) and will result in "failed to rename
> vethXXXXXX->eth0 : File exists" error, so I commented it out. Which is
> why I'm surprised if you say lxc-fedora works for you, because its
> default networking section should be the same (i.e. should also cause
> "eth0: File exists" error).

Works here.  The resulting concatenated config file for fedora is

lxc.network.type=veth
lxc.network.link=lxcbr0
lxc.network.flags=up
lxc.utsname = democf
lxc.tty = 4
lxc.pts = 1024
lxc.rootfs = /var/lib/lxc/democf/rootfs
lxc.mount  = /var/lib/lxc/democf/fstab
# uncomment the next line to run the container unconfined:
#lxc.aa_profile = unconfined
#cgroups
lxc.cgroup.devices.deny = a
# /dev/null and zero
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
# consoles
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 4:0 rwm
lxc.cgroup.devices.allow = c 4:1 rwm
# /dev/{,u}random
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
# rtc
lxc.cgroup.devices.allow = c 254:0 rwm

> Try replacing your container config networking section (lxc.network.*)
> with the ones from your working ubuntu container, or the one I pasted
> earlier (i.e. you should NOT have any line that says "lxc.network.name
> = eth0", only those three lines above for lxc.network.*).

They're identical, nothing to replace.

> Since you already have a working /etc/resolv.conf (i.e. NOT a symlink)
> inside the container, can you try rebooting the container? If it works
> (i.e. got networking on boot), I can push a simple fix (which
> basically would just change "cp -a" to just "cp").

It does seem to have networking on boot now, so I think you're on to
something there.
- Dan

On Sun, Aug 26, 2012 at 11:09 AM, Dan Kegel <dank@...> wrote:
>> The default networking config section from lxc-fedora (upstream
>> version), when used in Ubuntu host would add an additional interface
>> in the container (because it's basically a separate, additional
>> networking section) and will result in "failed to rename
>> vethXXXXXX->eth0 : File exists" error, so I commented it out. Which is
>> why I'm surprised if you say lxc-fedora works for you, because its
>> default networking section should be the same (i.e. should also cause
>> "eth0: File exists" error).
>
> Works here.  The resulting concatenated config file for fedora is
>
> lxc.network.type=veth
> lxc.network.link=lxcbr0
> lxc.network.flags=up

That's because you use ubuntu's version of lxc-fedora. The upstream
version would've caused an error.

>> Since you already have a working /etc/resolv.conf (i.e. NOT a symlink)
>> inside the container, can you try rebooting the container? If it works
>> (i.e. got networking on boot), I can push a simple fix (which
>> basically would just change "cp -a" to just "cp").
>
> It does seem to have networking on boot now, so I think you're on to
> something there.

Please re-download lxc-centos.in and try again. You need to remove the
cache first (rm -rf /var/cache/lxc/centos/x86_64/6/rootfs) so that it
will be recreated.

... or if you just want new containers to work, just replace
/etc/resolv.conf in /var/cache/lxc/centos/x86_64/6/rootfs with a
working version.

-- 
Fajar

On Sat, Aug 25, 2012 at 10:24 PM, Fajar A. Nugraha <list@...> wrote:
> Please re-download lxc-centos.in and try again. You need to remove the
> cache first (rm -rf /var/cache/lxc/centos/x86_64/6/rootfs) so that it
> will be recreated.

Hmm.   Is UTSNAME supposed to be defined on entry to the template?
It isn't here, leading to /etc/sysconfig/network and
/etc/sysconfig/network-scripts/ifcfg-eth0
containing bad
HOSTNAME=
lines.  (The fedora template has same problem.)
I worked around this with:

--- lxc-centos.in	2012-08-27 10:45:56.301880000 -0700
+++ /usr/lib/lxc/templates/lxc-centos	2012-08-27 12:12:30.271501887 -0700
@@ -52,7 +52,7 @@
 DEVICE=eth0
 BOOTPROTO=dhcp
 ONBOOT=yes
-HOSTNAME=${UTSNAME}
+HOSTNAME=${name}
 NM_CONTROLLED=no
 TYPE=Ethernet
 MTU=${MTU}
@@ -61,7 +61,7 @@
     # set the hostname
     cat <<EOF > ${rootfs_path}/etc/sysconfig/network
 NETWORKING=yes
-HOSTNAME=${UTSNAME}
+HOSTNAME=${name}
 EOF

     # set minimal hosts

The next problem is a bug in lxc-start-ephemeral, which complains
sed: can't read
/var/lib/lxc/demo_centos6-temp-j4G0FcH/rootfs/etc/hostname: No such
file or directory
The error comes from the line
    # Update the ephemeral lxc's configuration to reflect the new
container name.
    sudo sed -i -e "s/$LXC_BASE/$LXC_NAME/" $LXC_DIR/fstab
$LXC_DIR/config $LXC_DIR/rootfs/etc/hostname $LXC_DIR/rootfs/etc/hosts

I worked around this with
--- lxc-start-ephemeral.orig	2012-08-27 12:19:03.935513119 -0700
+++ lxc-start-ephemeral	2012-08-27 12:19:54.471514479 -0700
@@ -154,7 +154,19 @@
     LXC_MOUNTED=1

     # Update the ephemeral lxc's configuration to reflect the new
container name.
-    sudo sed -i -e "s/$LXC_BASE/$LXC_NAME/" $LXC_DIR/fstab
$LXC_DIR/config $LXC_DIR/rootfs/etc/hostname $LXC_DIR/rootfs/etc/hosts
+    # Check all the places known distros keep hostnames.
+    # FIXME: should we sanity check the hostname to make sure it
contains no bad chars?
+    for file in $LXC_DIR/fstab $LXC_DIR/config \
+                $LXC_DIR/rootfs/etc/hostname \
+                $LXC_DIR/rootfs/etc/hosts \
+                $LXC_DIR/rootfs/etc/sysconfig/network \
+                $LXC_DIR/rootfs/etc/sysconfig/network-scripts/ifcfg-eth0
+    do
+        if test -f "$file"
+        then
+            sudo sed -i -e "s/$LXC_BASE/$LXC_NAME/" $file
+        fi
+    done

     # Update the fstab to have all bind mounts be ephemeral.
     sudo cp $LXC_DIR/fstab $LXC_DIR/fstab.old

Thanks,
Dan

On Mon, Aug 27, 2012 at 12:27 PM, Dan Kegel <dank@...> wrote:
> Hmm.   Is UTSNAME supposed to be defined on entry to the template?
> It isn't here, leading to /etc/sysconfig/network and
> /etc/sysconfig/network-scripts/ifcfg-eth0
> containing bad
> HOSTNAME=
> lines.  (The fedora template has same problem.)

I see the ubuntu ppa replaced $UTSNAME with $name in lxc-fedora already,
so my change is probably right.

On Mon, Aug 27, 2012 at 12:27 PM, Dan Kegel <dank@...> wrote:
> The next problem is a bug in lxc-start-ephemeral, which complains
> sed: can't read
> /var/lib/lxc/demo_centos6-temp-j4G0FcH/rootfs/etc/hostname: No such
> file or directory
> The error comes from the line
>     # Update the ephemeral lxc's configuration to reflect the new
> container name.
>     sudo sed -i -e "s/$LXC_BASE/$LXC_NAME/" $LXC_DIR/fstab
> $LXC_DIR/config $LXC_DIR/rootfs/etc/hostname $LXC_DIR/rootfs/etc/hosts

Filed as
https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1042431

Or is there a better way?

I just realized that the --bindhome option is only in
the ubuntu template, so I also filed a bug
https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1043004
about that.   I guess I could try adding it to the centos
template as a first step.

BTW it would also be nice if all the templates agreed on the option
for specifying the version.  Right now I think it's -r for ubuntu and -R for
centos.