zjails Wiki

Managemet tool for ZFS Jails on FreeBSD with VNET support

Status: Beta

Brought to you by: petertoth

Quick howto

NOTE: before first use please edit /usr/local/sbin/zjail and modify the "my $pool" variable (you need your ZFS pool name here) also set your "my interface" variable - this is used for non VNET jails only and selects which interface to use by default (can be overridden later on per jail basis)

Run "zjail" utility as bellow.

   zjail list [ALL templates replicated remote]
   zjail show jailname [caps]
   zjail create jailname ip= hostname= [property=value]
   zjail start jailname
   zjail stop [-c] jailname
   zjail restart
   zjail rename jailname newname
   zjail move jailname container=[local replicated]
   zjail clone jailname clonename ip= hostname= [property=value]
   zjail startall
   zjail stopall [-c]
   zjail set [property=value] jailname
   zjail install
   zjail destroy jailname
   zjail backup jailname
   zjail restore jailname
   zjail console jailname
   zjail take_template jailname templatename
   zjail update [-b -s] jailname
   zjail set rlimit=on jail01
   zjail cap jail01
   zjail used jail01
   zjail uncap jail01
   zjail show jail01 caps

NOTE: There are some undocumented options and functions which will be cleaned up and recorded at a later time.

   - set up zjail environment from scratch
   # zjail install

   - create a jail jail01
   # zjail create ip=192.168.1.1 hostname=jail01.local \
         defaultroute=192.168.1.254 netmask=24 jail01

   - clone jail01 to jail02
   # zjail clone jail01 ip=192.168.1.2 hostname=jail02.local \
         defaultroute=192.168.1.254 netmask=24 jail02

   - list all jails including base installation
   # zjail list ALL

   - list only local and replicated jails
   # zjail ls or zjail list

   - drop into jail01
   # zjail co jail01 or zjail console jail01

   - set hostname of jail02
   # zjail set hostname=jail03.local jail02

   - set boot priority to 20 (default is 99, number < 99 means higher
     start, stop and boot priority)
   # zjail set priority=20 jail02

   - turn off boot time start
   # zjail set boot=off jail02

   - turn deduplication on for jail02
   # zjail set dedup=on jail02

   - set notes for jail02
   # zjail set notes="This jail is for temporary use only" jail02

   - show everything about jail02
   # zjail show jail02

   - assign two interfaces - vnet0 vnet1 - to jail02
   # zjail set vnet_if="vnet0 vnet1" jail02

   - connect vnet0 to vswitch1 and vnet1 to vswitch2 on jail02
   # zjail set vnet0=vswitch0 vnet1=vswitch1 jail02

   - destroy jail02
   # zjail destroy jail02

   - soft restart jail01
   # zjail restart jail01

   - cold restart jail01
   # zjail restart -c jail01

   - thin create jail03 from default base template
   # zjail clone base ip=192.168.1.105 hostname=jail03.local jail03

   - take template jail01-tmp from jail01
   # zjail take_template jail01 jail01-tmp

   - list templates only
   # zjail list templates

   - turn VNET on for jail01
   # zjail set vnet=on jail01

   - binary update jail01
   # zjail update -b jail01

   - source update jail01
   # zjail update -s jail01

   - binary update all jails
   # zjail update -b ALL

   - turn on resource caps
   # zjail set rlimit=on jail01

   - limit memory to 512M but only warn with the log action
   # zjail set memory=512M jail01
   # zjail set memory_act=log jail01
   # zjail cap jail01

   - limit memory to 512M but this time deny process with deny action
   # zjail set memory=512M jail01
   # zjail set memory_act=deny jail01
   # zjail cap jail01

   - resource limit jail
   # zjail cap jail01

   - display used resourced
   # zjail used jail01

   - remove resource limits
   # zjail uncap jail01

   - display configured resource limits
   # zjail show jail01 caps

Before setting any resource limits please make sure you understand man rctl(8)!

SETUP

Pre-installation requirements

Minimum OS requirements : ZFSv28, FreeBSD 9.2 (amd64),
Kernel with VNET support (if VNET is required)
Required Perl modules: POSIX, Getopt::Lucid, Net::FTP, Perl6::Form

KERNEL
Recompile your KERNEL with VNET and if needed with resource containers
support:

add the following lines to your KERNEL config:

options VIMAGE
options RACCT # Resource accounting
options RCTL # Controls resource limits

WARNING: - exclude ROUTETABLES options (fibs) from the kernel as this
will make the system extremely unstable with VNET!
- exclude ALTQ completely from a VMAGE enabled kernel as it will crash your box
sooner or later!
- make sure uether kernel module is not loaded with a VIMAGE kernel, it will
result in a boot time PANIC!

PACKAGES
This is a full pkg list generated from the test machine
(you don't need to install all, just install the ones stated above "Required Perl modules")

p5-Authen-SASL
p5-Class-Data-Inheritable
p5-Devel-StackTrace
p5-Digest-HMAC
p5-Exception-Class
p5-Exception-Class
p5-GSSAPI
p5-Getopt-Lucid
p5-Net
p5-Net-FTP
p5-Perl6-Export
p5-Perl6-Form
p5-Scalar-List-Utils
p5-Test-Simple

DEVFS
No changes required since 9.1 RELEASE - ZFS is included in the default jail ruleset number 4.

RC.CONF
Add these lines to /etc/rc.conf (customize it to your needs)
---------------------- /etc/rc.conf -----------------------------
cloned_interfaces="bridge0 bridge1 bridge2"
ifconfig_bridge0_name="vswitch0"
ifconfig_bridge1_name="vswitch1"
ifconfig_bridge2_name="vswitch2"
ifconfig_vswitch0="inet 10.5.100.254 netmask 255.255.255.0 up"
ifconfig_vswitch1="inet 10.6.100.254 netmask 255.255.255.0 up"
ifconfig_vswitch2="inet 10.7.100.254 netmask 255.255.255.0 up"

devd_enable="YES"
devfs_load_rulesets="YES"

gateway_enable="YES"

JAIL REPLICATION

For the replication part to work, please install sysutils/zfsnap and
configure periodic snapshotting for zfspool/zjails/jds/replicated.

After zfsnap configuration create a cron entry for replicate.sh.

Simple replication script:
replicate.sh
#!/bin/sh

NODE_A=`/sbin/zfs list -Ho name -d1 -t snapshot tank/zjails/jds/replicated | tail -1`
NODE_B=`ssh root@jailhost02 "/sbin/zfs list -Ho name -d1 -t snapshot tank/zjails/jds/remote | tail -1 | sed s/remote/replicated/g"`
LOCK="/var/run/zfs-replica.lock"

if [ -e $LOCK ] ; then
    echo "WARNING: ZFS sync already in progress ...exiting!"
    exit 1
fi

if [ ! -e $LOCK -a "$NODE_A" != "$NODE_B" ] ; then
    touch $LOCK
    /sbin/zfs send -R -I $NODE_B $NODE_A | ssh -C jailhost02 "zfs recv -F tank/zjails/jds/remote"
    rm $LOCK
fi

exit 0

This will replicate all your jails in the replicated zfs filesystem
from node A to node B. Please note, this is not a real time replication
solution, but rather a periodic active->passive type. One could use
HAST to do block level real-time replication.

OTHER CONSIDERATIONS

Jails rc.conf file (taken from ezjail's recommendations)
# Disable rpc
rpcbind_enable="NO"

#Prevent jails executing cron jobs at the same time
cron_flags="$cron_flags -J 15"

# Disable network sockets for syslog
syslogd_flags="-ss"

# Disable sendmail
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

Disable adjkerntz in jail's crontab