Issue with aufs, boot2docker, Percona and O_DIRECT

[Christie W. <chr...@de...>]

Hello,

We have run into an issue running Percona with O_DIRECT when using
boot2docker (which uses aufs). We have been able to work around this
problem by forcing docker not to use aufs in this case, but thought you
might be interested in case there is an underlying problem that needs to be
resolved.

* Observed behaviour:
- The mysql process becoming unresponsive and all attempts to access (e.g.
ls) /var/lib/mysql (the mysql data directory) hang indefinitely.
- Attempts to access /var/lib/mysql inside the container, and also outside
the container (navigating to the associated directory in /var/lib/docker)
both hang.

* Important info:
- This percona option must be set for the issue to occur:
innodb_flush_method = O_DIRECT
- This occurs when running Percona mysql in a docker container with using
versions of boot2docker > 1.8.2, but versions <= 1.8.2 do not have the
issue.
- The boot2docker commit that introduced this issue is this commit where
the kernel and aufs versions are bumped:
https://github.com/boot2docker/boot2docker/commit/7fb8d2d010e2682512b82c7cebd0f743d61b44aa
- The problem does not occur if /var/lib/mysql is declared as a VOLUME when
building the docker image (docker will create a volume on the host, not
using aufs)
- The problem does not occur if a directory on the host is mounted into
/var/lib/mysql

It looks like docker has explicitly added support for direct IO with aufs
by default (https://github.com/docker/docker/pull/10534) so it seems to me
like the assumption is that this use case should work (even if the best
practice is to declare /var/lib/mysql as a VOLUME).

* Steps to reproduce:

# Using latest boot2docker (
https://github.com/boot2docker/boot2docker/releases/tag/v1.10.2), or any
version > 1.8.2, create a VM:

docker-machine create --driver virtualbox --virtualbox-cpu-count "1"
--virtualbox-memory "4096" repro

# Build a percona image - with the O_DIRECT option set - using this VM:

cat > ./my.cnf <<- EOM
[client]
socket = /var/lib/mysql/mysql.sock

[mysqld_safe]
socket = /var/lib/mysql/mysql.sock

[mysqld]
user = mysql
pid-file = /var/lib/mysql/mysql.pid
socket = /var/lib/mysql/mysql.sock
datadir = /var/lib/mysql
bind-address = 0.0.0.0
innodb_flush_method = O_DIRECT
EOM

cat > Dockerfile <<- EOM
FROM centos:6.6

RUN yum install -y
http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm
RUN yum install -y Percona-Server-server-55

ADD my.cnf /etc/my.cnf
#VOLUME ["/var/lib/mysql"]

ENTRYPOINT ["mysqld"]
EXPOSE 3306
EOM

eval "$(docker-machine env repro)"
docker build -t percona -f Dockerfile .

# Run a container using this image
docker run -d --name test_db percona

# In one terminal window, exec into the container and watch /var/lib/mysql
eval "$(docker-machine env repro)"
docker exec -it test_db bash -c "TERM='xterm' watch ls -la /var/lib/mysql"

# In another terminal window, run this command to attempt to create 20k
tables in this container
eval "$(docker-machine env repro)"
docker exec -it test_db bash -c 'mysql -u root -e "create database testing"
&& date && for i in {1..20000}; do echo "create table table_$i (ID
bigint(20) unsigned NOT NULL AUTO_INCREMENT, time timestamp NOT NULL
DEFAULT CURRENT_TIMESTAMP, f1 varchar(16) COLLATE utf8_bin NOT NULL, f2
char(16) COLLATE utf8_bin NOT NULL,  f11 varchar(16) COLLATE utf8_bin NOT
NULL,  f111 varchar(16) COLLATE utf8_bin NOT NULL,   f22 varchar(16)
COLLATE utf8_bin NOT NULL,  f222 varchar(16) COLLATE utf8_bin NOT NULL,
PRIMARY KEY (ID));" | mysql -u root testing; echo -n "table_$i "; done'

# At some point, usually less than a minute later, the watch command will
hang, and any attempts to access /var/lib/mysql in the container or on the
host will hang

* Other info requested

This is the info the aufs github README requests be included in this email:

- /proc/mounts (instead of the output of mount(8))
docker@repro:~$ cat /proc/mounts
tmpfs / tmpfs rw,relatime,size=3641328k 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
devpts /dev/pts devpts rw,relatime,mode=600,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
/dev/sda1 /mnt/sda1 ext4 rw,relatime,data=ordered 0 0
cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls cgroup rw,relatime,net_cls 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/net_prio cgroup rw,relatime,net_prio 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,relatime,hugetlb 0 0
Users /Users vboxsf rw,nodev,relatime 0 0
/dev/sda1 /mnt/sda1/var/lib/docker/aufs ext4 rw,relatime,data=ordered 0 0
none
/mnt/sda1/var/lib/docker/aufs/mnt/61519d5d301361fe9094a732a8e79168930311bbf78298deda27f94665547a9d
aufs rw,relatime,si=82437bd0fc319d67,dio,dirperm1 0 0
shm
/mnt/sda1/var/lib/docker/containers/065adec0e8ca3533546118cc0930bd28d0aaa21312f8442d2d4e5c6c4bc2dc56/shm
tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
nsfs /var/run/docker/netns/8fed2b224c57 nsfs rw 0 0

- /sys/module/aufs/*
root@repro:/sys/module/aufs# for f in `find . -type f`; do echo "$f:"; cat
"$f"; done
./parameters/brs:
1
./parameters/allow_userns:
N
./version:
4.1.13+-20160215
./uevent:
cat: can't open './uevent': Permission denied

- /sys/fs/aufs/* (if you have them)
root@repro:/sys/fs/aufs# for f in `find . -type f`; do echo "$f:"; cat
"$f"; done
./si_82437bd0fc319d67/br0:
/mnt/sda1/var/lib/docker/aufs/diff/61519d5d301361fe9094a732a8e79168930311bbf78298deda27f94665547a9d=rw
./si_82437bd0fc319d67/br1:
/mnt/sda1/var/lib/docker/aufs/diff/61519d5d301361fe9094a732a8e79168930311bbf78298deda27f94665547a9d-init=ro+wh
./si_82437bd0fc319d67/br2:
/mnt/sda1/var/lib/docker/aufs/diff/5e08725fc1388127a22c01345ba5d84dbb11511cb75d616b49d2ec848fb79792=ro+wh
./si_82437bd0fc319d67/br3:
/mnt/sda1/var/lib/docker/aufs/diff/86e34d747e3ae936effd10b0059b178d56d33ad7ff4fb70a32d550e7b83f66a6=ro+wh
./si_82437bd0fc319d67/br4:
/mnt/sda1/var/lib/docker/aufs/diff/5cc807e6634443dec30c3c22ae5b04e97e3d131709df81d8cae67b86e4bb9bf1=ro+wh
./si_82437bd0fc319d67/br5:
/mnt/sda1/var/lib/docker/aufs/diff/0d8442795212973eaae827f91c0c2547b0dc27a641ddc7da30435fe6539a8b0d=ro+wh
./si_82437bd0fc319d67/br6:
/mnt/sda1/var/lib/docker/aufs/diff/59f9ed479d38bfada4bb7af9c64874df256370870071a8c83098c9d7f4f4c2e5=ro+wh
./si_82437bd0fc319d67/xi_path:
/dev/shm/aufs.xino
./si_82437bd0fc319d67/brid0:
16384
./si_82437bd0fc319d67/brid1:
16385
./si_82437bd0fc319d67/brid2:
16386
./si_82437bd0fc319d67/brid3:
16387
./si_82437bd0fc319d67/brid4:
16388
./si_82437bd0fc319d67/brid5:
16389
./si_82437bd0fc319d67/brid6:
16390

- /debug/aufs/* (if you have them)
Does not exist

- linux kernel version
  if your kernel is not plain, for example modified by distributor,
  the url where i can download its source is necessary too.
root@repro:/sys/module/aufs# uname -a
Linux repro 4.1.18-boot2docker #1 SMP Sat Feb 20 08:24:27 UTC 2016 x86_64
GNU/Linux

https://github.com/boot2docker/boot2docker/releases/tag/v1.10.2

- aufs version which was printed at loading the module or booting the
  system, instead of the date you downloaded.

This is the output of docker info:

root@repro:/sys/module/aufs# docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 8
Server Version: 1.10.2
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 7
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 4.1.18-boot2docker
Operating System: Boot2Docker 1.10.2 (TCL 6.4.1); master : 611be10 - Mon
Feb 22 22:47:06 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.858 GiB
Name: repro
ID: G2RW:EP4U:STXW:MP7M:255S:GUY4:YOXH:ZOPR:5UPB:KN3V:TT7D:DWQ2
Debug mode (server): true
 File Descriptors: 23
 Goroutines: 48
 System Time: 2016-03-07T23:22:46.938269001Z
 EventsListeners: 0
 Init SHA1:
 Init Path: /usr/local/bin/docker
 Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox

This version of boot2docker is built with aufs 4.1.13+ (commit
1aa8a143457147473edc97035a7f4b89f9d234ae).
https://github.com/boot2docker/boot2docker/blob/611be104ad81065b8d1cf71b73346b59e7b2d7ed/Dockerfile#L29

- configuration (define/undefine CONFIG_AUFS_xxx)
https://github.com/boot2docker/boot2docker/blob/611be104ad81065b8d1cf71b73346b59e7b2d7ed/kernel_config#L4017

CONFIG_AUFS_FS=y
# CONFIG_AUFS_BRANCH_MAX_127 is not set
# CONFIG_AUFS_BRANCH_MAX_511 is not set
# CONFIG_AUFS_BRANCH_MAX_1023 is not set
CONFIG_AUFS_BRANCH_MAX_32767=y
CONFIG_AUFS_SBILIST=y
# CONFIG_AUFS_HNOTIFY is not set
# CONFIG_AUFS_EXPORT is not set
CONFIG_AUFS_XATTR=y
# CONFIG_AUFS_FHSM is not set
# CONFIG_AUFS_RDU is not set
# CONFIG_AUFS_SHWH is not set
# CONFIG_AUFS_BR_RAMFS is not set
# CONFIG_AUFS_BR_FUSE is not set
CONFIG_AUFS_BR_HFSPLUS=y
CONFIG_AUFS_BDEV_LOOP=y
# CONFIG_AUFS_DEBUG is not set

- kernel configuration or /proc/config.gz (if you have it)
https://github.com/boot2docker/boot2docker/blob/611be104ad81065b8d1cf71b73346b59e7b2d7ed/kernel_config

Thanks very much!

-- 

*Christie Wilson*
Software Engineer
*Demonware*