aufs kernel oops (ubuntu 18.04 bundled aufs)
Status: Beta
Brought to you by:
sfjro
Menu
▾
▴
aufs kernel oops (ubuntu 18.04 bundled aufs)
|
From: Kirill K. <kol...@gm...> - 2019-05-29 18:00:46
|
Hi, I was testing my aufs-related fixes to dockerd, and came upon this kernel oops: # dmesg .... [35180.792125] ------------[ cut here ]------------ [35180.792127] kernel BUG at /build/linux-3btXxq/linux-4.15.0/fs/aufs/dynop.c:207! [35180.795621] invalid opcode: 0000 [#1] SMP PTI [35180.797690] Modules linked in: veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc aufs overlay nls_iso8859_1 kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel input_leds joydev serio_raw sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear aesni_intel aes_x86_64 virtio_blk crypto_simd virtio_net cryptd glue_helper psmouse virtio_scsi floppy [35180.832904] CPU: 2 PID: 18881 Comm: dockerd Not tainted 4.15.0-50-generic #54-Ubuntu [35180.836106] Hardware name: DigitalOcean Droplet, BIOS 20171212 12/12/2017 [35180.838893] RIP: 0010:dy_bug+0x9/0x10 [aufs] [35180.840793] RSP: 0018:ffffb1a984017a20 EFLAGS: 00010246 [35180.842931] RAX: ffffffff9dc4f060 RBX: ffff9b5d8f890700 RCX: 0000000000000000 [35180.845755] RDX: 0000000000000000 RSI: ffffffff9dc4f060 RDI: ffff9b5d8f890720 [35180.849012] RBP: ffffb1a984017a20 R08: ffff9b5e9fd27120 R09: ffff9b5d9b814060 [35180.853563] R10: ffffffff9cff2100 R11: 0000000000000c40 R12: ffff9b5d9b814060 [35180.858243] R13: ffff9b5d9b814000 R14: ffff9b5cb40d7c58 R15: ffff9b5d9c295e00 [35180.865344] FS: 00007f8e7bff7700(0000) GS:ffff9b5e9fd00000(0000) knlGS:0000000000000000 [35180.869508] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35180.872336] CR2: 000000c4200f8160 CR3: 0000000183c94006 CR4: 00000000007606e0 [35180.875245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35180.878249] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35180.881038] PKRU: 55555554 [35180.882316] Call Trace: [35180.883546] au_dy_iaop+0x1c3/0x320 [aufs] [35180.885077] ? au_iinfo_init+0x6e/0xd0 [aufs] [35180.886772] au_new_inode+0x4bb/0x730 [aufs] [35180.888616] epilog+0xfd/0x190 [aufs] [35180.890320] add_simple+0x2b1/0x420 [aufs] [35180.893184] au_aopen_or_create+0x46/0x60 [aufs] [35180.895149] aufs_atomic_open+0x1c8/0x3b0 [aufs] [35180.896990] ? aufs_permission+0x190/0x2d0 [aufs] [35180.899162] ? __inode_permission+0x5b/0x160 [35180.900976] path_openat+0xaff/0x1770 [35180.902637] ? path_openat+0xaff/0x1770 [35180.904326] ? au_h_iptr+0x16/0x40 [aufs] [35180.906018] do_filp_open+0x9b/0x110 [35180.907605] ? __check_object_size+0xaf/0x1b0 [35180.909913] ? __alloc_fd+0xb2/0x170 [35180.912472] do_sys_open+0x1bb/0x2c0 [35180.915163] ? do_sys_open+0x1bb/0x2c0 [35180.917923] SyS_openat+0x14/0x20 [35180.920395] do_syscall_64+0x73/0x130 [35180.925637] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [35180.928896] RIP: 0033:0x4b98ca [35180.930976] RSP: 002b:000000c422c76d10 EFLAGS: 00000202 ORIG_RAX: 0000000000000101 [35180.935378] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00000000004b98ca [35180.939158] RDX: 0000000000080040 RSI: 000000c4252efa40 RDI: ffffffffffffff9c [35180.942999] RBP: 000000c422c76d90 R08: 0000000000000000 R09: 0000000000000000 [35180.945841] R10: 00000000000001ed R11: 0000000000000202 R12: ffffffffffffffff [35180.948867] R13: 000000000000003d R14: 000000000000003c R15: 0000000000000049 [35180.951815] Code: 01 00 48 85 c0 74 07 48 89 87 c8 00 00 00 5d c3 66 90 0f 1f 44 00 00 55 48 89 e5 e8 a2 c5 93 dc 5d c3 0f 1f 44 00 00 55 48 89 e5 <0f> 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 fa 48 63 47 f0 48 [35180.960552] RIP: dy_bug+0x9/0x10 [aufs] RSP: ffffb1a984017a20 [35180.962961] ---[ end trace 9c8a22b601f49ad0 ]--- This is on Ubuntu 18.04 with their latest stable kernel (based on 4.15), and here's the version of aufs kmod: # modinfo aufs filename: /lib/modules/4.15.0-50-generic/kernel/fs/aufs/aufs.ko alias: fs-aufs version: 4.15-20180219 description: aufs -- Advanced multi layered unification filesystem author: Junjiro R. Okajima <auf...@li...> license: GPL srcversion: 9C7732CB810A119C03C59CE depends: retpoline: Y intree: Y name: aufs vermagic: 4.15.0-50-generic SMP mod_unload signat: PKCS#7 signer: sig_key: sig_hashalgo: md4 parm: brs:use <sysfs>/fs/aufs/si_*/brN (int) parm: allow_userns:allow unprivileged to mount under userns (bool) Kernel sources for this can be found by doing git clone git:// git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/bionic git reset --hard Ubuntu-4.15.0-50.54 I'm not sure I can recreate this without docker, but what the test does is pretty simple: it prepares and mounts an aufs (with a parent) when creates a few directories and empty files in it, then unmounts it and removes everything. All this is massively parallel, but on different aufs mounts. Note that I was running this test for days, and only got this oops once, so I'm pretty sure it is very hard to reproduce. Unfortunately I had to reboot the node since this happened. If I will reproduce it again, please let me know what other information might be helpful. Docker is written in Go, and from its stack dump I see a goroutine (thread) got stuck in open() syscall while trying to create a file: goroutine 363684 [syscall, 70 minutes]: syscall.Syscall6(0x101, 0xffffffffffffff9c, 0xc4252efa40, 0x80040, 0x1ed, 0x0, 0x0, 0x6e, 0xc4252ef9d0, 0x6e) /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5 syscall.openat(0xffffffffffffff9c, 0xc4252ef9d0, 0x6e, 0x80040, 0x1ed, 0xc4252ef960, 0x6f, 0x6f) /usr/local/go/src/syscall/zsyscall_linux_amd64.go:38 +0xae syscall.Open(0xc4252ef9d0, 0x6e, 0x80040, 0x1ed, 0xc4252ef8f0, 0x6f, 0xc4252ef9d0) /usr/local/go/src/syscall/syscall_linux.go:51 +0x50 os.openFileNolog(0xc4252ef9d0, 0x6e, 0x40, 0x1ed, 0x2, 0xc4252ef9d0, 0x6e) /usr/local/go/src/os/file_unix.go:168 +0x7a os.OpenFile(0xc4252ef9d0, 0x6e, 0x40, 0xc4000001ed, 0x6e, 0x0, 0x0) /usr/local/go/src/os/file.go:269 +0x5f github.com/docker/docker/daemon/initlayer.Setup(0xc4230d3ce0, 0x5e, 0x0, 0x0, 0x12491ed, 0x1) /go/src/github.com/docker/docker/daemon/initlayer/setup_unix.go:50 +0x94e github.com/docker/docker/daemon.(*Daemon).setupInitLayer(0xc4203d6000, 0xc4230d3ce0, 0x5e, 0x0, 0xc4200beb48) /go/src/github.com/docker/docker/daemon/daemon.go:1033 +0x57 github.com/docker/docker/daemon.(*Daemon).(github.com/docker/docker/daemon.setupInitLayer)-fm(0xc4230d3ce0, 0x5e, 0x45, 0x0) /go/src/github.com/docker/docker/daemon/daemon_unix.go:934 +0x3e github.com/docker/docker/layer.(*layerStore).initMount(0xc4203b5310, 0xc421892980, 0x40, 0xc420582600, 0x40, 0x0, 0x0, 0xc4226b3bb0, 0x0, 0x0, ...) /go/src/github.com/docker/docker/layer/layer_store.go:662 +0x1de github.com/docker/docker/layer.(*layerStore).CreateRWLayer(0xc4203b5310, 0xc4218928c0, 0x40, 0xc420040a00, 0x47, 0xc4233f2ec0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/docker/docker/layer/layer_store.go:516 +0x537 github.com/docker/docker/daemon.(*Daemon).setRWLayer(0xc4203d6000, 0xc423c50fc0, 0xc422cbfc00, 0x0) /go/src/github.com/docker/docker/daemon/create.go:251 +0x11f github.com/docker/docker/daemon.(*Daemon).create(0xc4203d6000, 0x0, 0x0, 0xc422bc3e00, 0xc422cbfc00, 0xc4258e3420, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/docker/docker/daemon/create.go:133 +0x236 github.com/docker/docker/daemon.(*Daemon).containerCreate(0xc4203d6000, 0x0, 0x0, 0xc422bc3e00, 0xc422cbfc00, 0xc4258e3420, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/docker/docker/daemon/create.go:60 +0x1a4 github.com/docker/docker/daemon.(*Daemon).ContainerCreate(0xc4203d6000, 0x0, 0x0, 0xc422bc3e00, 0xc422cbfc00, 0xc4258e3420, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/docker/docker/daemon/create.go:33 +0x9d Regards, Kir. |
