#27 aufs-utils fails to build with musl libc due to use of non-standard FTW_ACTIONRETVAL

[Natanael Copa]

It would be nice to build aufs-utils with musl libc since is built statically and statically linked musl is significantly smaller than statically linked gnu libc.

But...

aufs-utils fails to build with musl libc with the following error:

plink.c: In function 'ftw_list':
plink.c:168:10: error: 'FTW_SKIP_SUBTREE' undeclared (first use in this function)
return FTW_SKIP_SUBTREE;
^~~~~~~~~~~~~~~~
plink.c:168:10: note: each undeclared identifier is reported only once for each function it appears in
plink.c:170:10: error: 'FTW_CONTINUE' undeclared (first use in this function)
return FTW_CONTINUE;
^~~~~~~~~~~~
plink.c: In function 'ftw_cpup':
plink.c:184:10: error: 'FTW_SKIP_SUBTREE' undeclared (first use in this function)
return FTW_SKIP_SUBTREE;
^~~~~~~~~~~~~~~~
plink.c:186:10: error: 'FTW_CONTINUE' undeclared (first use in this function)
return FTW_CONTINUE;
^~~~~~~~~~~~

The GNU extension FTW_ACTIONRETVAL is not implemented in musl libc's nftw (neither in the BSD). To fix this we have two options:

• ship an ntfw gnu compatible implementation (eg copy glibc's ntfw.c)
• refactor the code to not use nftw.

[J. R. Okajima]

"Natanael Copa":

The GNU extension FTW_ACTIONRETVAL is not implemented in musl libc's nftw (neither in the BSD). To fix this we have two options:

• ship an ntfw gnu compatible implementation (eg copy glibc's ntfw.c)
• refactor the code to not use nftw.

Hmm, I have to check musl libc again...
I don't think it a good idea to copy glibc's ntfw.c because it will
force us to keep watching glibc's bugfix.

I will try these steps.

• read the source of musl libc and search other useful function.
• ask the developers of musl libc about the file tree walker function.
• if they don't have such plan, I will try implementing the substitution
  in aufs-utile.
  This is my current plan, and it may take long time I am afraid.

How do you think Natanael?
Or do you want to implement by yourself, and contribute to aufs-util? :-)

J. R. Okajima

[Natanael Copa]

I will try these steps.
- read the source of musl libc and search other useful function.
- ask the developers of musl libc about the file tree walker function.
- if they don't have such plan, I will try implementing the substitution
in aufs-utile.

I did the 2 first steps already. I could not find any substitution. (and you can read the POSIX docs instead of musl source). I also talked with musl developers on IRC (#musl on freenode) and the conclusion was similar to your third step.

Or do you want to implement by yourself, and contribute to aufs-util? :-)

I thought of it, but I don't have any experience with aufs in general and I'm not familiar with the implementation and internals. So since I currently don't understand the problem it solves, I'd likely not come up with a good solution.

[J. R. Okajima]

"Natanael Copa":

I did the 2 first steps already. I could not find any substitution. (and you can read the POSIX docs instead of musl source). I also talked with musl developers on IRC (#musl on freenode) and the conclusion was similar to your third step.

Ah, you are going much ahead of me.
Ok, FTW_ACTIONRETVAL is glibc specific. In order to support non-glibc
users, aufs-util has to implement a substitution.

Here is my current solution. Ah, I should say "dirty workaround" instead
of "solution." See the patch below.
Additionally, since pseudo-link (I call it plink") is very special for
aufs, another patch is required to aufs kernel-space.
If you have any opinion, please let me know.

J. R. Okajima

(for aufs-util.git)
diff --git a/Makefile b/Makefile
index 8a44acc..3aaa764 100644
--- a/Makefile
+++ b/Makefile
@@ -58,6 +58,11 @@ LibUtil = libautil.a
LibUtilObj += perror.o proc_mnt.o br.o plink.o mtab.o
LibUtilHdr = au_util.h

+ifeq (${NoLibcFTW},yes)
+override CPPFLAGS += -DNO_LIBC_FTW
+Cmd += auplink_ftw
+endif
+
# suppress 'eval' for ${v}
$(foreach v, CPPFLAGS CFLAGS INSTALL Install ManDir LibUtilHdr, \
$(eval MAKE += ${v}="$${${v}}"))
@@ -108,6 +113,9 @@ c2sh c2tmac ver: CC = ${HOSTCC}
.INTERMEDIATE: c2sh c2tmac ver

install_sbin: File = auibusy aumvdown auplink mount.aufs umount.aufs
+ifeq (${NoLibcFTW},yes)
+install_sbin: File += auplink_ftw
+endif
install_sbin: Tgt = ${DESTDIR}/sbin
install_ubin: File = aubusy auchk aubrsync #auctl
install_ubin: Tgt = ${DESTDIR}/usr/bin
diff --git a/README b/README
index 0e583dd..8f3edf0 100644
--- a/README
+++ b/README
@@ -68,6 +68,10 @@ Makefile in this tree has some customizable make-variables.
The default is BuildFHSM=no.
$ make BuildFHSM=yes

+- NoLibcFTW

• specify "yes" if your libc doesn't support FTW_ACTIONRETVAL which was
• added since glibc-v2.3.3.
  +
  o /sbin/mount.aufs, /sbin/umount.aufs
  Helpers for util-linux-ng package. You should NOT invoke them
  manually. Just install them by "make install".
  diff --git a/plink.c b/plink.c
  index 7f79656..28c3788 100644
  --- a/plink.c
  +++ b/plink.c
  @@ -245,6 +245,66 @@ void au_clean_plink(void)
  #endif
  }

+#ifdef NO_LIBC_FTW
+#define FTW_ACTIONRETVAL 0
+static int au_nftw(const char *dirpath,

• int (fn) (const char fpath, const struct stat *sb,
• int typeflag, struct FTW *ftwbuf),
• int nopenfd, int flags)
  +{
• int err, fd, i;
• mode_t mask;
• FILE *fp;
• ino_t *p;
• char *action, ftw[1024], tmp[] = "/tmp/auplink_ftw.XXXXXX";
  +
• mask = umask(S_IRWXG | S_IRWXO);
• fd = mkstemp(tmp);
• if (fd < 0)
• AuFin("mkstemp");
• umask(mask);
• fp = fdopen(fd, "r+");
• if (!fp)
• AuFin("fdopen");
  +
• ia.p = ia.o;
• p = ia.cur;
• for (i = 0; i < ia.nino; i++) {
• err = fprintf(fp, "%llu\n", (unsigned long long)*p++);
• if (err < 0)
• AuFin("%s", tmp);
• }
• err = fflush(fp) || ferror(fp);
• if (err)
• AuFin("%s", tmp);
• err = fclose(fp);
• if (err)
• AuFin("%s", tmp);
  +
• action = "list";
• if (fn == ftw_cpup)
• action = "cpup";
• else
• fflush(stdout); / inode numbers /
• i = snprintf(ftw, sizeof(ftw), "auplink_ftw %s %s %s",
• tmp, dirpath, action);
• if (i > sizeof(ftw))
• AuFin("snprintf");
• err = system(ftw);
• if (err == -1)
• AuFin("%s", ftw);
• else if (WEXITSTATUS(err))
• AuFin("%s", ftw);
  +
• return err;
  +}
  +#else
  +#define au_nftw nftw
  +#ifndef FTW_ACTIONRETVAL
  +#error FTW_ACTIONRETVAL is not defined on your system
  +#endif
  +#endif
  +
  static int do_plink(char cwd, int cmd, int nbr, union aufs_brinfo brinfo)
  {
  int err, i, l, nopenfd;
  @@ -304,8 +364,8 @@ static int do_plink(char cwd, int cmd, int nbr, union aufs_brinfo brinfo)
  nopenfd = OPEN_LIMIT;
  else if (nopenfd > 20)
  nopenfd -= 10;
• nftw(cwd, func, nopenfd,
• FTW_PHYS | FTW_MOUNT | FTW_ACTIONRETVAL);
• au_nftw(cwd, func, nopenfd,

• FTW_PHYS | FTW_MOUNT | FTW_ACTIONRETVAL);
  / ignore /

  if (cmd == AuPlink_FLUSH) {

(a new file, auplink_ftw, in aufs-util.git)

!/bin/sh

usage: $0 inum_list dir list|cpup

tmp=/tmp/$$
set -eu
rc=${DebugRc:-/etc/default/aufs}
. $rc

inum=$1
dir=$2
action=$3

build the grep pattern

sed -e 's/^/^/' $inum > $tmp.inum

Find()
{
find $dir -name $AUFS_WH_PLINKDIR -prune \
-o -printf "%i %p\n" | #2> /dev/null |
grep -w -f $tmp.inum |
cut -f2- -d' ' ||
:
}

err=0
case $3 in
list)
Find
;;
cpup)
Find |
xargs -r touch -a
;;
*)
echo Usage
err=1
;;
esac

rm -fr $tmp $tmp.*
exit $err

(for aufs3-linux.git)
diff --git a/fs/aufs/plink.c b/fs/aufs/plink.c
index 98252c5..2372c77 100644
--- a/fs/aufs/plink.c
+++ b/fs/aufs/plink.c
@@ -32,6 +32,7 @@ int au_plink_maint(struct super_block *sb, int flags)
{
int err;
pid_t pid, ppid;

• struct task_struct parent, prev;
  struct au_sbinfo *sbi;

  SiMustAnyLock(sb);
  @@ -46,11 +47,22 @@ int au_plink_maint(struct super_block *sb, int flags)
  goto out;

  / todo: it highly depends upon /sbin/mount.aufs /

• prev = NULL;

• parent = current;
• ppid = 0;
  rcu_read_lock();
• ppid = task_pid_vnr(rcu_dereference(current->real_parent));
• while (1) {
• parent = rcu_dereference(parent->real_parent);
• if (parent == prev)
• break;
• ppid = task_pid_vnr(parent);
• if (pid == ppid) {
• rcu_read_unlock();
• goto out;
• }
• prev = parent;
• }
  rcu_read_unlock();
• if (pid == ppid)

• goto out;

  if (au_ftest_lock(flags, NOPLMW)) {
  / if there is no i_mutex lock in VFS, we don't need to wait /

[Natanael Copa]

Thank you for looking into it. Unfortunally it sill dont build due to the following non-standard identifiers:

• __nftw_func_t
• FTW_SKIP_SUBTREE
• FTW_CONTINUE

plink.c: In function 'ftw_list':
plink.c:168:10: error: 'FTW_SKIP_SUBTREE' undeclared (first use in this function)
   return FTW_SKIP_SUBTREE;
          ^~~~~~~~~~~~~~~~
plink.c:168:10: note: each undeclared identifier is reported only once for each function it appears in
plink.c:170:10: error: 'FTW_CONTINUE' undeclared (first use in this function)
   return FTW_CONTINUE;
          ^~~~~~~~~~~~
plink.c: In function 'ftw_cpup':
plink.c:184:10: error: 'FTW_SKIP_SUBTREE' undeclared (first use in this function)
   return FTW_SKIP_SUBTREE;
          ^~~~~~~~~~~~~~~~
plink.c:186:10: error: 'FTW_CONTINUE' undeclared (first use in this function)
   return FTW_CONTINUE;
          ^~~~~~~~~~~~
plink.c: In function 'do_plink':
plink.c:312:2: error: unknown type name '__nftw_func_t'
  __nftw_func_t func;
  ^~~~~~~~~~~~~

I also think the name "NO_LIBC_FTW" is misleading. musl libc does have a POSIX compliant nftw implementation. I would recommend using #ifdef __GLIBC__ to test if you should use GNU extensions and fall back to standard if not.

[J. R. Okajima]

"Natanael Copa":

Thank you for looking into it. Unfortunally it sill dont build due to the following non-standard identifiers:

• __nftw_func_t
• FTW_SKIP_SUBTREE
• FTW_CONTINUE

Ok, now I started the compile-test with musl libc which is debian-ized
v1.1.5. After a few fixes for musl, I got this.

$ make NoLibcFTW=yes BuildFHSM=yes CPPFLAGS=-I$PWD/../aufs4-linux/D/usr/include CC=musl-gcc
:::
make[1]: Entering directory '/x2/aufs/aufs-util/libau'
musl-gcc --shared -Wl,-soname,libau.so.2 -s \
-o libau.so.2.7 libau.o rdu_lib.o rdu.o rdu64.o pathconf.o -ldl -lpthread
rdu64.o: In function readdir': rdu64.c:(.text+0x20f): multiple definition ofreaddir'
rdu.o:rdu.c:(.text+0x20f): first defined here
rdu64.o: In function readdir_r': rdu64.c:(.text+0x279): multiple definition ofreaddir_r'
rdu.o:rdu.c:(.text+0x279): first defined here
collect2: error: ld returned 1 exit status
Makefile:43: recipe for target 'libau.so.2.7' failed
make[1]: *** [libau.so.2.7] Error 1
make[1]: Leaving directory '/x2/aufs/aufs-util/libau'

I don't think it sane. How can users re-define readdir() or other
standard libc functions? Did I miss some command line options which are
specific to musl libc? Is it impossible at all with musl libc?

I also think the name "NO_LIBC_FTW" is misleading. musl libc does have a POSIX compliant nftw implementation. I would recommend using #ifdef __GLIBC__ to test if you should use GNU extensions and fall back to standard if not.

You are right.
I don't remember the details, but maybe I was thinking about my local
test. My environment had glibc only. For such env, judging by GLIBC
or GNU_LIBRARY was not a good idea obviously. Now I've install musl
libc and will make NO_LIBC_FTW obsolete (if everything goes well).

J. R. Okajima

[J. R. Okajima]

make[1]: Entering directory '/x2/aufs/aufs-util/libau'
musl-gcc --shared -Wl,-soname,libau.so.2 -s \
-o libau.so.2.7 libau.o rdu_lib.o rdu.o rdu64.o pathconf.o -ldl -lpthread
rdu64.o: In function readdir': rdu64.c:(.text+0x20f): multiple definition ofreaddir'
rdu.o:rdu.c:(.text+0x20f): first defined here
:::

I don't know what is wrong.
The linker finds the multiple definition of `readdir', but it should not
happen. This symbol is switched by a macro Rdu64 at the
compile-time. Actually GCC recognize it correctly, and we can see
readdir64 is defined instead of readdir.

$ gcc -O -Wall -Werror -fPIC -I.../aufs4-linux/D/usr/include -D_GNU_SOURCE -I./libau -DAUFHSM_CMD="/usr/bin/aufhsm" -DMOUNT_CMD="/bin/mount" -DUMOUNT_CMD="/bin/umount" -DAUFHSM -DNO_LIBC_FTW -DAUPLINK_FTW_CMD="/sbin/auplink_ftw" -DRdu64 -DNDEBUG -D_REENTRANT -I. -c -o rdu64.o rdu64.c
$ nm rdu64.o | grep readdir
0000000000000000 t rdu_readdir
000000000000020f T readdir64
0000000000000279 T readdir64_r
0000000000000008 C real_readdir64
0000000000000008 C real_readdir64_r

With the same source file and the same options at all, musl-gcc doesn't
recognize readdir64. Instead it generate as readdir.

$ musl-gcc -O -Wall -Werror -fPIC -I.../aufs4-linux/D/usr/include -D_GNU_SOURCE -I./libau -DAUFHSM_CMD="/usr/bin/aufhsm" -DMOUNT_CMD="/bin/mount" -DUMOUNT_CMD="/bin/umount" -DAUFHSM -DNO_LIBC_FTW -DAUPLINK_FTW_CMD="/sbin/auplink_ftw" -DRdu64 -DNDEBUG -D_REENTRANT -I. -c -o rdu64.o rdu64.c
$ nm rdu64.o | grep readdir
0000000000000000 t rdu_readdir
000000000000020f T readdir
0000000000000279 T readdir_r
0000000000000008 C real_readdir64
0000000000000008 C real_readdir64_r

This is the cause of multiple definition of `readdir' error. But I don't
know why. If you can, would you try applying this patch and compiling
with musl libc?

J. R. Okajima

[J. R. Okajima]

I don't know what is wrong.
The linker finds the multiple definition of `readdir', but it should not
happen. This symbol is switched by a macro Rdu64 at the
compile-time. Actually GCC recognize it correctly, and we can see
readdir64 is defined instead of readdir.

I've found that musl libc defines
#define readdir64 readdir
when _GNU_SOURCE is defined. That is the reason of my link error.
Hmm, what should we do, define or undefine _GNU_SOURCE for musl libc?
That is the question.

J. R. Okajima

[J. R. Okajima]

I've found that musl libc defines
#define readdir64 readdir
when _GNU_SOURCE is defined. That is the reason of my link error.
Hmm, what should we do, define or undefine _GNU_SOURCE for musl libc?
That is the question.

Because of the global variable 'environment', aufs-util wants to keep
_GNU_SOURCE. Here is my current solution for musl libc.
If you can, please try it. I am busy now and cannot test it. It will be
tomorrow maybe.

If the attached patch is dropped from the delivered mail to you, then SF
mail server or bug tracking system must have a criminal feature. Please
refer to https://sourceforge.net/p/aufs/bugs/27/.

J. R. Okajima

[J. R. Okajima]

Because of the global variable 'environment', aufs-util wants to keep
_GNU_SOURCE. Here is my current solution for musl libc.
If you can, please try it. I am busy now and cannot test it. It will be
tomorrow maybe.

I have finished my tests for musl (and glibc). I will release these
commits on next Monday. If you want to try earlier, I can send
personally.

aufs4.1

+f606f9b 2016-09-15 musl-libc 4/4, minor, just to be compilable
+34a40ba 2016-09-15 musl-libc 3/4, separate libau.so individually
+22f8a7b 2016-09-15 musl-libc 2/4, au_decode_mntpnt()
+fe9c398 2016-09-15 musl-libc 1/4, extract error_at_line() for two versions
+b4b0b39 2016-09-15 musl-libc 0/4, extract au_nftw() fot two versions
+296fead 2016-09-15 tiny, spacing after the period
+e5c6abd 2016-09-15 bugfix, respect -v option even mtab is procfs
+09293f5 2016-09-15 return at once when EPERM instead of loop
+24e9ed6 2016-09-15 refine the error msg from regcmp()
+63aed74 2016-09-15 update the forgotten copyright and libau minor version
+93b34d6 2016-09-14 bugfix, make a default errno for error_at_ilne(3)
+2759929 2016-09-13 bugfix, auplink, handling the non-aufs mountpoint

J. R. Okajima