GPFS support

  • Hi,

    If at all possible, I'm considering patching rpc.rquotad to support IBM's GPFS filesystem:
    GPFS has its own set of commands such as mmedquota, mmlsquota, and mmrepquota but lacks a port of the rpc.quotad service.

    I believe it shouldn't be too difficult since:
        Implemented as the Virtual File System (VFS),
        GPFS supports the UNIX file interface so that
        applications need not be changed to take advantage
        of the new file system.
    as seen here:

    Any pointers on where I should start from? Should I implement a quotaio_gpfs.c file?

    • Jan Kara
      Jan Kara

      If you implement quotaio_gpfs.c, then quota tools would be able to operate on quota files in gpfs format. To have rpc.rquotad support, you actually need something different. Userspace communicates with kernel about quota using quotactl interface and
      rpc.rquotad uses this interface (namely GETQUOTA and SETQUOTA calls). So you need to implement hooks for this interface for your filesystem... The sources for quotactl interface in kernel are in linux/fs/quota.c.

    • Thank you for your answers.

      In the meantime, I had a deeper look at this issue. Here is my current understanding of the situation. I would be grateful if you could help by answering a few questions, time permitting:

      1) I wrote a proof-of-concept rquotad program for GPFS, based on the BSD source code and directly calling the GPFS API instead of the kernel API. It's available for download from an IBM forum, in this thread:
      This helped me understand how it all works and already provides us with a drop-in replacement for the Linux rpc.rquotad, supporting only GPFS file systems.
      It's enough for my current needs, but of course I'd like to give back to the community something that could be integrated into existing code - and hopefully maintained with that code ;-)

      2) I may be wrong, but I think GPFS does not make quota information available through the kernel API. It's not just a question of format, I think the information is simply not there. This information is made available through the GPFS user-space API:
      What would be the easiest way to check that GPFS file-systems indeed do not expose quota information through the usual kernel API?

      3) If indeed GPFS does not expose quota information through the usual kernel API, I could ask IBM to fix this situation and make it available. Maybe it's not done because of licensing issues (GPFS is not GPL'ed). I really don't know, hopefully I can get some feedback from IBM support on this one... Let's call this solution A.
      Otherwise, I could stick to my current code, with two different code paths, one for GPFS and using the GPFS API, another for all other file-systems and using the kernel API. Let's call this solution B.
      Solution A would be preferable over solution B, wouldn't it?

      4) Concerning solution B: From a technical point-of-view, would you accept patches that implement solution B? Again, that would mean two different code paths, one using the GPFS API, the other one using the kernel API.

      5) Concerning solution B: From a licensing point-of-view, there's an incompatibility. Linux DiskQuota is GPL'ed as far as I can see. The GPFS library containing gpfs_quotactl() is provided under a closed-source commercial license. Either Linux DiskQuota or IBM would have to change their license to be able to integrate the code.
      On the Linux DiskQuota side: Would a GPL exception for GPFS be acceptable?
      On the IBM side: I could ask IBM to release the gpfs_quotactl() part of under GPL. They have already done that for Samba in the past, exposing parts of the GPFS API in the GPL'ed library and its <gpfs_gpl.h> header:
      Would you accept patches for GPFS support under solution B, if IBM re-licensed the relevant parts of their code under GPL? If the answer is yes, I'll start pestering IBM about this...

    • Here is how I test whether quota information is made available through the kernel API:

      $ cat test-quotactl.c
      #include <sys/quota.h>
      #include <stdio.h>
      #include <string.h>
      #include <errno.h>
      int main() {
          const char special[] = "/neurospin/home/papadopoulos";
          int id = 16044;
          struct dqblk addr;
          int cmd = QCMD(Q_GETQUOTA, USRQUOTA);
          if (quotactl(cmd, special, id, (caddr_t)&addr) < 0) {
              fprintf(stderr, "quotactl() failed: %s (%d)\n",
                  strerror(errno), errno);
      $ cc -o test-quotactl test-quotactl.c
      $ ./test-quotactl
      quotactl() failed: Block device required (15)
      $ mmlsquota -u 16044 /neurospin/home/papadopoulos
      mmlsquota: The device name /neurospin/home/papadopoulos starts with a slash, but not /dev/.
                               Block Limits                                    |     File Limits
      Filesystem type             KB      quota      limit   in_doubt    grace |    files   quota    limit in_doubt    grace  Remarks
      home       USR         2023216    5242880    6291456      10192     none |    31813       0        0       51     none

      As you can see, it looks like quotactl() cannot access quota information for this path. The GPFS command can. Is there any other way to retrieve quota information from the kernel?

    • Ooops... In the test code posted above, please change from:
      const char special[] = "/neurospin/home/papadopoulos";
      const char special[] = "/dev/home";

      The resulting error is:
      $ ./test-quotactl
      quotactl() failed: No such device (19)

      The device associated to /neurospin/home is /dev/home:
      $ df -k /neurospin/home/papadopoulos
      Filesystem           1K-blocks      Used Available Use% Mounted on
      /dev/home            2339553280 191806976 2147746304   9% /neurospin/home

    • Jan Kara
      Jan Kara


      First I'd like to thank you for your work and care for the community :). Now to your questions:
      As far as I can say, GPFS does not provide its quota information over the standard quotactl interface. Part of the reason is that they need to pass to userspace two additional values - the uncertainty in block and inode accounting. Definitely the most preferable solution would be to get together all the cluster filesystem people - i.e. at least also GFS2, OCFS2 and ask what quota information would they like to be passed to user space and then extend current kernel interface so that it is able to support it (and we could accomodate XFS at the same time - currently it has special syscall because of similar reasons). Now for designing the interface, one basically doesn't need to know more than the structure they pass which can hardly be a part of some license issues... So this would be a way to go which would be beneficial to all the involved parties.

      Given that GPFS is not in mainline kernel and until they change the licence to GPL it cannot be there (BTW: some people find kernel modules with proprietary license illegal due to GPL), I'm reluctant to do any substantial changes to quota-tools to ease them their life with proprietary kernel module... So this basically rules out your plan B (sorry, I hate to make life harder for users because of proprietary modules but this is the only leverage community has against them - besides suing which I don't think would do any good to anybody).

      The plan A has the disadvantage that it will take some time for this to be done, even more because currently I have more urgent assignments than improving quota kernel interface... I've added this to my todo list I'll see when I can have a look at it :)