Davfs cache behavior

monstruooo
2009-03-13
2013-04-16
1 2 3 > >> (Page 1 of 3)
  • monstruooo
    monstruooo
    2009-03-13

    Hello

    We are using fuse-davfs to mount WebDAV server as a local folder for an application that performs a considerable amount of reads and writes. It appears that we have a certain problem because davfs is single threaded. We see davfs regularly going into 100% CPU while stopping serving requests and it seems that during this time davfs is clearing the cache. Is it possible to disable the cache completely or something?

    Thanks

     
    • Werner Baumann
      Werner Baumann
      2009-03-14

      davfs2 v0 does no caching.
      But if your CPU load is 100% there is something wrong.
      Of courxe I don't know what version of davfs2 you are using, if you use davfs2 at all. What's fuse.dav?

      Cheers
      Werner
      p.S.: If you use davfs2 as network file system in a LAN you should reconsider your choice.

       
    • monstruooo
      monstruooo
      2009-03-14

      Hello Werner

      We are a sort of a border case. Our application, that uses davfs, can't handle too many requests. So we scale it up by employing more servers. However the load it generates seems to be just enough to create troubles. In fact, we don't have problems on LAN, but on WAN because of the delay. It looks like davfs is processing requests one by one and so because of the delay the requests start piling up in the queue, slowing down and eventually making crash the application.

      The version we are using is 1.3.2 and we are moving to 1.3.3. I am now testing 1.3.3. There is another question I would like to ask is this is why davfs makes two PUTs for a write. One seems to be an empty one. Is it possible to make it write it all in one go?

       
    • monstruooo
      monstruooo
      2009-03-15

      BTW

      One of the issues that we saw with davfs is that it deletes the contents of user files if accessed during webdav server outage.  Say the webdav server is taken out when a file was open. The user then tries to save it and receives an error. However after the webdav server is back online, even an attempt to read such a file makes davfs to erase the contents of the file by posting an empty PUT.  Are you aware of this?

       
    • Werner Baumann
      Werner Baumann
      2009-03-15

      davfs2 should not do 2 PUT requests for one write and it should not do empty PUT requests.
      Maybe this is related to the Expect 100 header which is used by davfs2 and which is not supported by all servers. Please try option "use_expect100 0". If the problem still exists, please fetch the HTTP-traffic or set option "debug most" and send a part of the traffic or log-entries that show an emptiy and a double PUT. Please also check the lost+foud directory. When davfs2 thinks it impossible to upload a file it should put it there.

      davfs2 being single threaded is a problem under high load. Only implementing multithreading would help.

      Cheers
      Werner

       
    • monstruooo
      monstruooo
      2009-03-15

      Thank you for your replies

      expect100 is set to 0. The precheck too. I tried to set  if_match_bug to 0 too but it appears that the bug affects precisely the version of apache we are using. We are running RedHat EL4 U4.  And yet davfs insists on making two PUTs instead of one. I would be actually very glad to get rid of this empty PUT, but I can't make it go away. And debug most does not log any messages to the messages log. Though I admit that we've got the whole thing already RPMed into something called fuse-davfs2-1.3.3-3.i386.rpm from a company that's developing this application. Do you think we should recompile it from sources?

       
    • monstruooo
      monstruooo
      2009-03-16

      Hi Werner

      I have a capture. Where should I upload it?

      Basically this is a copy paste from the console:

      [root@CA1-H wd_load]# echo no > /var/openweb/user/yes.txt
      [root@CA1-H wd_load]#
      15.859903 192.168.90.12 -> 6.6.6.6      TCP 19905 > http [ACK] Seq=764 Ack=6438317 Win=7576 Len=0
      15.860193      6.6.6.6 -> 192.168.90.12 HTTP Continuation or non-HTTP traffic
      15.860200      6.6.6.6 -> 192.168.90.12 HTTP Continuation or non-HTTP traffic
      15.860205 192.168.90.12 -> 6.6.6.6      TCP 19905 > http [ACK] Seq=764 Ack=6441237 Win=8956 Len=0
      15.860214      6.6.6.6 -> 192.168.90.12 HTTP Continuation or non-HTTP traffic
      15.860219      6.6.6.6 -> 192.168.90.12 HTTP Continuation or non-HTTP traffic
      15.860224      6.6.6.6 -> 192.168.90.12 HTTP Continuation or non-HTTP traffic
      15.860466      6.6.6.6 -> 192.168.90.12 HTTP Continuation or non-HTTP traffic
      15.860471      6.6.6.6 -> 192.168.90.12 HTTP Continuation or non-HTTP traffic
      15.869930 192.168.90.12 -> 6.6.6.6      TCP 19905 > http [ACK] Seq=764 Ack=6448084 Win=6196 Len=0
      15.927536 192.168.90.12 -> 6.6.6.6      TCP 19905 > http [FIN, ACK] Seq=764 Ack=6448084 Win=52560 Len=0
      15.927692      6.6.6.6 -> 192.168.90.12 TCP http > 19905 [ACK] Seq=6448084 Ack=765 Win=7984 Len=0
      17.904265 192.168.90.12 -> 6.6.6.6      TCP 19906 > http [SYN] Seq=0 Len=0 MSS=1460 WS=2
      17.905105      6.6.6.6 -> 192.168.90.12 TCP http > 19906 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 WS=2
      17.905117 192.168.90.12 -> 6.6.6.6      TCP 19906 > http [ACK] Seq=1 Ack=1 Win=5840 Len=0
      17.905145 192.168.90.12 -> 6.6.6.6      HTTP PUT /openweb/yes.txt HTTP/1.1
      17.905320      6.6.6.6 -> 192.168.90.12 TCP http > 19906 [ACK] Seq=1 Ack=416 Win=6912 Len=0
      17.906313      6.6.6.6 -> 192.168.90.12 HTTP HTTP/1.1 204 No Content
      17.906320 192.168.90.12 -> 6.6.6.6      TCP 19906 > http [ACK] Seq=416 Ack=308 Win=6912 Len=0
      17.906343      6.6.6.6 -> 192.168.90.12 TCP http > 19906 [FIN, ACK] Seq=308 Ack=416 Win=6912 Len=0
      17.906865 192.168.90.12 -> 6.6.6.6      TCP 19906 > http [FIN, ACK] Seq=416 Ack=309 Win=6912 Len=0
      17.907030      6.6.6.6 -> 192.168.90.12 TCP http > 19906 [ACK] Seq=309 Ack=417 Win=6912 Len=0
      17.907179 192.168.90.12 -> 6.6.6.6      TCP 19907 > http [SYN] Seq=0 Len=0 MSS=1460 WS=2
      17.908020      6.6.6.6 -> 192.168.90.12 TCP http > 19907 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 WS=2
      17.908031 192.168.90.12 -> 6.6.6.6      TCP 19907 > http [ACK] Seq=1 Ack=1 Win=5840 Len=0
      17.908048 192.168.90.12 -> 6.6.6.6      HTTP HEAD /openweb/yes.txt HTTP/1.1
      17.908222      6.6.6.6 -> 192.168.90.12 TCP http > 19907 [ACK] Seq=1 Ack=398 Win=6912 Len=0
      17.908367      6.6.6.6 -> 192.168.90.12 HTTP HTTP/1.1 200 OK
      17.908374 192.168.90.12 -> 6.6.6.6      TCP 19907 > http [ACK] Seq=398 Ack=251 Win=6912 Len=0
      17.908384      6.6.6.6 -> 192.168.90.12 TCP http > 19907 [FIN, ACK] Seq=251 Ack=398 Win=6912 Len=0
      17.908954 192.168.90.12 -> 6.6.6.6      TCP 19907 > http [FIN, ACK] Seq=398 Ack=252 Win=6912 Len=0
      17.909113      6.6.6.6 -> 192.168.90.12 TCP http > 19907 [ACK] Seq=252 Ack=399 Win=6912 Len=0
      28.154576 192.168.90.12 -> 6.6.6.6      TCP 19908 > http [SYN] Seq=0 Len=0 MSS=1460 WS=2
      28.155128      6.6.6.6 -> 192.168.90.12 TCP http > 19908 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 WS=2
      28.155140 192.168.90.12 -> 6.6.6.6      TCP 19908 > http [ACK] Seq=1 Ack=1 Win=5840 Len=0
      28.155157 192.168.90.12 -> 6.6.6.6      TCP [TCP segment of a reassembled PDU]
      28.155222 192.168.90.12 -> 6.6.6.6      HTTP PUT /openweb/yes.txt HTTP/1.1
      28.155333      6.6.6.6 -> 192.168.90.12 TCP http > 19908 [ACK] Seq=1 Ack=416 Win=6912 Len=0
      28.155360      6.6.6.6 -> 192.168.90.12 TCP http > 19908 [ACK] Seq=1 Ack=419 Win=6912 Len=0
      28.155723      6.6.6.6 -> 192.168.90.12 HTTP HTTP/1.1 204 No Content
      28.155730 192.168.90.12 -> 6.6.6.6      TCP 19908 > http [ACK] Seq=419 Ack=308 Win=6912 Len=0
      28.155741      6.6.6.6 -> 192.168.90.12 TCP http > 19908 [FIN, ACK] Seq=308 Ack=419 Win=6912 Len=0
      28.156278 192.168.90.12 -> 6.6.6.6      TCP 19908 > http [FIN, ACK] Seq=419 Ack=309 Win=6912 Len=0
      28.156438      6.6.6.6 -> 192.168.90.12 TCP http > 19908 [ACK] Seq=309 Ack=420 Win=6912 Len=0
      28.156585 192.168.90.12 -> 6.6.6.6      TCP 19909 > http [SYN] Seq=0 Len=0 MSS=1460 WS=2
      28.157309      6.6.6.6 -> 192.168.90.12 TCP http > 19909 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 WS=2
      28.157321 192.168.90.12 -> 6.6.6.6      TCP 19909 > http [ACK] Seq=1 Ack=1 Win=5840 Len=0
      28.157337 192.168.90.12 -> 6.6.6.6      HTTP HEAD /openweb/yes.txt HTTP/1.1
      28.157505      6.6.6.6 -> 192.168.90.12 TCP http > 19909 [ACK] Seq=1 Ack=398 Win=6912 Len=0
      28.157631      6.6.6.6 -> 192.168.90.12 TCP [TCP segment of a reassembled PDU]
      28.157638 192.168.90.12 -> 6.6.6.6      TCP 19909 > http [ACK] Seq=398 Ack=270 Win=6912 Len=0
      28.157650      6.6.6.6 -> 192.168.90.12 TCP http > 19909 [FIN, ACK] Seq=270 Ack=398 Win=6912 Len=0
      28.158255 192.168.90.12 -> 6.6.6.6      TCP 19909 > http [FIN, ACK] Seq=398 Ack=271 Win=6912 Len=0
      28.158407      6.6.6.6 -> 192.168.90.12 TCP http > 19909 [ACK] Seq=271 Ack=399 Win=6912 Len=0
      16 packets dropped
      5663 packets captured
      [root@CA1-H davfs2-1.3.3]#

      And this is our configuration:

      dav_user        root
      dav_group       root
      debug           most
      precheck        0
      use_expect100   0
      use_locks       0
      cache_dir       /var/cache/davfs2
      cache_size      6000
      table_size      36864

      The rest of options are in default

       
    • monstruooo
      monstruooo
      2009-03-16

      BTW

      I was reading FAQ and saw there something about empty files. Reminds me of that empty PUT we have first. However we don't use locks, but maybe davfs ignores this setting.

      This is what we have in our fstab:

      http://webdavserver/openweb/ /var/openweb/user davfs user,noauto,nolocks 0 0

       
    • Werner Baumann
      Werner Baumann
      2009-03-16

      Does not look like an empty PUT. Both PUT requests are about 400 bytes. As davfs2 sends very few headers there should be a body of about 200 Bytes in both cases.
      The second PUT happens about 10 seconds later and has three more bytes. This looks like the file has been changed imediately after the first PUT.

      But what is strange is this:
      28.155222 192.168.90.12 -> 6.6.6.6 HTTP PUT /openweb/yes.txt HTTP/1.1
      28.155333 6.6.6.6 -> 192.168.90.12 TCP http > 19908 [ACK] Seq=1 Ack=416 Win=6912 Len=0
      28.155360 6.6.6.6 -> 192.168.90.12 TCP http > 19908 [ACK] Seq=1 Ack=419 Win=6912 Len=0
      28.155723 6.6.6.6 -> 192.168.90.12 HTTP HTTP/1.1 204 No Content
      Why is ACK increased by three for no apparent reason? It would be helpful to see the full traffic (including the content of the packages). Could you send it to my private email address werner.baumann@onlinehome.de ?

      There should be more log files than "messages" on your system. /var/log should have a lot of files. Please check them for the davfs2 log. It would be important to see whether there have been additional file system calls. davfs2 uses log-facility DAEMON and log-level DEBUG. Where the messages end up depends on the configuration of your syslog daemon.

      "Though I admit that we've got the whole thing already RPMed into something called fuse-davfs2-1.3.3-3.i386.rpm from a company that's developing this application. Do you think we should recompile it from sources?  "
      You should ask them for the sources and look for patches. You might also tell the name of the company and their website.

      Cheers
      Werner

       
    • monstruooo
      monstruooo
      2009-03-16

      Well. There is only one file. The empty PUT always appears immediately. The second one appears after delay_upload expires. The company is actually OpenWave, but they have no downloads on their site. I am checking logging with them right now. I will ask them for sources, but I bet they do it from that one you have on your site

      By the way. We are impressed by davfs in principle. You are doing a great work. From the moment davfs becomes multi-threaded it will be a killer. We tried NFS and it has its own host of problems. It's no panacea

       
    • monstruooo
      monstruooo
      2009-03-16

      I am not confusing something? This is what I should have in syslog.conf

      daemon.debug                                            /var/log/davfs2.conf

      and this in davfs2.conf

      debug           most

       
    • Werner Baumann
      Werner Baumann
      2009-03-17

      You don't need "/var/log/davfs2.conf". But your syslog daemon should save log-messages from the mount.davfs daemon in some file under /var/log. This may be messages, syslog, daemon.log or whatsoever, but it should not send them to /dev/null. If your RedHat system is not really weird, I bet they are in one of these files already.

      "debug most" must be in davfs2.conf. It tells davfs2 to send a lot of debug messages to the syslog daemon.

      The HTTP-trafic you sent shows indeed an empty PUT (Content-Length: 0) (the 200 Bytes are authentication data). But the question is why. The debug messages would show the requests from the kernel to mount.davfs and many of the actions of mount.davfs. This would help to see where the problem occurs.

      Can you tell, how you created the file?

      Cheers
      Werner

       
    • monstruooo
      monstruooo
      2009-03-18

      Hi Werner

      The file was created by echoing a string into the file. But I was monitoring hundreds of files over the last months and it's always like this.

      Another thing is that if I, say, take WebDAV server offline and then try to write a file, I get an error. However, if I try access such a file later, before fetching it davfs first empties it with this empty PUT !!! :)

      I am checking my logs but can find nothing until now

       
    • monstruooo
      monstruooo
      2009-03-18

      Another thing that I wanted to ask is about the cryptic reference to some well known web server in davfs2.conf man page.

      "if_match_bug
                    Some servers do not handle If-Match and If-None-Match-headers correctly. This option tells  mount.davfs  to use HEAD instead of thes headers.  0 = no, 1 = yes.
                    Default: 1 (this bug is in the most widely used HTTP-server)"

      My question is if this is Apache. Because I would gladly set it to 0 but then I get precondition failed errors from Apache.

       
    • Werner Baumann
      Werner Baumann
      2009-03-18

      if_match_bug:
      Yes it is Apache with mod_dav and mod_dav_fs. The latest one or two releases of apache should no longer have this bug. But your version seems to be one of the version with this bug.

      Empty PUT:
      Are there *always* two PUTs with the first one empty, or does this only happen when the server could temporarily not been reached? In the second case:
      When you open a file, that is not yet opened by any process, but is dirty (means changed locally and the changes are not yet stored on the server), then davfs2 will first try to upload this dirty file. But I have still no idea, why this file is empty.

      Please check your syslog configuration. Maybe it does not sove all log messages. But I can't help with the syslog configuration (never had problems on Debian an so I am not familiar with this).

      Please also check the lost+found directory in your mounted davfs2 file system. davfs2 will move files into this directory when it thinks it will be permanently impossible to save the file to the server.

      Cheers
      Werner

       
    • monstruooo
      monstruooo
      2009-03-19

      Hi Werner

      I am trying to get this logging thing right, but I guess I need to get OW to fix their rpm or just recompile everything myself.  So I will stop bothering you as without logs it's impossible to figure out what this application thinks it's doing with these empty PUTs.

      There is always an empty PUT. Always.  It's just that after outages there comes an empty PUT and nothing else.

      Thanks for your help. I will get back to you when I get logging working

       
    • monstruooo
      monstruooo
      2009-03-21

      Hi Werner

      Just to let you know. After much work I succeeded to port the latest Apache RPM 2.2.3 for Redhat to our EL4 U4 and even get it working. However, setting if_match to 0 keeps generating Precondition Failed errors on writes. This part looks hopeless

       
    • Werner Baumann
      Werner Baumann
      2009-03-21

      The if_match-bug was fixed in Apache 2.2.8. So it will not work with apache 2.2.3.

      I use Debian because of their slow release cycle (and many other reasons) and because I don't want to upgrade every other day. But Debian Lenny (stable) comes with apache 2.2.9 while Etch (oldstable) had 2.2.3. So it looks like Debian is on the bleeding edge, at least compared to Red Hat Enterprise Linux.

      The latest release of Apache is 2.2.11.

      I had testet the Apache fix for this bug before the release of 2.2.8 by patching 2.2.3 with this fix. I will test apache 2.2.9 these days and send you notice about the result.

      Cheers
      Werner

       
    • monstruooo
      monstruooo
      2009-03-21

      Thank you

      I will check if it's possible to feed such a patch to rpmbuild. Basically I am recompiling RPMs from sources. I bet I can insert such a patch in the process. What is this patch by the way? Do you have a link?

       
    • monstruooo
      monstruooo
      2009-03-21

      The thing is that I can't find such a patch on their site. I will be satisfied with 2.2.3 or 2.2.8 if I can patch them with this fix.

       
    • monstruooo
      monstruooo
      2009-03-21

      Sorry for bothering you again.

      This patch you were talking about can be applied to 2.0.52 too? This is the version we have on the disk of RedHat EL4. I did created an RPM for 2.2.3 but I had to disable one dependency there and it was generally a lot of work.

      I would prefer to stay as close to version of Apache that's native to this RedHat release as possible.

       
    • monstruooo
      monstruooo
      2009-03-21

    • monstruooo
      monstruooo
      2009-03-21

      Do you think it will work for 2.0.52?

       
    • monstruooo
      monstruooo
      2009-03-21

      I was only unsure about what -p switch to give it. So I don't use it at all

       
    • monstruooo
      monstruooo
      2009-03-21

      Ok. The if_match works but the HEAD don't go away. The empty PUT is also here

      247.716477 10.168.250.211 -> 192.168.200.223 HTTP PUT /openweb/yes.txt HTTP/1.1
      247.801395 192.168.200.223 -> 10.168.250.211 HTTP HTTP/1.1 204 No Content
      247.803481 10.168.250.211 -> 192.168.200.223 HTTP HEAD /openweb/yes.txt HTTP/1.1
      247.803734 192.168.200.223 -> 10.168.250.211 HTTP HTTP/1.1 200 OK
      258.802667 10.168.250.211 -> 192.168.200.223 HTTP PUT /openweb/yes.txt HTTP/1.1
      258.803488 192.168.200.223 -> 10.168.250.211 HTTP HTTP/1.1 204 No Content
      258.805214 10.168.250.211 -> 192.168.200.223 HTTP HEAD /openweb/yes.txt HTTP/1.1
      261.730322 10.168.250.222 -> 192.168.200.223 HTTP GET /openweb/test.html  HTTP/1.0
      261.730507 192.168.200.223 -> 10.168.250.222 HTTP HTTP/1.1 200 OK (text/html)

       
1 2 3 > >> (Page 1 of 3)