You may use rsync via ssh after breakfast and before sunrise too.
Sorry, I simply do not understand what you intend to do, how davfs2 and rsync are meant to interact.
Anyway:
rsync via ssh is a fine tool to synchronize data on a local file system with data on a remote host. In my opinion, involving yet another tool for remote access (like davfs2) into this, will only degrade performance and reliability. So why do you want to do this? (Fooling the firewall with HTTP?)
Cheers
Werner
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
thanks for the reply. Why would I want to use rsync together with davfs2?
I have a web.de club membership that includes several GB online storage called SmartDrive. In Linux connection is possible either via a web application directly in the browser or via "webdavs://webdav.smartdrive.web.de/" and Konqueror (see "http://img.web.de/v/mail/help/redesign/smartdrive/netzlaufwerk/linux/smartdrive_01.gif". I would like to use this online storage for online backups but copying files via Konqueror or the web application is not really fast. In addition, there is no automatic way to make sure that only files that are new or have been changed since the last backup are transferred. But if I could connect via rsync to the smartdrive rsync could take care to copy only files that are new and parts that changed from files that have changed which should decrease bandwidth usage dramatically. In addition, this could be integrated into a backup script that would make using the web.de smartdrive as online backup storage as simple as clicking an application launcher button in the panel to start the backup scripts which would make smartdrive much more useful (similar to services such as offered e.g. by rsync.net) at least for me.
SSH would be one way to provide a secure connection, but I guess, if I connect to a webdavs server the connection is already encrypted so therefore my question should be actually if it would be possible "to use rsync to transfer files to a webdavs server that has been mounted via davfs2 in Linux similar as I could e.g. to a locally mounted USB drive".
Steffen
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
now I see what you want to do and what the problems are.
You want to use smartdrive for backup, but the only way to access smartdrive is via http/webdav. You can not access smartdrive directly with rsync, as you would need shell-access to the remote host.
Your idea should work like this:
- You mount smartdrive (via https/Webdav) into your local file system using davfs2.
- Now you can use rsync locally to sync your directories with the mounted davfs2 file system.
- davfs2 will use WebDAV to upload the changed files to smartdrive.
I never did it myself, but it should work, as rsync uses normal file access. But there are some drowbacks compared with the usual way of using rsync via ssh:
- davfs2 will only upload complete files. It can not do the incremental stuff rsync usually does, and that makes rsync very efficient.
- davfs2 uses a local cache on disk. This will make it more responsive and your application should profit from this too. But it needs local disk space for this. You should allow for a large cache size, so rsync can do most of its work with the local cache, and davfs2 will upload most of the files in the background, when rsync has already finished.
- rsync usually does some sophisticated stuff, including checksums, to reduce the amount of data to transfer. This could be a disadvantage in this case. When rsync reads a file on the remote host, it must be transfered by davfs2 into the local cache first (if it is not already there). This could make the process really and unnecessary slow. As rsync only works as a sophisticated copy-program in your case, it might be better to use cp instead. cp has an options (-u) to copy only files that are newer than the ones in the davfs2 file system (= smartdrive) and it would not need to read the files, but only reads file meta data like mtime.
A command like "cp -pru directory/to/backup dav/" might do the job. It shold not download files (like rsync might do, but I am not sure) (please look at the manuals of cp and rsync).
In any case: please carefully examine the result of your rsync or cp commands, before you trust in this kind of backup.
Cheers
Werner
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
thinking it over again, I found a serious problem that will probably make your intended use of davfs2 impossible. The problem:
Backup tools, and cp -pu and rsync too, depend on correctly set file times, especially the modification time (mtime), to work properly. davfs2 can not guarantee this for two reasons.
1) There is a bug in davfs2, that prevents the file system from setting the file times. The -p option of cp will not work at the moment. I will fix that bug in the next release.
But even with that bug fixed, there remains another problem.
2) When you write to a file on a davfs2 file system, davfs2 will locally set and remember file times according to the rules, and everthing seems to work fine. But there are no means in WebDAV to allow davfs2 to set the Last-Modified-Date for the files on the server. The server will set file times in its own way when the file is uploaded (after it has been closed locally). Because of this, the Last-Modified-Date on the server will usually be newer than the locally remembered mtime. This is not a problem, as long as the file system is mounted. davfs2 will remember both times, using the local times when accessed from local applications, so these applications will not be confused.
But there can't be different modification times for ever. At some time davfs2 has to synchronize the local mtime with the servers Last-modified-date. This is done, when the file system is unmounted and mounted again.
Even if davfs2 would not do this syschronization: between two mounts it will only remember information of files in the cache.
Conclusion:
When you unmount a davfs2 file system and mount it again at some later time, file times may have changed according to the time information from the server. Tools like cp -pu and rsync can not rely on these times to determine what files have changed.
Cheers
Werner
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Is it possible to use rsync via ssh after mounting a webDAV server via davfs2 in Linux (e.g. Ubuntu 8.04)?
You may use rsync via ssh after breakfast and before sunrise too.
Sorry, I simply do not understand what you intend to do, how davfs2 and rsync are meant to interact.
Anyway:
rsync via ssh is a fine tool to synchronize data on a local file system with data on a remote host. In my opinion, involving yet another tool for remote access (like davfs2) into this, will only degrade performance and reliability. So why do you want to do this? (Fooling the firewall with HTTP?)
Cheers
Werner
Hello Werner,
thanks for the reply. Why would I want to use rsync together with davfs2?
I have a web.de club membership that includes several GB online storage called SmartDrive. In Linux connection is possible either via a web application directly in the browser or via "webdavs://webdav.smartdrive.web.de/" and Konqueror (see "http://img.web.de/v/mail/help/redesign/smartdrive/netzlaufwerk/linux/smartdrive_01.gif". I would like to use this online storage for online backups but copying files via Konqueror or the web application is not really fast. In addition, there is no automatic way to make sure that only files that are new or have been changed since the last backup are transferred. But if I could connect via rsync to the smartdrive rsync could take care to copy only files that are new and parts that changed from files that have changed which should decrease bandwidth usage dramatically. In addition, this could be integrated into a backup script that would make using the web.de smartdrive as online backup storage as simple as clicking an application launcher button in the panel to start the backup scripts which would make smartdrive much more useful (similar to services such as offered e.g. by rsync.net) at least for me.
SSH would be one way to provide a secure connection, but I guess, if I connect to a webdavs server the connection is already encrypted so therefore my question should be actually if it would be possible "to use rsync to transfer files to a webdavs server that has been mounted via davfs2 in Linux similar as I could e.g. to a locally mounted USB drive".
Steffen
Hello Steffen,
now I see what you want to do and what the problems are.
You want to use smartdrive for backup, but the only way to access smartdrive is via http/webdav. You can not access smartdrive directly with rsync, as you would need shell-access to the remote host.
Your idea should work like this:
- You mount smartdrive (via https/Webdav) into your local file system using davfs2.
- Now you can use rsync locally to sync your directories with the mounted davfs2 file system.
- davfs2 will use WebDAV to upload the changed files to smartdrive.
I never did it myself, but it should work, as rsync uses normal file access. But there are some drowbacks compared with the usual way of using rsync via ssh:
- davfs2 will only upload complete files. It can not do the incremental stuff rsync usually does, and that makes rsync very efficient.
- davfs2 uses a local cache on disk. This will make it more responsive and your application should profit from this too. But it needs local disk space for this. You should allow for a large cache size, so rsync can do most of its work with the local cache, and davfs2 will upload most of the files in the background, when rsync has already finished.
- rsync usually does some sophisticated stuff, including checksums, to reduce the amount of data to transfer. This could be a disadvantage in this case. When rsync reads a file on the remote host, it must be transfered by davfs2 into the local cache first (if it is not already there). This could make the process really and unnecessary slow. As rsync only works as a sophisticated copy-program in your case, it might be better to use cp instead. cp has an options (-u) to copy only files that are newer than the ones in the davfs2 file system (= smartdrive) and it would not need to read the files, but only reads file meta data like mtime.
A command like "cp -pru directory/to/backup dav/" might do the job. It shold not download files (like rsync might do, but I am not sure) (please look at the manuals of cp and rsync).
In any case: please carefully examine the result of your rsync or cp commands, before you trust in this kind of backup.
Cheers
Werner
Hello Werner,
thanks a lot for these suggestions. I will give it a try in the next weeks to see if this kind of simple backup will work.
Cheers, Steffen
Hello Steffen,
thinking it over again, I found a serious problem that will probably make your intended use of davfs2 impossible. The problem:
Backup tools, and cp -pu and rsync too, depend on correctly set file times, especially the modification time (mtime), to work properly. davfs2 can not guarantee this for two reasons.
1) There is a bug in davfs2, that prevents the file system from setting the file times. The -p option of cp will not work at the moment. I will fix that bug in the next release.
But even with that bug fixed, there remains another problem.
2) When you write to a file on a davfs2 file system, davfs2 will locally set and remember file times according to the rules, and everthing seems to work fine. But there are no means in WebDAV to allow davfs2 to set the Last-Modified-Date for the files on the server. The server will set file times in its own way when the file is uploaded (after it has been closed locally). Because of this, the Last-Modified-Date on the server will usually be newer than the locally remembered mtime. This is not a problem, as long as the file system is mounted. davfs2 will remember both times, using the local times when accessed from local applications, so these applications will not be confused.
But there can't be different modification times for ever. At some time davfs2 has to synchronize the local mtime with the servers Last-modified-date. This is done, when the file system is unmounted and mounted again.
Even if davfs2 would not do this syschronization: between two mounts it will only remember information of files in the cache.
Conclusion:
When you unmount a davfs2 file system and mount it again at some later time, file times may have changed according to the time information from the server. Tools like cp -pu and rsync can not rely on these times to determine what files have changed.
Cheers
Werner
Hello Werner,
looks as if I will have to look into other possibilities for remote backup.
But thanks a lot for your time.
Steffen