I was thinking of forking this project for an iPhone specific solution (or contribute under this project).
Something the community of iPhone developers could use that was very user friendly and required a minimal amount of configuration.
I think we also need a security solution as well as this Application would be used for all apps. ie, one install per many.
Therefore, the security would ensure that iPhone apps can't overwrite / read backed up data from other iPhone apps. My basic thinking there was for the iPhone app to generate a random secret which it stores on the iPhone and sends to the Bonjour server. No other app is allowed to overwrite or read from that store without that secret.
Finally, in case of a recovery, the user would be required to retype the secret from the server into the iPhone in the use case the iPhone app lost the secret.
A password would be optional as well for those who are concerned about outside users hacking into their local network.
Let me know what you think of the idea of a fork / seperate build and my thoughts on security.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
SyncDocs has sample code for iPhone applications that can be included in open or closed source software (LGPL). The most up to date code is in CVS. It's an Xcode project wrapped inside an Eclipse project (because Eclipse has much better CVS support). The module name is "SyncDocs-Client-iPhone".
I believe SyncDocs currently meets the needs for something that's user friendly and requires a minimal amount of configuration. These are, at least, my main goals for the project.
Currently there is no way for any application to get files backed up from another application (unless the user specifically adds a file to the list from the downloads directory). In fact, an application cannot directly get files that it has itself backed up without user intervention.
Each application sends it's app ID as part of the protocol. If that's not sufficient I suppose adding a random secret would make that a little more secure. There is a problem however. The whole point of backing up files is to guard against the app install bug where everything for the app is cleared. So the only case where the user would reasonably want to restore files would imply the secret was lost. It increases security, but it would essentially force the user to enter random data every time the backup proved useful.
Adding HTTP Digest Authentication to the client/server protocol is in the "TODO" list. If you implemented it for me, I would be in your debt. If a user has rogue agents on their network, that is likely going to be a larger concern than an unsecured web server, but of course "weakest link" arguments in security are specious. It's not a current concern, but I fully support authentication at some point before SyncDocs moves to 1.0.
I don't believe the features you've described imply a fork is necessary, as long as the user can disable them (to improve "user-friendliness").
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Fair comments all, however, I think I need to make myself a bit more clear.
This is not an individual file syncing mechanism for the iPhone that I wish to create, but rather a backup solution for consumers who wish to be able to do backup / recovery with their application.
As you can imagine, these are two completely different species.
One involves an appreciation for the file structure (your current version, the file syncing mechanism), and the other has more of an App View and any insight into file structure would be an advanced feature and not a common use case.
eg: for applications with many files to backup, the current solution would be rather confusing.
I'd be happy to contribute a separate application which shares code base under the SyncDocs name. But I believe it's a separate user interface.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
When you refer to "many files to backup" are you talking about 20? 30? 50? In that case, perhaps you might consider porting rsync to the iPhone.
SyncDocs provides a client and server which, generally speaking, provide what I think your requirements are.
Perhaps I'm confused because there are several things I envision for SyncDocs that are not currently implemented. These may cover your needs, they may not.
Please provide a more specific example of a case where SyncDocs doesn't measure up to what you need.
Thanks,
-Chris.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think you'll need to put yourself in the place of someone who has 100,000+ (millions if you consider the collective community) users and imagine how he'll support each user backing up his/her data.
Basically it boils down to something which is very user friendly (ie: no configuration/clicking/dragging required) but is also very secure.
Certainly syncdocs theoretically supports backing up data, but practically, it's not a mass market solution which needs to be dumbed down quite a bit.
I'm moving forward on this. I'm trying to decide between this and webdav and will probably start implementing today.
I'll try to share your code base as much as possible in case you're interested in accepting a separate application under the Syncdocs name at a later date, however I need a community based solution which is follows my description above so I'll be going in that direction.
I'll probably set something up on Google Code and leave open the possibility of re-merging back into your tree.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The SyncDocs protocol is the same as webDAV (or perhaps I should say it's based on webDAV). The difference is that SyncDocs is not a full webDAV server. It only supports GET and PUT. Not even OPTIONS.
Another important point is that the SyncDocs server runs on the user's computer. That is the one major difference between SyncDocs and all other solutions that I'm aware of. It would scale quite well to 100,000+ users :-) There is less of a need to worry about security and there is no single point of consolidated data to create a target for attackers.
There is no server out in the cloud that the user must pay a monthly fee for space on and the user can continue to synchronize even if the company providing it (for SyncDocs I guess that's me) is no longer around or offering support.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
SyncDocs improves upon webDAV by offering MD5 checksums to ensure data integrity. Although some webDAV servers may support it through a HEAD request, Apache 2 does not do so by default.
SyncDocs can also handle synchronization of many applications, whereas with a straight webDAV approach the user would need a different webDAV share to be set up for each application.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think this project needs to be able to save and restore directory structures and that is the goal I'm heading towards. But the immediate goal was something so developers would have a solution to backup and restore data, so our users stop blaming us for the Apple updater bug that erases all their data.
I'm not sure you'll get to the no configuration because you'll need to pick a server at least. As a user I'm not going to trust my financial data to a server I don't control. On the SyncDocs server side of this...based on the current experience I doubt a zero configuration server is possible either because of various networking issues.
I'd love to see a webDAV solution, though. This would allow backups to mobileMe or other webDAV severs. I've talked with Chris a little about moving the SyncDocs protocol towards that of webDAV so all you'd really have to do on the program side is ask for a webDAV server address from the user.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hey Hardy, thanks for posting. You inspired me to check out SyncDocs :)
Why can't you use a default config? "iPhoneBonjourBackupServer" .. if detected, it automatically uses it.
I think security would be up to the App developer. If he's particularly paranoid (as you have good reason to be), he can query the user for the server, but most apps I think can just go out and find the backupserver on the local network.
As for webdav, I'm curious though, were you thinking of hosting your own webdav backup server? I'm not sure people are hyper excited about hosting their financial data over the internet, either. Your probably going to have to encrypt it and that means going through the encryption stuff with Apple and iTunes. Not fun .. I've been there done that.
I had another approach I looked into today - GData API. Basically, I was going to upload portions of my backup file to the GoogleDocs text (just basically use the sqlite3 code to dump it).
Kinda kludgy, but when you think about it, pretty sweet because a user would just need a gmail username/pwd which is pretty easy to get.
I don't like that approach for various reasons though:
1. I am pretty sure Apple isn't interested in me using something other than Mobile Me for the iPhone
2. I'm pretty sure Google isn't interested in me using their Google Docs as a backup datastore that doesn't help push their Google Apps platform
3. Their objective C sample code wasn't kept up to date
4. I kept on getting kCFEDomainErrorCFNetwork errors all over the place.
Good reasons to go with it:
1. No software or local network needed.
2. Works over edge, 3G, etc.
3. Free
4. Automatically secure with username/password protection
5. Leveraging a large brand name (Google) that people are comfortable with and people know
I might have gone with it except for that error, which might be something simple that I'm missing. Still, I think it has a lot of potential for the risk takers.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Well, if you have an iPhone, download air sharing (while it's still free!) and give it a try.
It's pretty nice, I have to say (though I haven't got it to work on Vista, yet).
Basically it implements WebDav and Bonjour on the iPhone, which is pretty cool. You add the network drive to your computer and then you can peruse your iPhone as a mapped drive on your computer. Very nice.
Push/Pulling to a WebDav out on the net is cool as well, but I have to ask, what are the good options out there for the naive user? Is there a good webdav repository that any user can easily use to backup his iPhone? or would we have to maintain our own?
Maintaining a backup repository on the net isn't a simple investment. It one of those things that you spend oodles of time and money and has very little to show for it, except when it doesn't work!
I agree with Chris, that probably the best way to go is use your local network (assuming this is what he was saying), as you can be assured that it will be continue to be supported, and for those who can't, they'll have to simply rely on the iPhone backup which hopefully was fixed in 2.1
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for the tip about Air Sharing. I'm downloading it now.
I guess it doesn't help that I want to view and edit FreeMind mindmap files[1], since Air Sharing doesn't support that format. It looks like there's no way to take the files uploaded using Air Sharing and move them to/from a different application. Apple has to explicitly allow applications that ability for that to work.
Hah! That's exactly what I wanted air share for as well.
However, I'm pretty satisfied with simply exporting to HTML and then saving those to the share on my iPhone. I like having a big monitor for editting maps.
My main freemind install/maps are on my vista though, so I have to wait for the upgrade for better connectivity.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I was thinking of forking this project for an iPhone specific solution (or contribute under this project).
Something the community of iPhone developers could use that was very user friendly and required a minimal amount of configuration.
I think we also need a security solution as well as this Application would be used for all apps. ie, one install per many.
Therefore, the security would ensure that iPhone apps can't overwrite / read backed up data from other iPhone apps. My basic thinking there was for the iPhone app to generate a random secret which it stores on the iPhone and sends to the Bonjour server. No other app is allowed to overwrite or read from that store without that secret.
Finally, in case of a recovery, the user would be required to retype the secret from the server into the iPhone in the use case the iPhone app lost the secret.
A password would be optional as well for those who are concerned about outside users hacking into their local network.
Let me know what you think of the idea of a fork / seperate build and my thoughts on security.
SyncDocs has sample code for iPhone applications that can be included in open or closed source software (LGPL). The most up to date code is in CVS. It's an Xcode project wrapped inside an Eclipse project (because Eclipse has much better CVS support). The module name is "SyncDocs-Client-iPhone".
I believe SyncDocs currently meets the needs for something that's user friendly and requires a minimal amount of configuration. These are, at least, my main goals for the project.
Currently there is no way for any application to get files backed up from another application (unless the user specifically adds a file to the list from the downloads directory). In fact, an application cannot directly get files that it has itself backed up without user intervention.
Each application sends it's app ID as part of the protocol. If that's not sufficient I suppose adding a random secret would make that a little more secure. There is a problem however. The whole point of backing up files is to guard against the app install bug where everything for the app is cleared. So the only case where the user would reasonably want to restore files would imply the secret was lost. It increases security, but it would essentially force the user to enter random data every time the backup proved useful.
Adding HTTP Digest Authentication to the client/server protocol is in the "TODO" list. If you implemented it for me, I would be in your debt. If a user has rogue agents on their network, that is likely going to be a larger concern than an unsecured web server, but of course "weakest link" arguments in security are specious. It's not a current concern, but I fully support authentication at some point before SyncDocs moves to 1.0.
I don't believe the features you've described imply a fork is necessary, as long as the user can disable them (to improve "user-friendliness").
Fair comments all, however, I think I need to make myself a bit more clear.
This is not an individual file syncing mechanism for the iPhone that I wish to create, but rather a backup solution for consumers who wish to be able to do backup / recovery with their application.
As you can imagine, these are two completely different species.
One involves an appreciation for the file structure (your current version, the file syncing mechanism), and the other has more of an App View and any insight into file structure would be an advanced feature and not a common use case.
eg: for applications with many files to backup, the current solution would be rather confusing.
I'd be happy to contribute a separate application which shares code base under the SyncDocs name. But I believe it's a separate user interface.
When you refer to "many files to backup" are you talking about 20? 30? 50? In that case, perhaps you might consider porting rsync to the iPhone.
SyncDocs provides a client and server which, generally speaking, provide what I think your requirements are.
Perhaps I'm confused because there are several things I envision for SyncDocs that are not currently implemented. These may cover your needs, they may not.
Please provide a more specific example of a case where SyncDocs doesn't measure up to what you need.
Thanks,
-Chris.
Thanks for getting back to me.
I think you'll need to put yourself in the place of someone who has 100,000+ (millions if you consider the collective community) users and imagine how he'll support each user backing up his/her data.
Basically it boils down to something which is very user friendly (ie: no configuration/clicking/dragging required) but is also very secure.
Certainly syncdocs theoretically supports backing up data, but practically, it's not a mass market solution which needs to be dumbed down quite a bit.
I'm moving forward on this. I'm trying to decide between this and webdav and will probably start implementing today.
I'll try to share your code base as much as possible in case you're interested in accepting a separate application under the Syncdocs name at a later date, however I need a community based solution which is follows my description above so I'll be going in that direction.
I'll probably set something up on Google Code and leave open the possibility of re-merging back into your tree.
The SyncDocs protocol is the same as webDAV (or perhaps I should say it's based on webDAV). The difference is that SyncDocs is not a full webDAV server. It only supports GET and PUT. Not even OPTIONS.
Another important point is that the SyncDocs server runs on the user's computer. That is the one major difference between SyncDocs and all other solutions that I'm aware of. It would scale quite well to 100,000+ users :-) There is less of a need to worry about security and there is no single point of consolidated data to create a target for attackers.
There is no server out in the cloud that the user must pay a monthly fee for space on and the user can continue to synchronize even if the company providing it (for SyncDocs I guess that's me) is no longer around or offering support.
SyncDocs improves upon webDAV by offering MD5 checksums to ensure data integrity. Although some webDAV servers may support it through a HEAD request, Apache 2 does not do so by default.
SyncDocs can also handle synchronization of many applications, whereas with a straight webDAV approach the user would need a different webDAV share to be set up for each application.
I think this project needs to be able to save and restore directory structures and that is the goal I'm heading towards. But the immediate goal was something so developers would have a solution to backup and restore data, so our users stop blaming us for the Apple updater bug that erases all their data.
I'm not sure you'll get to the no configuration because you'll need to pick a server at least. As a user I'm not going to trust my financial data to a server I don't control. On the SyncDocs server side of this...based on the current experience I doubt a zero configuration server is possible either because of various networking issues.
I'd love to see a webDAV solution, though. This would allow backups to mobileMe or other webDAV severs. I've talked with Chris a little about moving the SyncDocs protocol towards that of webDAV so all you'd really have to do on the program side is ask for a webDAV server address from the user.
Hey Hardy, thanks for posting. You inspired me to check out SyncDocs :)
Why can't you use a default config? "iPhoneBonjourBackupServer" .. if detected, it automatically uses it.
I think security would be up to the App developer. If he's particularly paranoid (as you have good reason to be), he can query the user for the server, but most apps I think can just go out and find the backupserver on the local network.
As for webdav, I'm curious though, were you thinking of hosting your own webdav backup server? I'm not sure people are hyper excited about hosting their financial data over the internet, either. Your probably going to have to encrypt it and that means going through the encryption stuff with Apple and iTunes. Not fun .. I've been there done that.
I had another approach I looked into today - GData API. Basically, I was going to upload portions of my backup file to the GoogleDocs text (just basically use the sqlite3 code to dump it).
Kinda kludgy, but when you think about it, pretty sweet because a user would just need a gmail username/pwd which is pretty easy to get.
I don't like that approach for various reasons though:
1. I am pretty sure Apple isn't interested in me using something other than Mobile Me for the iPhone
2. I'm pretty sure Google isn't interested in me using their Google Docs as a backup datastore that doesn't help push their Google Apps platform
3. Their objective C sample code wasn't kept up to date
4. I kept on getting kCFEDomainErrorCFNetwork errors all over the place.
Good reasons to go with it:
1. No software or local network needed.
2. Works over edge, 3G, etc.
3. Free
4. Automatically secure with username/password protection
5. Leveraging a large brand name (Google) that people are comfortable with and people know
I might have gone with it except for that error, which might be something simple that I'm missing. Still, I think it has a lot of potential for the risk takers.
Well, if you have an iPhone, download air sharing (while it's still free!) and give it a try.
It's pretty nice, I have to say (though I haven't got it to work on Vista, yet).
Basically it implements WebDav and Bonjour on the iPhone, which is pretty cool. You add the network drive to your computer and then you can peruse your iPhone as a mapped drive on your computer. Very nice.
Push/Pulling to a WebDav out on the net is cool as well, but I have to ask, what are the good options out there for the naive user? Is there a good webdav repository that any user can easily use to backup his iPhone? or would we have to maintain our own?
Maintaining a backup repository on the net isn't a simple investment. It one of those things that you spend oodles of time and money and has very little to show for it, except when it doesn't work!
I agree with Chris, that probably the best way to go is use your local network (assuming this is what he was saying), as you can be assured that it will be continue to be supported, and for those who can't, they'll have to simply rely on the iPhone backup which hopefully was fixed in 2.1
Thanks for the tip about Air Sharing. I'm downloading it now.
I guess it doesn't help that I want to view and edit FreeMind mindmap files[1], since Air Sharing doesn't support that format. It looks like there's no way to take the files uploaded using Air Sharing and move them to/from a different application. Apple has to explicitly allow applications that ability for that to work.
[1]http://freemind.sourceforge.net/wiki/index.php/Main_Page
Hah! That's exactly what I wanted air share for as well.
However, I'm pretty satisfied with simply exporting to HTML and then saving those to the share on my iPhone. I like having a big monitor for editting maps.
My main freemind install/maps are on my vista though, so I have to wait for the upgrade for better connectivity.
I'm working on a native FreeMind editor for the iPhone.
See http://freemind.sourceforge.net/wiki/index.php/User:Clevans