|
From: Frank Tetzel <te...@us...> - 2010-08-17 16:10:40
Attachments:
repotest.tar.xz
|
Hi all, i had some thoughts about an addon repository for GAE i want to share with you. As physfs is now integrated, addon installation is really easy. There's an addons directory next to the config files where one can drop one's addons, even still zipped. So i thought about some kind of addon manager (could be also integrated) which should work quite similar to package managers in linux distributions, no dependencies which makes it much more easier. So, i made a simple prototype, see attachment. It's a command line tool, so start it with a shell. It depends on libarchive, libcurl and libmhash, xdelta is included as it does not have a lib. I didn't tried it on windows. Start it without any argument to get a list of mods and then type the number of the mod to get it (there's only military, but use your imagination ;) ). Run it with one argument (e.g. ./repotest 1) and it switches the repository to simulate an update. Have a look at local and repo afterwards. Be warned it's just hacked together. When running the update you will notice that it uses xdelta for binary patches, so the download would be as small as possible (megapack v3 to v4 gives a 22MB compressed patch instead of 118MB full v4, by military it's 3MB to 14MB). There are some problems with it. The decoder/patcher sometimes does not work correctly, atm. And we would need to add .tar support to physfs. Unpacking a tarball and then later repacking the same files gives different checksums (metadata of files?) which xdelta does not like. For the first version we could skip this feature. The main problem is to get a server. Atm it's on my quota on university which isn't big. We can use it for testing but not really for releasing. So, what do you think? Do we even need such a thing? I just had fun hacking on it... Greetings, Frank. |
|
From: James McCulloch <si...@gm...> - 2010-08-18 13:09:13
|
Hi Frank, what nice surprise this was :-) > i had some thoughts about an addon repository for GAE i want to share with > you. > <snip/> > So, i made a simple prototype, see attachment. It's a command line tool, so > start it with a shell. It depends on libarchive, libcurl and libmhash, xdelta > is included as it does not have a lib. I didn't tried it on windows. Very nice, tried it out and downloaded military, as I didn't have it on my linux box anyway ;-) Haven't looked at the code or tried to get it to work with windows, but everything you mentioned for deps looks like it should be fine. > Start it without any argument to get a list of mods and then type the number > of the mod to get it (there's only military, but use your imagination ;) ). > Run it with one argument (e.g. ./repotest 1) and it switches the repository to > simulate an update. Have a look at local and repo afterwards. Be warned it's > just hacked together. > > When running the update you will notice that it uses xdelta for binary > patches, so the download would be as small as possible (megapack v3 to v4 > gives a 22MB compressed patch instead of 118MB full v4, by military it's 3MB > to 14MB). There are some problems with it. The decoder/patcher sometimes does > not work correctly, atm. And we would need to add .tar support to physfs. > Unpacking a tarball and then later repacking the same files gives different > checksums (metadata of files?) which xdelta does not like. For the first version > we could skip this feature. I take it the tar'ed file is what is being diffed ? Seems to make most sense, a shame we couldn't just do it all zipped, but diffs of that isn't going to work ... but tar is just a straightforward archive format, so it would be fairly straight forward to add to PhysFS. > The main problem is to get a server. Atm it's on my quota on university which > isn't big. We can use it for testing but not really for releasing. I've been thinking of registering a domain name for us for a while, I have a friend you can host for us, but I'd have to make enquiries about how much he pays for upstream data, before I can claim that will be doable. If anyone has good ideas for a domain name, I'll at least do that soon either way. > So, what do you think? Do we even need such a thing? I just had fun hacking on > it... I think it sounds great, possibly we could tie https://sourceforge.net/apps/trac/glestae/ticket/101 in to it as well, I think hailstone was mostly looking for program updates, but both program and custom data would be cool in my opinion. Will have a closer look at what you've done soon, but I vote a big YES! I have my old linux box back, so I'm looking to get the multiplayer all fixed up in the near future (and get 0.3 out too, of course...), but the major source tree changes are, I think, all done now, so if wanted to branch and continue work, I promise not move things around any more :-) Also, as 0.3 draws nearer, I'm thinking about how we should proceed with version numbers again, but I'll save that for another day, because it's getting late and I'm rather tired. Cheers, - James |
|
From: Frank Tetzel <te...@us...> - 2010-08-19 12:37:06
|
Hi James, > Very nice, tried it out and downloaded military, as I didn't have it > on my linux box anyway ;-) it's not really the full thing. I think i accidently missed the scenarios, for example. You might want to get the real thing > I take it the tar'ed file is what is being diffed ? Seems to make most > sense, a shame we couldn't just do it all zipped, but diffs of that > isn't going to work ... but tar is just a straightforward archive > format, so it would be fairly straight forward to add to PhysFS. Yep, the diff is between the tar files. xdelta can only handle single files and compression changes the binaries too much, so the diff is much bigger. Tarball seemed to be the easiest way to go. The diff is also always to the latest version, so no incremental update. xdelta can merge diffs. > I've been thinking of registering a domain name for us for a while, I > have a friend you can host for us, but I'd have to make enquiries > about how much he pays for upstream data, before I can claim that will > be doable. If anyone has good ideas for a domain name, I'll at least > do that soon either way. Cool. No need to rush here. We don't need it very soon. > I think it sounds great, possibly we could tie > https://sourceforge.net/apps/trac/glestae/ticket/101 in to it as well, > I think hailstone was mostly looking for program updates, but both > program and custom data would be cool in my opinion. That's a bit difficult. On Linux normally the package manager keeps the software up-to-date and one needs root rights to write to the common install folders. Which makes me think that we need some kind of compatibility check if the mod runs on the installed glest version. > Will have a closer look at what you've done soon, but I vote a big > YES! I have my old linux box back, so I'm looking to get the > multiplayer all fixed up in the near future (and get 0.3 out too, of > course...), but the major source tree changes are, I think, all done > now, so if wanted to branch and continue work, I promise not move > things around any more :-) That actually depends if we want to integrate it and how. Or use a separate application as downloading takes some time. Either way downloading should be in the background. Atm, i'll better help getting 0.3 ready... > Also, as 0.3 draws nearer, I'm thinking about how we should proceed > with version numbers again, but I'll save that for another day, > because it's getting late and I'm rather tired. I think we should go on with 0.4. The third number should be used for bugfix releases. Greetings, Frank. |
|
From: James McCulloch <si...@gm...> - 2010-08-20 11:43:11
|
On Thu, Aug 19, 2010 at 10:01 PM, Frank Tetzel <te...@us...> wrote: >> I've been thinking of registering a domain name for us for a while, I >> have a friend you can host for us, but I'd have to make enquiries >> about how much he pays for upstream data, before I can claim that will >> be doable. If anyone has good ideas for a domain name, I'll at least >> do that soon either way. > > Cool. No need to rush here. We don't need it very soon. Yeah, I'll take my time :-) If and when I do it, we'll also need a website! Which is more work... >> I think it sounds great, possibly we could tie >> https://sourceforge.net/apps/trac/glestae/ticket/101 in to it as well, >> I think hailstone was mostly looking for program updates, but both >> program and custom data would be cool in my opinion. > > That's a bit difficult. On Linux normally the package manager keeps the software > up-to-date and one needs root rights to write to the common install folders. > Which makes me think that we need some kind of compatibility check if the mod > runs on the installed glest version. Ok, didn't really consider that... maybe we can have an auto update for windows and do a version check on linux, and simply inform the user an update is available. Of course this would require us to provide our own packages, haven't played with CPack at all yet, but that should be possible to do without too much pain. The compatibility check could be tricky, we could come up with some way for mod authors to tag their work with the minimum version, to do the job for us... but then we'd have to decide where such a tag would be, 'mod' could mean a whole new tech-tree, or just some scenarios for magitech, etc.. > That actually depends if we want to integrate it and how. Or use a separate > application as downloading takes some time. Either way downloading should be > in the background. Atm, i'll better help getting 0.3 ready... I was thinking integrated would be nice, but I guess we could do it external and then integrate it later. >> Also, as 0.3 draws nearer, I'm thinking about how we should proceed >> with version numbers again, but I'll save that for another day, >> because it's getting late and I'm rather tired. > > I think we should go on with 0.4. The third number should be used for bugfix > releases. Yeah, this is pretty much what I was thinking... I might spend some more time shuffling tickets around on the weekend, and create milestones for 0.4 & 0.5 and move the 'features' from 0.3.1 and 0.3.2 into those (and possibly some into 0.3). I'm thinking atm my main focuses will be multi-player for 0.4 and AI for 0.5. Create a ticket for this, and a separate one for tar support and maybe the minimum-version thing and anything else you can think of, I'll leave it to you to decide what milestone to attach them to :-) Cheers, - James |