#236 Server/Workspace configuration & Archetype build streamlining

Kevin Zheng
Bloody Shade

At the current state, the build process (at least under windows) involves several pre-reqs and running different scripts in order to make a base for compilation and archetype construction, none of which are very well documented and/or detailed, and some require some manual modifications in order to work properly.

I'm opening this so we can discuss a better way to do things, a way to streamline these processes.

Here's a few suggestions:
1- Remove perl dependency by porting required scripts to python, which is already used throughout the code and should be able to do anything currently done through the perl scripts
2- Remove unnecessary steps in dev workspace creation and unnecessary scripts from the build process
3- Properly document build steps, and keep said documentation up to date
4- Combine scripts that work on the same level (mostly archetype stuff) into one script, instead of having several scattered around
5- Rework folder structures so that there is no need for duplicate folders or copy/pasting content around. Also, centralize scripts in one location, and have them look/work for/inside the new folder structures

These are just my opinions, things I think would make it way easier in the long run to maintain the code and provide a much quicker and organized way to create/run a server.


  • Mark Wedel
    Mark Wedel

    I'm not sure how many people actually compile crossfire from source on windows (instead of using the pre-built binaries). Though it could probably be argued if it was easier, perhaps more people would do so.

    However, IMO the perl dependency isn't a major issue - if the person compiling does so, they are responsible for getting all the programs necessary. In many ways, this isn't any different on unix - one may have to get certain developer packages installed, etc, to compile it.

    In addition, I believe the java editor can collect the archetypes - so rather than re-writing the existing perl collect script, it may just be more reasonable to use the java editor to do that work (I'm not sure if it could be batched to do so)

    The other suggestions all seem fine to me. Certainly more/better documentation is never a bad thing. I'm not sure what point #4 really gains - certainly if there is a high amount of duplication, consolidating them may make sense. But many different scripts tends to be a unix thing.

    I'm also note sure what is meant by point #5. This may be something specific to compiling on windows. On unix, it really isn't much more than configure, make, make install, download/install maps, and run the server - that is pretty well optimized.

    Most of the developers of crossfire do so on unix of some type or another - so most likely, specific examples of what changes are needed may need to be spelled out, since from the unix side, the developers probably do not see the same issues.

    • Bloody Shade
      Bloody Shade

      I understand your point about pre-reqs being responsibility of the person intending to compile/use the code, that isn't really the point I'm trying to make though. What I'm trying to convey is that it just isn't necessary, why use 2 different scripting languages? It would be better to just stick to one instead, this removes some complexity from the code base and allows people to focus on writing well on one of them instead of having to know both.

      Not sure how much I like the idea of having to use an editor to just build the archetypes used by the server, it seems more fitting for the server package (or the arch package) to be able to do that itself. Then again I'm not sure what you guys prefer.

      #4 mostly goes along with #5 in this case, in windows it's a bit of a pain to get the archetype scripts to work correctly.
      Let's say I need to generate and update my arch files, the steps to do so are quite simple (though still has unnecessary steps), run (under the new structure) server/lib/collect, copy the files generated over to server/share/, run server/lib/adm/collect_images.pl and copy the files over to server/share/

      Now, first problem comes from the fact that we have to even copy files around, why not just have lib become share? it's redundant to have 2 folders with basically the same set of files.
      Second problem, it's not really well documented (although I could find more info now by checking the *nix instructions) that arch should be inside server/lib/, either way, you get errors unless you run things in specific directories (you can't simply call the scripts from anywhere)
      I have to cd into lib, run collect, then you'd think you need to go into adm to run collect_images.pl, but instead this one also needs to be run from lib, otherwise it can't find image_info. This kind of thing is really bad, first because there's hardcoded stuff they look for in specific places (collect can't run from elsewhere unless you copy util.pl over, collect_images.pl wants arch to be in the current directory) and at the same time, they are hardcoded wrongly, if it expects arch to be in server/lib, wouldn't it be better to just check for "../arch" instead of in the current directory? Same goes for image_info.

      My new update_arch.bat looks like this, now: http://codepad.org/QC6b9BIs
      Pretty straight forward, but the fact I had to do it in the first place is not that good.

      Also, why output to treasures.bld instead of just treasures? Am I missing an intermediate step here?

      Last edit: Bloody Shade 2014-04-13
  • Mark Wedel
    Mark Wedel

    The use of 2 different scripting languages is somewhat historical - at the time they were written, python barely existed and was virtually unknown, and perl was widely used. Times change. I'm just not sure it is worth it to rewrite an existing script to python just to match the scripting language used within the game. The official releases have the archetypes pre-collected, so the number of people who need to collect their own archetypes is probably a fairly low number.

    In terms of point #4, this really is a unixism, where read-only, platform independent files are supposed to be stored in the 'share' directory (in the very distant past, the server did look for those files in the lib directory, so you could use your build tree as your run tree without needing to copy any files about). However, I think even now, if you modify include/win32.h so that LIBDIR, DATADIRare 'lib' instead of 'share', that might fix up the issue you describe.

    The treasures.bld is also somewhat historical - at one time in the past, there was a file called treasures - the treasures were not collected, which made for an odd case of adding an arch required updating the arch repository and submitting a patch or the like for the treasures file.

    So logic to collect treasures was added. However, the old treasures file did not disappear immediately - it took some time to take out those different treasures and put them in the matching directory for the archetype they describe. So during that period, it would take the treasures file and collect the treasures, and basically merge those 2 things together into the treasures.bld file

    After the treasures file did disappear completely (and logic for merging the files was removed), the treasures.bld just remained.

    • Bloody Shade
      Bloody Shade

      I see, so basically the dev environment didn't seem to evolve with time, and that's also a bad thing IMO.
      About the scripting languages, I'm a perfectionist, so to me it feels wrong and bloated to have that even if the number of people having to run them is limited, to me it should be done better even if only 1 person ever had to run it, but then again, that's a personal view, although the need for two scripting languages is still unjustified at this point in time.

      About keeping both share and lib dirs, even for a unixism, it seems unjustified.

      As soon as I get some more time, I'll check out those perl scripts and see if I can port them myself. (not much into perl either, so I don't know it that well)

  • Mark Wedel
    Mark Wedel

    A lot of this comes down to time and resources. Until somewhat recently, python was not required to run the server - you could run without the python plugin. And the perl collect.pl works - if it was completely broken, then someone might have taken the effort to rewrite. Also, I think every unix based system has perl installed by default, so was never any issue of needing to get extra programs installed.

    But a lot of history and old/odd things also have to do with resources. One could spend a lot of time cleaning up various aspects, re-writing scripts, etc, which may not be a bad thing, but at the same time, doesn't actually change the way the game plays, so tend to get left behind. And in most cases, it really is a very minor difference (treasures.bld vs treasures isn't really messing up many people - I think you're the first to notice this in the past few years)

    I tend to agree with the share vs lib vs all the other directory setup, but it is set up that way per linux standard. So it probably helps those that are familiar with the standard - they would know where to look and it makes sense to them, where as the setup makes no sense for windows users. Though as noted, since those settings are isolated in the win32.h file, I would think an alternate/better setup for windows could be done.

    You also do have some old developers like me who are actually more familiar with perl than python, which is also one likely reason some of those scripts have remained in that fashion - the old timers know how to modify them when changes are needed.

  • Bloody Shade
    Bloody Shade

    Here's an early version of lib/adm/collect_images.pl ported to python by myself:

    It works fine at least under windows, I don't really have the infrastructure in place to test it on *nix to see if that works though, if anybody would be so kind as to test it.

    I compared the outputs from the original perl script with the ones from mine and they are 100% equal though, so it should be fine.

    I'm not the best programmer around, so my code might need some tweaking, but I hope it's good enough of a start.
    I'll work on collect next, again once I find some more time.

    Last edit: Bloody Shade 2014-04-19
  • Kevin Zheng
    Kevin Zheng

    • status: open --> analyzed
    • assigned_to: Kevin Zheng
    • Group: -->