Info about the dark engine

Displacer
2007-07-01
2013-04-29
  • Displacer
    Displacer
    2007-07-01

    I'm in the middle of reverse engineering system shock 2. I have some info about the database header worked out, don't know if you know it or not. Its the "dead beef" entry at the end of the header. This is used to determine if the file is a valid database file, its the first thing checked when opening a database and if it doesn't match the file is immediately closed and the open routine exits.

     
    • Filip Volejnik
      Filip Volejnik
      2007-07-02

      Thanks for the info. Actually, the format of Dark's database files is already well known. You'll find our implementation in CVS - src/base/file/FileGroup.cpp / .h - the DarkFileGroup class. What is not yet know, is some parts of the PHYS_SYSTEM chunk and AI chunks, I guess. There is some code written in proto/phys_system, that reads physical attributes of awake/asleep objects, but the contact info, which is next to this, is not well understood. WR/WRRGB format is already decoded and understood well (small exception being lightmap UV shifts, which I could not fit well). Some room/AI stuff will probably need further work. Telliamed did an awsome job on this, as well as others. You can find DarkUtils/DarkLib on his internet site.

      The property/link reading/writing will be soon finished in OPDE, services for links/properties are quite complete...

      Anyway, thanks for the info. If you'll find out anything interesting, be sure to know that I'll be very interested!

       
    • Displacer
      Displacer
      2007-07-02

      Yea, I know about most of the db being understood, I looked over DarkDBDefs.h in your source and saw some unknowns, and uninformative names (which may be unknowns) in some of the structures and as I crawl slowly through the code wanted to help fill in some of the stuff. Another thing in the header you have marked as zero and one. These are version numbers (in DarkDBHeader structure). A call to _VersionNumsCompare is made on these and exits if they don't match.

      I understand that some of these are not major finds, but I'm a stickler for exactness :)

       
    • Albert hamik
      Albert hamik
      2009-07-05

      Well, I understand that the dark engine was based primarily around the quake engine. Just how much is the dark engine LIKE Quake? I mean, they obviously are similar in that they use a skybox, and that objects and characters are limited to what is made within the playing field. Also, just how much time did it take to discover the file system for dark? Am I making an understandable statement that appropriately describes the system which makes up dark.

       
      • Filip Volejnik
        Filip Volejnik
        2009-07-06

        Hmm, Dark Engine was written from the scratch and is not based on quake at all - from where did you get that impression? The mission file format was already understood when this project started - it was originally decomposed by Tels who worked on the cowed/cowgen (and spent tremendous amount of time on this task).

         
        • Albert hamik
          Albert hamik
          2009-07-07

          Well, not exactly a copy, but rather, inspired by the quake engine. Did you think I was trying to break new ground? No, I'm simply making an observation!

          However, it is my question on how far exactly we've gotten into the development of OPDE, and whether or not its development has at all given us light on what type of engine the original dark engine was.

           
          • Filip Volejnik
            Filip Volejnik
            2009-07-07

            Oh, now I understand. We know quite a lot about the original renderer, and you're right to conclude there are some similarities between dark and quake - but I would not say they are exact copies or such. Both use lightmaps, both use a BSP tree. Quake has precomputed visibility sets, Dark computes pvs dynamically using portal traversal (screen space). Quake is exterior based, Dark is interior based. Dark introduced animated lights which don't exist in quake - switchable lightmaps that can be modulated and added to the final lightmap. Dark also uses surface cache (quake used one as well, if I rembember correctly) which helps with batching (it uses quite a lot of textures so caching surfaces into only a few distinct textures really helps to avoid render state change overhead).

            We have a working renderer already that works quite well. The original engine was more dynamic as it tried to render as minimal set as possible, which was important at that time. I tried to be more effective (read - low CPU load per frame) in this area, still this has no easy  solution. Maybe new generation of GPUs will help us in this area.

             
            • Albert hamik
              Albert hamik
              2009-07-14

              But obviously portability is a minor (though primary) concern. Of course most systems today are more than capable of handling the demands of the dark engine. Including my linux system (an old dimension b110, which has no new graphics card in it: just a intel gma chipset), which used to run thief 2 perfectly as a windows system, but linux does poorly with directx emulation, so you know...

              Also, about movie playback: I looked at the video sizes in my main thief 2 folder, and I freaked out at how large the videos were in comparison to the game itself. Now, this is a wee too ahead of the primary concerns, but what is the ideal codec and video format you plan on giving support for?

               
              • Filip Volejnik
                Filip Volejnik
                2009-07-20

                Those videos use INDEO5 codec (if I remeber well). I'd like to convert them to some free container/codec combination, theora or matroska based. Codec - not decided, we'll see. After all we could also just link to FFMPEG and use the functionality provided by that library...

                 
  • Jonny Morgue
    Jonny Morgue
    2009-11-04

    How is this coming along and is there anything I can do to help? I just learned of this project today after struggling to run TG/T2 and hitting dead ends.

    And I like the work this project has done so far, flying around the training grounds definitely looks great.

     
  • Filip Volejnik
    Filip Volejnik
    2009-12-09

    Hi Johnymorgue!

    Sorry for the delay. We welcome any help we can get. Currently we seek developers to help with coding. If you are interested you should start by downloading the source code from svn and trying if it builds and runs for you. There is somewhat lazy updated TODO file in the svn trunk that could work as a starting point of any coding effort, or you can write patryn or me and ask what needs to be done.

     
  • Dusan Kostic
    Dusan Kostic
    2010-04-19

    Hello,

    You mentioned that mission file format is known to you. Can you tell me where I can find any information about it. I am more interested in mis format for SS2 but I guess thief format would be good for start (at least for geometry).

    Also, it is good to see that someone has decided to do DarkEngine port (since original code is unavailable).
    Cheers :)

     
  • Filip Volejnik
    Filip Volejnik
    2010-04-19

    Hello to you too,

    yes, the .mis format is (only partially) known. In order for me to help you, you'd have to be more specific at what you seek - if it is worldrep (geometry of the level), then most up-to date info can probably be found in our SVN in directory src/services/worldrep, texture handling is implemented in src/services/material. You'll also need to handle the mis format itself (which is a file container of sorts) - that can be found in src/base/file.  There is no text documentation of the formats so far, as it only took time to maintain the wiki and there was a little interest in the resulting data - not worth the effort.

    You can also find much good info on the formats used on Telliamed's Thief Tech pages (http://www.thiefmissions.com/telliamed/) and on displacer's wiki space (http://systemshock2.wikispaces.com/).

     
  • Dusan Kostic
    Dusan Kostic
    2010-04-19

    Hello Volca,

    Thanks for quick reply. I am also thinking about recreating DarkEngine but for the purpose of SS2 using open source 3D engine (like Irrlicht). Although, I never played any Thief game, I understand that they share much similarities. For the time being, I will focus on getting information on .mis files, and playing with DromED/ShockED to see changes they made to .mis files. Geometry will do fine for the beginning.
    Thanks for pointers to your source and for links. I will see what I can dig up.

     
  • Filip Volejnik
    Filip Volejnik
    2010-04-19

    Hi,

    Hmm, are you sure you want to do this in parallel with the effort this project already undertook? It was a lot of work to even get to this state (to be able to display levels with models and lighting), in the past we also tried irrlicht and then migrated to ogre3d.

    We support all DarkEngine games including SS2, so I'd think it would be better to join our efforts instead of duplicating them - what do you think?

    Or maybe I misunderstood the task you want to achieve. But  - writing a clone of DarkEngine means following it's steps very closely, the data driven nature of the engine is not easily reproduced :)

     
  • Dusan Kostic
    Dusan Kostic
    2010-04-19

    Hi,
    We share common goals, and duplicating same results is a waste of time. After all, we all want to see DE as soon as possible.
    For now I am just getting information.

     
  • Filip Volejnik
    Filip Volejnik
    2010-04-21

    Hi,

    Sure, I understand, if you'll need any help understanding, let me know, I can help you.