File transfer between client and server.

Developers
CJP
2002-10-14
2002-11-17
  • CJP

    CJP - 2002-10-14

    When playing in a (multiplayer) game session, all programs (that is: the server and all clients) have to agree about the track and the cars.

    When one client thinks it is driving on track A, another thinks it is driving on track B, and the server, which calculates the physics, thinks in terms of track C, very weird effects will result.

    So, all programs have to use the same track files. But, because users are able to make their own tracks, a system has to be developed to distribute these files automatically. The question is: how?

    C.J.P.

     
    • CJP

      CJP - 2002-10-14

      I think the track file has to be selected on the server side. So, the track file and all files needed for the track (like tile files and texture files), need to be available on the server side.

      When a client connects to the server, the client tells which car it wants. When the server starts the game, it tells to all clients how they can get the files they need. Then, both server and clients (down)load their files.

      In order to reduce download times on slow connections, a system has to be developed to detect when a file is already present on the downloading computer.

       
    • CJP

      CJP - 2002-10-14

      What kind of protocol can be used to transfer files for UltimateStunts? Let me mention the following:

      ---1: existing protocols ((s)ftp, (s)http, scp)---

      I don't want to include a (big) implementation of one of these protocols in the UltimateStunts binaries. So we could use external (already existing) programs for file transfer.

      Advantage:
      -it takes less development time.

      Disadvantage:
      -The server computer ALWAYS needs to run the server side program of the protocol used. This won't be a problem on professional gameservers, but think of a standalone win95 box...

      ----------------------------------------------------------
      ---2: A self-developed protocol---

      Advantages
      -The protocol can be fit exactly to our needs
      -We don't need external programs to be installed

      Disadvantages
      -Takes a lot of development time
      -The game protocol uses UDP, while for file transfer TCP is better.
      -We have to think about privacy-security, because the server is able to copy files over a network.

      ----------------------------------------------------------
      ---3: A shared directory---

      In some situations, a file transfer protocol is unneccesary complicated. Think of a standalone computer, where server and clients run on the same machine. Or think of a local (home) network, when directories are shared with SMB or NFS.

      In these situations it is better when the server just tells the clients in which directories they have to look.

      Of course, this can't be the only solution. For situations when there is no shared directory, there will always have to be a network-solution.

      C.J.P.

       
    • Nobody/Anonymous

      Hi.

      Well the thing that looks most interesting to me is a game independent network next to the game wich handles the file sharing.
      Thinking about file updates, I immediately thought of CVS. Although making every user of the game install and setup a CVS Tree seems a bit of an overkill. Nevermind the amount of users that will probably give-up half way.

      So, user friendly should probably be the key term in setting this up.

      I'd say create your own protocol, or find another opensource project with the same problem (one of the reasons to go opensource;-).

      Best regards,
        Bram

       
      • CJP

        CJP - 2002-10-16

        I just got another idea:

        We want the game to use UDP, while for file transfer, TCP is preferred. So we want to use both of them anyway. Besides that, UltimateStunts already has some communication which would run better on the (reliable) TCP.

        So I suggest using the following protocol-design:

        (S=server, C=client)

        1:
        S listens on TCP port
        2:
        C connects to TCP port
        3:
        S tells which track to use, which cars can be used, and how to get it (either via the TCP connection or in a local directory)
        4:
        C gets the track and car files (if it doesn't have them already). File transfer over the TCP connection goes  like in FTP.
        5:
        Other C's are doing the same
        6:
        S decides to start the game, and it sends new data, like the number of cars, and the cars used by different players. It also tells which UDP port will be used.
        7:
        S starts listening to the UDP port. It is ready for the main game communication.
        8:
        C reloads the cars used, or unloads the unused cars (implementation specific). It is ready for the main game communication.
        9:
        Main game communication:
        -S continuously sends UDP packages with position data
        -C sends input data over TCP (only when input data has changed).

        10:
        When all clients have either reached the finish, or have their cars laying upside-down somewhere on the track, or crashed their cars, then S decides to end the game.

        When S wants to end the game, it sends some impossible UDP data (like a velocity equal to the speed of light) repeatedly. It would also be possible to send an "end" command over the TCP line, but then the client would have to listen to the TCP line continuously. This could slow things down (but if it doesn't, then we can still do the TCP option).

        11:
        If we are using the UDP "end" method, the C confirm the receiving of the "end"-package over the TCP line.

        12:
        both C and S stop using the UDP communication.

        13:
        S saves replay data, and tells the C how to get it (either via the TCP connection or in a local directory)

        14:
        C get replay data using the same protocol as in 4.

        15:
        Both C and S disconnect the TCP connection, and all players enjoy the game results :-)

        regards,
        C.J.P.

         
        • CJP

          CJP - 2002-11-04

          Hi,

          I've added a document to the UltimateStunts documentation, describing my ideas for the final network protocol. It's most interesting feature is that it has a package-based dependency checking system.

          regards,
          C.J.P.

           
    • bones

      bones - 2002-11-12

      UDP:
      To be more flexible you should support packages having a type, an id and then the data. Having this you could send types like -start signal- , -car positions- , -end signal- , -chat text- ... that makes much more sense than sending "impossible udp data".
      There must also be some packages where the other side confirms that it has arrived. On UDP you never know if a package arrives. If the confirmation does not receive the sender, it must resend the package automaticly. So there should be a flag in the package if it needs to be confirmed if it has arrived.

      TCP:
      I would design the TCP file protocol independent of the running game. If a client first connects to the server it will notice that it does not know the map they are playing. So it will ask "get current map". Then it might notice other missing dependencie files, and can ask the server "give me file xyz because i do not have it" and so on. I think the client should descide which files it needs from the server, because once downloaded, there is no need to fetch them again (like in unreal tournament).
      Once the client got all files it is ready to wait for the start signal.
      An implementation of this is easiely done, because you just program a stupid file server, that perferable runs on a different thread and does not touch the other server jobs.

       
      • CJP

        CJP - 2002-11-12

        UDP:
        Having some bits describing the type sounds like a good idea. It makes the network protocol less messy than my "imposssible data" idea. I used to be afraid to have not enough space in the (small) UDP packages, but 2 or 3 type bits together with some other flags could fit in a single byte. It's easy to create space by replacing the 9x32 bit rotation matrices by e.g. 3x16 bit rotation vectors.

        The position data, sent every frame by the server, does not necessarily have to be received every time by every client. When a frame is not received, then the movements on the screen of the client will be less smooth, but as long as it's smooth enough, I don't care about it. It's more important that frames are received in the right order. This is easily accomplished by adding a counter to each package (and testing for it).

        Of course, the user input data has to be checked. We don't want to send a message "key-left is still down" every frame: we just send a message once when the situation changes, but we have to be sure that it arrives.

        Currently, the input data is implemented on UDP, but my current idea is to switch it to a TCP connection, so that we don't have to do any checking.

        TCP:
        I agree that the client should decide which files are downloaded. The client can check if it already has some of the files. The general principle is quite simple:

        1: the server tells (direct or indirect) which files are needed
        2: the client checks which files are missing in his file-tree
        3: The client commands the server to give those files
        4: The server gives those files.

        Of course, there are some complications. How does the client identify a file? Just on the filename? Filename + exact size? checksum? Location in a directory tree? Please read the packaging idea in the document "Networking, file sharing and packaging standards".

        Why do we need a separate thread for the "fileserver"? The game can only be started when all clients are ready, that is: they have (down)loaded all things they need. So as long as the fileserver is not ready, there are no other jobs.

        Well, I can imagine one reason: the server needs to act as a fileserver for several clients at the same time. This will be a complicated job, possibly simplified when every client gets it's own thread on the server-side. But I don't know how this can be combined with a single network-interface.

        BTW, I don't know much about multithreading. I'm sure you can tell me how things can be implemented.

        C.J.P.

         
    • CJP

      CJP - 2002-11-13

      Because of the long ping-times on the internet and the low bandwidth of some users, the current simulation system (that is: only simulating on the server-side) is being criticised by Martin.

      So I want to discuss some ideas on what kinds of simulation should be done on the client-side, and what kinds of data should be sent between client and server.

      The current system:
      -----------------------------

      client gets user input
      client sends user input data to server
      server gets user input from clients
      server calculates physics
      server returns position data every frame to every client
      client gets posistion data
      client displays the new frame

      Possible new system:
      ----------------------------------
      client gets user input
      client sends user input data to server, including the exact time, to correct for ping time
      client makes an approximation of position data, taking input data into account
      client displays positions

      server gets user input from clients
      server calculates physics
      server returns position synchronisation data to every client
      client gets posistion data
      client synchronises simulation

       
    • bones

      bones - 2002-11-14

      Yes, think the new client simulation design for networking as you wrote should work.

      I addition I want to make sure that the client should NOT calculate interferences beween other cars or moving objects. This only the server should do in my opinion. Also crashes sould be descided only by the server. These events do highly depend on the exact position of other objects (cars), the client probably does not know. Right?!

       
      • CJP

        CJP - 2002-11-15

        Did you see the new document I put on the website about the GUI and game sessions etc.?

        ----

        OK, let's have a new look at the situation, to make things clear.

        Subject: What kind of simulation is done on which side of the line?
        Goal: We want the best solution in any situation.

        I listed the following situations:
        1 Local game, everything on a single computer
        2 LAN game
        3 Internet game

        1:
        We have a high bandwidth and a low ping-time, but the processing power is limited. There only is one graphical client. We want the physics to be done only once.
        Possible configurations:
        ----------------
        Graphical client, doing no physics
        Server, doing all physics
        AI clients, doing no physics
        ----------------
        Graphical client, doing all physics and acting as a server
        AI clients, doing no physics
        ----------------
        The second one gives less local network overhead, but the client code will be larger.

        2:
        We have a high bandwidth and a low ping-time. There are several graphical clients. Because of the good network, there is no need to do physics more than once (but in order to improve gaming quality, clients might use an approximation model).
        Possible configurations:
        ------------------
        Several graphical clients, doing no physics
        Server, doing all physics
        AI clients, doing no physics
        -----------------
        Several graphical clients, doing no physics
        One graphcial client, doing all physics, acting as a server
        AI clients, doing no physics
        -----------------
        The first one gives the possibility to run the server on a separate computer, where no graphical client takes processing power.

        3:
        We have a low bandwidth and low ping-times. There are several graphical clients. This is the most difficult situation, because of the bad network performance. However, we are playing on several computers, so it isn't that bad if physics are calculated more than once.
        Possible configurations:
        ------------------
        Several graphical clients, doing a physics approximation, or even a full physics simulation
        [A graphical client on the same computer as the server, not doing any physics because of high local bandwidth]
        A server, doing the main physics simulation
        AI clients, doing either a physics approximation, or a full physics simulation, or nothing at all (if they run on the same computer as the server)
        ------------------
        The same, but now with one of the graphical clients acting as a server.
        ------------------

        So, what factor should decide what kind of simulation is done on the client-side: it only depends on the network-connection between that particular client and the server. On high-end connections, no simulation, or perhaps a simple approximation, should be done. On low-end connections (e.g. a 700ms ping-time satellite), a full simulation on client-side might be needed.

        I think that the second-order approximation model I described is so simple, that the "no simulation"-option is not needed. Both other possibilities should be supported by the clients.

        BTW, I think that the requirements for the server program are so complicated, that it makes no sense to fully integrate server-functionality into the graphical client. Integrating the sound-, graphics-, input- and client-functionality of the client with the fileserver-, distribution- and server-functionality of the server seems to me like integrating a complete office-package and an operating-system into a single executable. And I still want to have a non-graphical server in the UltimateStunts-project, so why not simply use that one?

        My last remark:
        When playing over the (slow) internet, the server is the "master" simulation. In a conflict, the server is always right. The only reason that clients simulate is that the server cannot send a frame every 1/50 sec.. The client-side simulation is synchronised by frames sent by the server.

         
    • bones

      bones - 2002-11-14

      File transfer must be in own thread and totaly independent of the current game.
      Just imagine: Some players are racing one a sigle map again and again. Another player wants to join the game, but can not, because he does not have the map data. While the other players are still racing, the otherone can download the map and joint the game once the other players finished one round.

      We could identify the files by an md5-hash. This is a very usual way. See the "md5sum" program. But we could start just with filenames and replace it later (I guess there is no md5sum program for windows)

      The file-tranfer thing is very easy. If you have not started yet writing it, I guess I can do this very quickly.

       
      • CJP

        CJP - 2002-11-15

        You wrote:
        -------------
        Just imagine: Some players are racing one a sigle map again and again. Another player wants to join the game, but can not, because he does not have the map data. While the other players are still racing, the otherone can download the map and joint the game once the other players finished one round.
        ------------------
        Ehhh...?
        Someone wants to join a race while the others are still racing? I guess he'll have to wait until the race has finished, and then join the next race. But of course it would be good if he is able to download files while waiting for the race to be finished.

        You wrote:
        -----------------
        We could identify the files by an md5-hash. This is a very usual way. See the "md5sum" program. But we could start just with filenames and replace it later (I guess there is no md5sum program for windows)
        -------------------
        We could make some routine checking the identity of a file, and other parts of the program calling this routine, independent of how it is implemented. Then, we could decide what kind of routine we want.

        CJP

         
    • bones

      bones - 2002-11-17

      I posted a reply to this message to the "general issues", because it has nothing to do with the file transfer stuff.

       

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks