External debugger

  • rkd77

    rkd77 - 2010-11-29

    The fuse debugger sucks a bit. What do you think about the following scenario?
    the fuse listens on a unix socket or a tcp socket for commands (commands the same or similar to that of the current debugger), runs code
    and answer to the socket in plain text. Answers are good defined, eg.
    PC 32768
    BC 127
    Answers always enclosed by BEGIN END. In the middle registers and memory which were changed.
    For every command defined the format of the answer.
    Also ERROR answer for bad commands.

    Then the debugger could be an external program, written in a higher level language, Python for example.
    Going further, such a debugger could be "joined" with the z88dk and used as a C debugger.

  • Philip Kendall

    Philip Kendall - 2010-11-29

    This is certainly something that I've had at the back of my mind, so if anyone wants to prototype it, feel free :-)

    Some quick comments:

    * Plain text on the wire is bad. Use structured data.
    * Read about gdb's machine interface first (and any other similar things you know about). Don't reinvent the wheel.

  • Philip Kendall

    Philip Kendall - 2010-11-29

    And the other very important thing I meant to add there:

    * Talk to other emulator authors and get consensus on a standard API for doing this.

  • rkd77

    rkd77 - 2010-12-03

    I think that simplest, fastest and noninvasive method would be using the shared memory.
    The only change required on the fuse side would be to store all essential information in one block of memory (one big struct),
    and keep it "stable", do not change the order of the fields too often.
    Also when selecting an other machine, ROM banks would be copied, but this rather rare and fast, anyway.

  • Stuart Brady

    Stuart Brady - 2011-03-20

    I've recently heard from somebody who's added support for remote debugging of the Z80 to GDB.  Currently, the only remote stubs for this are one for my Z80 fork of QEMU, and a native Z80 stub.  Once we get the patches for this, perhaps we could integrate support for this into Fuse?  We could possibly even add something like Z80 stabs support to Z88DK/SDCC for source-level debugging.

    I would definitely agree that we should choose something that is acceptable to authors of all emulators.  If we need, we can always have more than one protocol, though.  There's quite a lot of software out there that supports GDB, so it's probably worth supporting.

  • Stuart Brady

    Stuart Brady - 2015-04-16

    Thanks! Yeah, I saw your blog, and it’s quite awesome. It’s something that people should reasonably expect to work, too. I believe our serial emulation could still use some work. I’m not sure whether using a pts (pseudoterminal) is the thing to do, so that baud rates, etc. can be set from outside of Fuse (assuming these are passed on somehow).

    That aside, yet, it would be great to support a gdb-stub directly. It’s good to have a standard to work to, but we can’t be seen to be taking GPL v3 code. Perhaps an English language description of the gdb-stub protocol would get us started.

    How this would interact with Fuse’s built-in debugger and its UI would obviously be a concern, but I’m sure we can figure something out, and we have to start somewhere.

    There may be certain fiddly details (NMOS Z80 bugs, MEMPTR) that need careful thought, as I’m not sure if we would want to change the protocol later. Reversible debugging, too. Of course, I don’t know the answers or we’d support this already, so perhaps you can help?

    Thanks again!

  • Stuart Brady

    Stuart Brady - 2015-04-18

    Thanks again.

    z80gdbserver is under GPL3 (“or later” but not earlier), so I cannot really refer to it too closely, and we certainly cannot use it in Fuse. Fuse is “GPL2 (or later)”.

    Qemu is licensed as “GPL2 (or later)”, though, so perhaps Leonardo Etcheverry’s additions to my Qemu Z80 target which add GDB support might be a starting point? : https://github.com/legumbre/qemu-z80

    I’m really not sure how this would fit into Fuse’s main loop, though! Is this something you could look at? Keep in mind we would prefer to keep the codebase single-threaded if possible.

    Last edit: Stuart Brady 2015-04-18
    • Tomaz Stih

      Tomaz Stih - 2015-04-24

      Hi Stuart,

      there is no "HW" support for GDB in Qemu. Qemu simply starts Z80 debugging stub on the emulated machine which communicates via GDB over serial port ... exactly as I do it inside my project. I actually used code of gdb-z80 stub from Leonardo.

      But there is an interesting folder in fuse source code called fuse/debugger. It already contains debugger functionality. Here is how I imagine it:

      1) We need a TCP/IP server inside fuse listening at some port, say 9999.
      2) When you execute command "target remote :9999" on GDB, GDB will connect to port 9999 (to fuse) and send a package.
      3) At this point fuse should freeze the execution. It seems there is a function in fuse called debugger_check, and a variable called debugger_mode in fuse/debugger/breakpoint.c. So I assume by extending it one could stop fuse in exactly the same way that build in debugger does.
      4) Now one simply talks to GDB and translate its commands to commands of build in debugger that are already programmed and reside in fuse/debugger folder.
      5) How to take control from fuse at start of debugging? Well - exactly in the same way as built in debugger does it.
      a) Instead of drawing debugger window one could simply draw GDB ACTIVE window.
      b) instead of keyboard entry this window should talk on port 9999.
      c) instead of being activated from menu it should be activated by opening tcp/ip connection to 9999.
      d) when you close this window...you close the connection just as you close the debugger, and start listening for new connections again.

      One would need to implement 6 basic commands.


      Last edit: Tomaz Stih 2015-04-24
  • Tomaz Stih

    Tomaz Stih - 2015-04-24

    I analysed this a bit further. It seems each user interface exposes functions ui_init and ui_event. The latter is the event loop and for fuse-gtk looks like this:

    while(gtk_events_pending()) gtk_main_iteration();

    So this looks like a good place for to check if connection has been established. With a non-blocking accept call. If connection detected...one triggers another menu option which in turns open GDB window. Here's what built in debugger does first:
    - fuse_emulation_pause();
    well...this seems handy. :) From here on it is more or less emulating the build in debugger window and translating GDB protocol to fuse debugger calls.

  • Stuart Brady

    Stuart Brady - 2015-04-24

    I agree, we should reuse as much of the existing infrastructure as possible, but I wonder, would it be possible to use both debuggers in parallel? I.e. can we safely attach with GDB and then continue to use Fuse's built-in debugger?

    There may be advantages to allowing this, as Fuse's debugger supports functionality that would be hard to support in GDB, such as breakpoints on peripheral page/unpage events, breakpoints on port reads or port writes, breakpoints on tstate counts.

    The two aspects of this that we will need to consider are:

    • Is it acceptable for Fuse to trap when GDB is attached due to a condition only supported in the built-in debugger? If not, this would be a bit of a headache from a UI point of view. We could have the built-in debugger's UI available, but breakpoints specific to the internal debugger would have to be greyed out.

    • Is it acceptable for Fuse to resume execution (or single-step) through the built-in debugger when GDB is attached? If not, we can handle this simply by greying out the single-step and continue buttons and disabling the continue, finish, next and step commands in Fuse's internal debugger.

    • Is it acceptable for Fuse to be unresponsive to GDB whenever a modal dialogue is accessed (affecting all UIs, most likely) or when paused when accessing menus (Win32 and Widget but not GTK+)?

    Also, how would the socket handling interact with Fuse's main loop? We use sound generation for synchronisation when running or a delay loop if sound is disabled, but otherwise we block in the UI code when paused. We can enter the internal debugger through the UI code which we've blocked in, but to access the external debugger when paused we need the UI code to poll on the external debugger socket. For SDL at least, it seems it should be possible to push a user defined event from a thread dedicated to monitoring the external debugger socket. Normally, we'd prefer to avoid using threads, but provided we only do so in the external debugger handling, and we make that be optional, perhaps that would be okay?

  • Stuart Brady

    Stuart Brady - 2015-04-24

    BTW, I think it's fine for this to be specific to one UI to start with. We can disable menu entries, etc. whilst the debugger is attached if that works around problems of the emulation blocking. We should put this in an experimental GDB branch to start with. It'd be good to get some simple socket handling in there to start with, and we can deal with the corner cases later. Is this something you'd feel happy making a start on? We can start with the packet format used in the existing GDB stubs, and change it later if we need.

    Debugger events can have an extra flag added to them to signify whether they're internal or external debugger events, if that helps for now. If we don't inform GDB that the internal debugger has trapped, then we are guaranteed that we can allow resumption from the internal debugger.

  • Tomaz Stih

    Tomaz Stih - 2015-04-24

    Yes. That could work.

    How about starting by me developing a small, generic gdb-stub.

    I would use this gdb-stub as a library. I would initialize it by calling gdbstub_init() passing it pointers to functions: step(), continue(), set_breakpoint(), get_regs(), put_regs(), read_mem(), write_mem(), on_data_ready().

    Then, upon detecting network input, I would call stub's on_data_received() function to pass whatever was received via tcp/ip to the stub. It would decode packet and call above functions, returning me the control. Upon stub's call to on_data_ready() I would send data that was passed by gdb-stub to the target.

    By writing mocks for these functions I could develop the stub as a small, limited, and easy to test project. Something similar to libspectrum, with callbacks. My mock would have gdb-z80 on one side and my terminal test app on the other. This way I can display contents of all packets, analyse protocol and manually control and tailor responses to the gdb-z80.

    Once minimal protocol is developed and understood, we can simply integrate it by implementing the above functions and adding socket loop and handlers inside fuse doing minimal changes and keeping the thing isolated?

    Last edit: Tomaz Stih 2015-04-24
  • Tomaz Stih

    Tomaz Stih - 2015-04-24

    Need to do a little research regarding dilemmas that you presented above. Think we could wake up Philip for that? I mean - he knows it and we need to reverse engineer it. :)

    As for running both debuggers simultaneously I need to check if one can extend the protocol to include custom commands. This way one could at least theoretically have all functionality available via GDB. But it sure looks like we are going to need at least some sort of hybrid debugging behaviour.

    UPDATE: Indeed, GDB supports something called a matchpoint. This is a breakpoint on memory location or on hardware (ports,etc.). However it is poorly documented and I doubt that gdb-z80 supports any of it?

    What do you think about simply extending current debugger window with a "Toggle GDB mode" button? When switched ON GDB takes over. When switched OFF GDB releases control to internal debugger. Wouldn't this be easier? This way we would control everything from internal debugger dialog and not touch the rest of code? We also don't need to worry about listening for incoming connections, finding an appropriate place in event loops etc. It's all user controlled from the debugger window.

    Last edit: Tomaz Stih 2015-04-24
  • Stuart Brady

    Stuart Brady - 2015-04-26

    It sounds like you have a good plan to get something started. Hopefully we can wake Phil up: I will try! He has quite a few pressing demands for his time these days so it may not be easy, but we'll see!

    Matchpoints sound exciting! Hopefully you can dig into that and find out some more. How flexible are they, exactly? I suppose gdb-z80 is unlikely to support them but this is something you could add. I wouldn't get too distracted by the specifics, though, as my dilemma really is just "how do we cope we things that don't fit into the model of GDB" and I'm sure we can invent new things if it turns out GDB copes just fine with what we do. :-)

    You may find the GDB developers somewhat co-operative in helping out in this regard, if we're an early user of the features, although they may complain that they do not want to support out-of-tree patches. I think we should look towards getting remote GDB support actually integrated into the GDB source tree but this might mean getting a FSF copyright assignment from all involved. I don't think it's a terribly drawn out process, TBH.

    I'd be happy to back you up if you want to ask around in #gdb on freenode, if the offer helps at all.

    Toggles may just do what we want perfectly well... and if we disable the internal debugger if the external debugger socket is set up (through a parameter when starting Fuse -- I don't think this should be a config option!) that might neatly solve at least part the problem... but it might be really neat if you could chop and change between the two in some circumstances, especially if we add extra features to the built-in debugger in future (such as listing of assembly comments, and calculator stack disassembly).

  • Stuart Brady

    Stuart Brady - 2015-05-04

    The other option is we don't compile the internal debugger if we compile the GDB stub. I'd be okay with that to begin with. There would be an option to the configure script to build the GDB stub instead of the internal debugger.


Log in to post a comment.