Menu
▾
▴
openocd-devel
|
From: Freddie C. <fre...@op...> - 2017-03-22 09:43:26
|
Hello! The idea I'm asking about is related to the Pretty Printing API in Python, which allows one to extend behaviour of GDB completely on "client side" - there's absolutely no need to modify any component of the toolchain to print your own types in any way you like. It would be really great if something like this existed for _supplying_ thread information to GDB. I'm not talking about the Python API to _query_ GDB about threads. ( https://sourceware.org/gdb/onlinedocs/gdb/Threads-In-Python.html ). Such feature may seem completely unimportant if you work with PC only, as there are just a few "old" operating systems and they are all handled perfectly fine by GDB and the toolchains, so there's no need to add/extend/modify anything here. But the situation looks completely different on deeply embedded platforms and I'm talking about microcontrollers here (like chips with ARM Cortex-M3 core). There are hundreds of RTOSes for that platforms and each is slightly different, so it's not possible to handle them all (or even some part of them) with some common tool in a generic way. Currently for RTOSes running on ARM the "solution" is provided by OpenOCD project. OpenOCD is a "GDB server" which communicates with the embedded target via JTAG interface and allows debugging such devices with GDB. Some time ago this project got support for a few popular RTOSes, so that when GDB queries OpenOCD for threads it supplies the info read from the target memory. Currently OpenOCD supports just a few (8) RTOSes. Although adding such support to OpenOCD is not "rocket science" it requires some effort and poses some challenges. 1. OpenOCD has slow release cycle - usually one can expect a release no more than once a year. With that in mind even if support for particular RTOS is added to OpenOCD, some platforms would have to wait years to actually get that. For example "stable" Debian has OpenOCD that is 3 years old, while "backports", "testing" and "unstable" have release from 2015 with some patches. 2. Once support is added to OpenOCD, the layout and naming of some RTOS structures has to be "frozen", as any change in that will break that support (the code to handle RTOS in OpenOCD is using internal knowledge about ABI of the RTOS). Obviously when these parameters change, they can (and should!) be updated in OpenOCD, but see item 1 above - some users would have to wait years to see that. Then it causes problems when you want to debug some new and old projects - they may need very specific versions of tools to work correctly. This may not be such a big deal for projects which actively participate with OpenOCD team, but some code for RTOS handling was written by people unrelated to the team/company of the RTOS, so it's not possible to expect them to keep everything up to date. Do note that such handling code can be made more generic by not hardcoding any offsets/addresses/etc., but it has to hardcode something anyway (at least the names of some objects, the assumptions of how a system arranges the threads, the stacking of registers)... 3. Above requirement is a big problem in case of developing your own RTOS, where you need good debugging tools to catch all the possible errors, but due to early stage of development the concept of freezing the layout/naming/configs is not very tempting. 4. The code to handle RTOS in OpenOCD is kind of a "middle man" - it has to query both GDB and target to get the whole picture. GDB has the info about symbols and their addresses, OpenOCD has access to the target. You need both to actually get something useful, as you have to read and interpret data from the addresses where some specific symbols were placed by linker. Then OpenOCD fills the info about existing threads and when GDB wants to know something about particular thread it asks OpenOCD to get that from the memory. For an example how such RTOS support code looks in OpenOCD, see here: http://repo.or.cz/openocd.git/blob/HEAD:/src/rtos/ChibiOS.c or more generally the files in this folder: http://repo.or.cz/openocd.git/tree/HEAD:/src/rtos Yesterday while writing my own pretty printer, I got an idea that this RTOS support could be done just like "pretty printers" are done. User would load his/her own Python script which would do the same thing OpenOCD is doing right now. As this would again be completely "client side", there would be no problem of incompatible versions of slow rate of releases. Such Python support script could easily be made part of the RTOS project or be automatically generated (to match current configuration). Would that be even possible to implement in GDB? Maybe something like that is possible even now, but not documented? Thanks for any info! Regards, FCh |
|
From: Liviu I. <il...@li...> - 2017-03-22 11:01:57
|
> On 22 Mar 2017, at 11:43, Freddie Chopin <fre...@op...> wrote: > > The idea I'm asking about is related to the Pretty Printing API in > Python, ... > Would that be even possible to implement in GDB? Maybe something like > that is possible even now, but not documented? even if this is possible, it does not cover all debugging tools (increasing in popularity right now, at least for FreeRTOS) that might need to introspect the RTOS, for example Eclipse plug-ins to display the status of semaphores, queues, memory allocators, etc. regards, Liviu |
|
From: Freddie C. <fre...@op...> - 2017-03-22 11:46:40
|
On Wed, 2017-03-22 at 13:01 +0200, Liviu Ionescu wrote: > even if this is possible, it does not cover all debugging tools > (increasing in popularity right now, at least for FreeRTOS) that > might need to introspect the RTOS, for example Eclipse plug-ins to > display the status of semaphores, queues, memory allocators, etc. The tools you mentioned clearly don't use OpenOCD's RTOS support or GDB's thread info for that anyway, as neither provides any info about things you mentioned (semaphores, queues, allocators, ...). So that hypothetical change that I suggested neither prevents nor enables them to do that. And if these tools do that now, they do that by directly accessing target memory via GDB, so no special support for anything like that is required. Regards, FCh |
|
From: Freddie C. <fre...@op...> - 2017-03-22 16:20:11
|
Hello Phil! On Wed, 2017-03-22 at 15:37 +0000, Phil Muldoon wrote: > Is it possible? Sure. But without looking at it in any real depth it > would be a major undertaking. GDB does not have the concept of > "Providers", it assembles state from many different sources: kernel, > glibc, libthread_db etc. Please note that for deeply embedded target these concepts don't apply - there's no standard "kernel" (usually there's also no separation between application and kernel), glibc is too huge and none of the "standard" facilities. From what I've seen > This state is internally structured and > relevant to GDB and is not really structured to receive arbitrary > input from an API at the moment. It could, but I think it would > require major surgery. Well, the info OpenOCD provides to GDB is structured quite nice. Basically OpenOCD just returns a list of tuples consisting of: thread name (string), thread extra info (string, in most implementation it's the info about state) and the thread ID (opaque value). Then it also informs GDB which of these is currently active. GDB may also request the list of thread registers, from what I understand by providing "thread ID" and expecting a string with registers as hex. All of that is done with GDB packets like "qThreadExtraInfo", "qfThreadInfo", "qC" and some more. In this case it would be enough to just get all that data with the Python script instead of querying GDB server (OpenOCD). > Currently the Scripting API is almost entirely focused with providing > information and not really designed to take information. You have > information providers APIs and data decorators APIs. I might got something wrong, but wouldn't this be an example of "information provider API"? Regards, FCh |
|
From: Daniel K. <op...@da...> - 2017-03-22 17:51:11
|
Hi Freddie, first of all, I feel the same way as you do. I'm one of those guys who have undergone the endeavour to add RTOS support to OpenOCD (in my case RIOT OS). All the problems (slow OpenOCD release cycle, hard-coded offsets/desynchronized information description, messy implementations in OpenOCD) you described are true, I only want to add another perspective. While trying to add support for RIOT, there was kind of a chicken-egg problem: without patches being merged in RIOT, you cannot test the OpenOCD implementation as well as the other way around when merging OpenOCD patches. Additionally, I've found it quite difficult to find people reviewing those patches on the OpenOCD side (it is not merged yet, I lost track of it). However, I totally understand that. Most of use are doing this in our free-time and so on, so I'd say this approach doesn't scale so well anyway. I have some experience with GDB Python scripting (there are also "new" OSes on x86), but I ran into the same problem as you did. To my knowledge (as of Dec 2016), there's no way to tell GDB about thread structures. That's why I ended up working around that by switching contexts myself. If it's of any help or interest to you, have a look at [1]. I'm not sure if it's possible for OpenOCD to add such support. It would make more sense to do that upstream in GDB IMHO, but it's the right direction. The upside of such an implemention is definetly that every project can manage these scripts in-tree and in sync with their development, with no dependency (as in providing patches) on OpenOCD anymore. Sadly, I cannot provide support on that matter at the moment because I'm lacking a use-case. Just wanted to add my 2 cents :) Cheers, Daniel [1] https://github.com/RWTH-OS/HermitCore/tree/devel/usr/gdb On 22.03.2017 10:43, Freddie Chopin wrote: > Hello! > > The idea I'm asking about is related to the Pretty Printing API in > Python, which allows one to extend behaviour of GDB completely on > "client side" - there's absolutely no need to modify any component of > the toolchain to print your own types in any way you like. > > It would be really great if something like this existed for _supplying_ > thread information to GDB. I'm not talking about the Python API to > _query_ GDB about threads. > ( https://sourceware.org/gdb/onlinedocs/gdb/Threads-In-Python.html ). > > Such feature may seem completely unimportant if you work with PC only, > as there are just a few "old" operating systems and they are all > handled perfectly fine by GDB and the toolchains, so there's no need to > add/extend/modify anything here. > > But the situation looks completely different on deeply embedded > platforms and I'm talking about microcontrollers here (like chips with > ARM Cortex-M3 core). There are hundreds of RTOSes for that platforms > and each is slightly different, so it's not possible to handle them all > (or even some part of them) with some common tool in a generic way. > > Currently for RTOSes running on ARM the "solution" is provided by > OpenOCD project. OpenOCD is a "GDB server" which communicates with the > embedded target via JTAG interface and allows debugging such devices > with GDB. Some time ago this project got support for a few popular > RTOSes, so that when GDB queries OpenOCD for threads it supplies the > info read from the target memory. Currently OpenOCD supports just a few > (8) RTOSes. > > Although adding such support to OpenOCD is not "rocket science" it > requires some effort and poses some challenges. > > 1. OpenOCD has slow release cycle - usually one can expect a release no > more than once a year. With that in mind even if support for particular > RTOS is added to OpenOCD, some platforms would have to wait years to > actually get that. For example "stable" Debian has OpenOCD that is 3 > years old, while "backports", "testing" and "unstable" have release > from 2015 with some patches. > > 2. Once support is added to OpenOCD, the layout and naming of some RTOS > structures has to be "frozen", as any change in that will break that > support (the code to handle RTOS in OpenOCD is using internal knowledge > about ABI of the RTOS). Obviously when these parameters change, they > can (and should!) be updated in OpenOCD, but see item 1 above - some > users would have to wait years to see that. Then it causes problems > when you want to debug some new and old projects - they may need very > specific versions of tools to work correctly. This may not be such a > big deal for projects which actively participate with OpenOCD team, but > some code for RTOS handling was written by people unrelated to the > team/company of the RTOS, so it's not possible to expect them to keep > everything up to date. Do note that such handling code can be made more > generic by not hardcoding any offsets/addresses/etc., but it has to > hardcode something anyway (at least the names of some objects, the > assumptions of how a system arranges the threads, the stacking of > registers)... > > 3. Above requirement is a big problem in case of developing your own > RTOS, where you need good debugging tools to catch all the possible > errors, but due to early stage of development the concept of freezing > the layout/naming/configs is not very tempting. > > 4. The code to handle RTOS in OpenOCD is kind of a "middle man" - it > has to query both GDB and target to get the whole picture. GDB has the > info about symbols and their addresses, OpenOCD has access to the > target. You need both to actually get something useful, as you have to > read and interpret data from the addresses where some specific symbols > were placed by linker. Then OpenOCD fills the info about existing > threads and when GDB wants to know something about particular thread it > asks OpenOCD to get that from the memory. > > For an example how such RTOS support code looks in OpenOCD, see here: > http://repo.or.cz/openocd.git/blob/HEAD:/src/rtos/ChibiOS.c > or more generally the files in this folder: > http://repo.or.cz/openocd.git/tree/HEAD:/src/rtos > > Yesterday while writing my own pretty printer, I got an idea that this > RTOS support could be done just like "pretty printers" are done. User > would load his/her own Python script which would do the same thing > OpenOCD is doing right now. As this would again be completely "client > side", there would be no problem of incompatible versions of slow rate > of releases. Such Python support script could easily be made part of > the RTOS project or be automatically generated (to match current > configuration). > > Would that be even possible to implement in GDB? Maybe something like > that is possible even now, but not documented? > > Thanks for any info! > > Regards, > FCh > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > OpenOCD-devel mailing list > Ope...@li... > https://lists.sourceforge.net/lists/listinfo/openocd-devel > |
|
From: Freddie C. <fre...@op...> - 2017-03-22 17:52:36
|
On Wed, 2017-03-22 at 18:23 +0100, Pierre-Marie de Rodat wrote: > I remember a talk at the last GNU Cauldron that talked about a > project > to describe way info and the like in a platfrom-independent way: > https://infinitynotes.org/wiki/Infinity > > It’s not a Python API and it’s still work in progress IIUC, but this > would not be something tied to GDB, so ideally other debugging tools > could use this info at some point. Would this fit your need? Looks like something targeting PC, so not really suited for a microcontroller. Regards, FCh |
|
From: Yao Qi <qiy...@gm...> - 2017-03-23 11:34:39
|
Freddie Chopin <fre...@op...> writes: >> Is it possible? Sure. But without looking at it in any real depth it >> would be a major undertaking. GDB does not have the concept of >> "Providers", it assembles state from many different sources: kernel, >> glibc, libthread_db etc. > > Please note that for deeply embedded target these concepts don't apply > - there's no standard "kernel" (usually there's also no separation > between application and kernel), glibc is too huge and none of the > "standard" facilities. From what I've seen > The differences between "deeply embedded target" and PC doesn't matter here. As Phil said, GDB has to assemble state from different sources, including OpenOCD and others. >> This state is internally structured and >> relevant to GDB and is not really structured to receive arbitrary >> input from an API at the moment. It could, but I think it would >> require major surgery. > > Well, the info OpenOCD provides to GDB is structured quite nice. > Basically OpenOCD just returns a list of tuples consisting of: thread > name (string), thread extra info (string, in most implementation it's > the info about state) and the thread ID (opaque value). Then it also > informs GDB which of these is currently active. GDB may also request > the list of thread registers, from what I understand by providing > "thread ID" and expecting a string with registers as hex. > > All of that is done with GDB packets like "qThreadExtraInfo", > "qfThreadInfo", "qC" and some more. > > In this case it would be enough to just get all that data with the > Python script instead of querying GDB server (OpenOCD). OpenOCD can provide thread information because it embedded the RTOS internal knowledge into it, as you mentioned in challenge #2 in your first mail. IMO, it is a pain. In order to get rid of this pain completely, python script is needed to *provide* thread information to GDB by parsing some RTOS's data structures, so that OpenOCD don't need these C code to get OS-specific thread information at all. Providing thread info isn't just printing thread name and state in "info threads". GDB internally needs thread information too. > >> Currently the Scripting API is almost entirely focused with providing >> information and not really designed to take information. You have >> information providers APIs and data decorators APIs. > > I might got something wrong, but wouldn't this be an example of > "information provider API"? I agree with Phil. Some python APIs are needed to provide thread information to GDB, but they are missing in GDB now. Note that we are working Linux kernel awareness debugging in GDB, in which we get Linux specific information, for example, kernel threads, via C code. We thought about doing it in python two years ago, but decide to implement it C first, and then, add needed python interfaces in GDB and move the code to python if possible. -- Yao (齐尧) |
|
From: Duane E. <op...@du...> - 2017-03-23 14:52:38
|
Hi - I’ve been following this thread
A while ago - I proposed a very different architecture that would solve a lot of this.
Fundamentally, there are two processes: (A) The GDB process and (B) the GDBSERVER process.
When debugging an application that runs under a true OS (such as linux) this works great.
But - when debugging anything bare metal, and that includes things like the Linux Kernel, and just about any RTOS in any micro-controller things are very different.
The problem is these things are two processes that cannot share knowledge of each other,
The current architecture is this: At the “target remote” - level - GDB sends various packets like Register Read/Write, Memory Read/Write packet. The problem is you need to write lots of GDB Protocol additions to add features - these extra things are not (and never) required when debugging an application under and Operating system. It is more then just “thread level information” - it extends to hardware registers, MMU state, Cache Information - things that an application debugger never sees and never uses.
In the current architecture it is the job of the GDB SERVER to translate these extra packets in to the actual operation. This makes sense when you are talking about a OS-APP-TARGET (i.e.: Windows, Linux, Mac, Cellphone apps) The OS solution is written once and is reused over and over again. And there are not that many of of these things. It works great for apps because there are very few of these things to write, and you are talking to an OS… not bare metal..
In the bare metal case, there are countless GDB SERVERS - and every one has to duplicate the process, the features and the packets. It is even worse when we talk about the RTOS support, every GDBSERVER solution has to write its own version of the RTOS support. Don’t forget about MMU level inspection, or Cache Debugging. These are things an APPLICATION level debugger never sees - a bare metal debugger sees these things all the time (of course depending on the chip, i.e.: ARMV8 kernel debug, vrs CortexM3 debug).
Bottom line: I believe the current gdb server solution is not well suited for some complex bare metal things - and that is what this thread is about.
What I proposed a while back was this:
What if we
1) introduced a “JTAG/SWD” protocol instead. For ARM (Cortex)- bare metal - all operations there are a series of DAP operations. i.e.: Dap Read/Dap Write.
2) For JTAG based chips (i.e.: earlier ARM 9 chips, Mips chips, etc) there are a number of JTAG level protocols (XILINX for example has a SOCKET based solution)
3) For SWD (cortex) based chips - all SWD operations can be ‘downgraded to JTAG’ - by a shim layer if needed.
4) Then - write a *NEW* target layer in GDB that uses this new protocol and effectively implement the memory read/write and register read/write operations in python, the bottom layer of that python implementation would send JTAG or SWD commands over the wire instead of “memory read” and “register read/write” packets.
The advantage I see is this:
GDB can be come very aware of the target MMU, for example - GDB can walk through all of the ARMv8 Hypervisor and
All of this is in one process - not spread across a protocol.
You have a simple common (and rich) script language (i.e.: Python) that lets one write some major scripts once - for all bare metal targets.
Flash programing operations (very common in micro-controllers) are written once - in Python - for all targets.
All of this is “write once” and it works - in one common way at the Python layer in GDB.
-Duane.
|
|
From: Freddie C. <fre...@op...> - 2017-03-26 14:11:54
|
Hi Duane! On Thu, 2017-03-23 at 07:33 -0700, Duane Ellis wrote: > Bottom line: I believe the current gdb server solution is not well > suited for some complex bare metal things - and that is what this > thread is about. > > What I proposed a while back was this: > [...] Wouldn't that actually result in OpenOCD's functionality being merged into GDB? It's an interesting idea, but I think it would be the hardest one to actually implement, given all the issues with JTAG interfaces and their configurations... Unless I misread your proposal? Maybe you suggest that GDB should not interact with OpenOCD as with a "GDB server" but communicate using some special interface designed just for that situation? Regards, FCh |
|
From: Freddie C. <fre...@op...> - 2017-03-23 16:41:07
|
On Thu, 2017-03-23 at 11:34 +0000, Yao Qi wrote:Providing > thread info isn't just printing thread name and state in "info > threads". GDB internally needs thread information too. But that's all GDB is getting from OpenOCD - thread name (string), extra info (string) and stack frame (registers). Unless I'm missing something, which is of course possible (; Regards, FCh |
|
From: Freddie C. <fre...@op...> - 2017-03-23 16:44:11
|
On Thu, 2017-03-23 at 10:50 +1300, Gareth McMullin wrote: > I am also of the opinion that this would be better achieved as a > Python extension to GDB. > A while ago I worked on some Python scripts that fake it in GDB, by > overriding the `thread` > and `info threads` CLI commands. > https://github.com/gsmcmullin/gdb_chibios/blob/master/chibios.py > (This is for ChibiOS 2 and is not currently maintained.) It > obviously > doesn't work > with frontends using MI. I had a crazy idea yesterday, that it should (maybe) be possible to implement a python "bridge" between GDB and OpenOCD. This script would open sockets for both ends and generally be transparent to the traffic. But it could intercept all the GDB packets about threads. This way this logic could be implemented completely in the script. This seems promising, but I'm not sure it would be possible to create such bridged connection between GDB and OpenOCD while intercepting some of the traffic... Regards, FCh |
|
From: Steven S. <sta...@sq...> - 2017-03-23 16:46:15
|
On Thu, Mar 23, 2017 at 11:40 AM, Freddie Chopin <fre...@op...> wrote: > On Thu, 2017-03-23 at 11:34 +0000, Yao Qi wrote:Providing > > thread info isn't just printing thread name and state in "info > > threads". GDB internally needs thread information too. > > But that's all GDB is getting from OpenOCD - thread name (string), > extra info (string) and stack frame (registers). > > Unless I'm missing something, which is of course possible (; > There's a bit more to it than that, but that's what users see most often. It's important to point out that GDB's remote target support isn't the best; it's not on par with a local target by any means. It wasn't until recently that thread names could be assigned/sign for remote targets (7.11.2 IIRC). To the OP, if you're interested in extending behavior, I'd suggest starting with a small PoC. I think you'll get a better idea of the issues at hand once you start exploring the GDB source. Steve |
|
From: Freddie C. <fre...@op...> - 2017-03-26 14:02:53
|
On Thu, 2017-03-23 at 11:46 -0500, Steven Stallion wrote: > There's a bit more to it than that, but that's what users see most > often. It's important to point out that GDB's remote target support > isn't the best; it's not on par with a local target by any means. It > wasn't until recently that thread names could be assigned/sign for > remote targets (7.11.2 IIRC). To the OP, if you're interested in > extending behavior, I'd suggest starting with a small PoC. I think > you'll get a better idea of the issues at hand once you start > exploring the GDB source. This would be a problem, as I'm not really a good desktop programmer (I'm working with microcontrollers only), I also know almost nothing about GDB interals. That's why I was hoping that something like that is actually possible now (directly in GDB or maybe with some evil trickery)... Regards, FCh |
|
From: Freddie C. <fre...@op...> - 2017-03-23 16:50:17
|
On Thu, 2017-03-23 at 10:23 +0100, Pierre-Marie de Rodat wrote: > What makes you think that? My understanding of the design is that > it's > centered on adding new metadata sections to ELF binaries and make > the > debugger use this metedata. This looks quite embedded-friendly to me. > Maybe I got the wrong impression, sorry. But the problem is that this project doesn't seem very active... Generally fragmenting the effort (yet another library with yet another standard and so on...) doesn't seem like a very good idea to me, that's why in my opinion a solution implemented completely in GDB would be better suited and would have a higher chance of "survival" and "adoption". But I might got the wrong impression again. However me and Liviu had a discussion about describing RTOS structure in a generic way and I'm still pretty certain that this is generally not possible in a generic and agnostic way. In the end it would become either extremely complex or you'd have to implement some kind of scripting/code to actually deal with that. Regards, FCh |
|
From: Liviu I. <il...@li...> - 2017-03-23 18:09:01
|
> On 23 Mar 2017, at 18:50, Freddie Chopin <fre...@op...> wrote: > > ... However me and Liviu had a discussion about describing RTOS structure > in a generic way and I'm still pretty certain that this is generally > not possible in a generic and agnostic way. In the end it would become > either extremely complex or you'd have to implement some kind of > scripting/code to actually deal with that. as I already mentioned, my next version of the DRTM library will use a compiled binary JSON, so everything that can be described in a JSON will be perfectly acceptable. are JSONs generic enough? I would say they are. will I make the DRTM library 'absolutely generic and agnostic' from the very beginning? definitely not realistic, I'll first define the data types and memory structures that I need for my µOS++. will this be expandable with more memory types? definitely yes! at the end, the number of ways a list of threads is kept may be large, at the limit each RTOS may invent a different scheme, but the number is still finite. ;-) my assumption is that relatively low and manageable, so it should be easier to add a new definition to an existing framework that is already fully functional, than to redo an implementation completely from scratch. and once you do it, the result should be directly available to all servers that use the DRTM library (OpenOCD, J-Link, QEMU being on my TODO list). regards, Liviu |
Thanks for helping keep SourceForge clean.
X