#21 add scripting support to gpsim?


Hi, have you considered adding scripting support to gpsim?

I think a properly-integrated embedded scripting interpreter (i.e. something like Python, AngelScript, GameMonkey, Lua or Ruby) could help modularize the gpsim architecture, while ensuring that maintenance and new contributions can be easily accomlished.

For starters, one could attempt to provide an interface to allow developers to write their own custom modules using such a scripting language. So that new modules can be easily contributed simply by dropping a corresponding script into the proper directory.

Likewise, custom simulator & debugger commands could be implemented more easily if there was a way to register such commands from within a corresponding script.

Ultimately, it would be truly cool to have the processors/microcontrollers themselves being realized via a script that's simply loaded interactively by the simulator core.

Performance should be hardly critical, given that nowadays scripting languages have pretty good runtime performance, in addition you can always cache pre-compiled bytecode, to facilitate faster running.

Besides, using a scripting language should enable more potential contributors to get PRODUCTIVELY involved in the project, without having to tackle with the core code base, that is new modules and processors could be contributed by people who wouldn't necessarily need to be directly involved in the project.

Given that there are several embedded scripting languages that feature a C++ oriented syntax (i.e. CINT, AngelScript or GameMonkey), it should also be pretty straight-forward to port the existing code base to provide corresponding scripting support.

In fact, AngelScript in particular is especially fast and suited for this very task, as it allows you to directly inteface native C++ types within and from your scripts.

Also, using a scripting language should be much more portable than using dynamic libraries.


  • Nobody/Anonymous

    Logged In: NO

    You may want to check out the mailing list archives, specifically the posts titled "Memory Leaks & multiple processors" (http://www.sourceforge.net/mailarchive/forum.php?thread_id=31686220&forum_id=10321). Browsing these discussions will show you that the gpsim folks have indeed already considered providing a more "dynamic" approach to describing processors, so that the actual logics would no longer be "hard-coded" but rather really be loaded at runtime, which would also facilitate a more rapid overall development cycle of the emulations themselves.
    However, so far it's been mainly considered a matter of coming up with a corresponding syntax/grammar to textually describe the basics of each processor, so that new processors could be built at runtime by parsing such a description file and dynamically plugging together the various low-level components (i.e. register, ram, port etc) and parametrizing the stuff then.
    I am not sure which -if any- of the aforementioned scripting languages provide support for multiple inheritance, however due to the discussions on gpsim-devel, I would assume that IF we can find such a scripting language it would truly make a corresponding implementation feasible. Maybe we should search freshmeat.net for some scripting languages which provide support for multiple inheritance, so that a processor could basically derive from all essential components?

    So, while I do see the benefits of having a textual description format available for the processors supported by gpsim, I also do see the overall benefits and flexibility of simply using an embedded scripting language for this very same purpose - in particular the idea to enable end-users to easily create (and contribute back!) their own modules in a scripted fashion seems extremely interesting.
    Thus, I'm sure that an increased emphasis on flexibility and modularization in gpsim would ultimately result in more contributors getting engaged in the project, where we could for example maintain a repository of scripts for various purposes (processor emulations, module emulations etc)

  • Nobody/Anonymous

    Logged In: NO

    Scripting is actually a pretty interesting idea, not only to create scripted processors and plugins/modules, but also to provide uC-components as scripts (for example UARTS, RAM, PORTS, ADC, DAC etc...)
    You could even use scripts to make stimuli more powerful.
    Adding scripting support to gpsim would mean that all of these areas could simultaenously benefit from increased extensibility, so that individual hardcoded components could be replaced step by step.
    I agree that this is certainly a good idea as it would enable people to get mor easily involved in the project, enabling contributions without requiring people to recompile gpsim or even have a build environment set up properly at all.

  • Nobody/Anonymous

    Logged In: NO

    I read up on the mailing list postings, this is actually a pretty interesting discussion.

    However, to actually conceive a proper and at least somewhat flexible description format
    without using a scripting language, you would have to make sure to take at least the following
    distinct items into account:

    1) uC architecture (built-in peripherals)
    2) uC instruction set & opcode mapping
    3) uC interaction mapping: effects of instructions on architecture status

    From an engineering point of view, these three items would preferably be tackled
    separately (in fact, doing so would automatically resemble following the design MVC pattern).

    So, 1) would serve the purpose to describe the architecture (in hardware-terms) of each
    controller/prozessor to be supported by gpsim, while 2) would describe the supported
    instructions in ASCII and their relevant (hex) opcode equivalents-finally, 3) would
    describe how each individual instruction would affect the state of hardware (basically,
    describing various complex bit/state (machine) manipulations).

    In fact, if you check out most existing tools that are related to gpsim in one way or another
    (i.e. sdcc, gputils/gpasm, picutils etc.), you'll find that these tools could eventually also
    benefit from a corresponding GENERIC solution-that is, a flexible way to properly, portably
    and FULLY describe a) the target hardware, b) the target instruction set and c) interaction
    patterns between hardware & individual instructions of the target (essentially, effect descriptions).

    However, given that the latter would inevitably -at some point- also mean to properly emulate
    runtime behaviour of the target hardware (i.e. latency/delays) and due to the fact, that you
    would normally want to accumulate some sort of "library" containing low-level peripheral
    primitives that are required for emulating most targets (rams, registers, ports and other
    peripherals), you might very soon end up re-inventing the wheel: VHDL.

    VHDL has been conceived for exactly this very purpose: describing system architectures in software.
    So, if pursued, any effort to come up with a generic hardware-description language specific to
    gpsim, would basically mean to re-invent VHDL or a dialect thereof.

  • Nobody/Anonymous

    Logged In: NO

    So, personally, I doubt that it would be particularly wise to try to make up your own VHDL-inspired
    makeup language: unlike any self-cooked VHDL-dialect we may come up with, VHDL is standardized
    and as such it is 1) well-known, 2) well-documented, 3) well-supported and 4) very widely used.

    So, there's indeed an increasing amount of tools and libraries available which could already be
    used to deal with VHDL files.

    Thus, rather than re-inventing the wheel, I would recommend to consider turning gpsim into a more
    generic emulator framework that may directly use VHDL files (i.e. downloaded from opencores.org)
    to emulate arbitrary micro-controllers. From a user's POV, this could for example work like this:

    1) download/create VHDL description for the target micro-controller
    2) create hex/cod image of your program for the corresponding target
    3) start up gpsim and tell it to emulate the controller using the VHDL file
    4) load hex/cod image of assembly code into emulator and run code in emulator

    That is, by recognizing target-describing as a task that needs to be split into at least the three
    distinct steps (as mentioned initially), you'll automatically end up with a re-usable solution that
    may be used by other tools like gpasm or sdcc, which would also require some/most of this info.

    Of course, a compiler (an optimizing one, that is) would also be interested in having some additional
    meta information available, in order to be able to estimate & assess the effiency/cost of certain

  • Nobody/Anonymous

    Logged In: NO

    scripting & gpsim has been previously attempted: search the mailing list archives for "SWIG"

  • Scott Dattalo

    Scott Dattalo - 2007-02-27

    Logged In: YES
    Originator: NO

    Could you guys log in to SF before posting. All one can see is debate between a bunch of nobodies!

    Anyway, I have no short term plans to add "scripting" to gpsim. By this, I mean that gpsim is not going to implement or emulate some new programming grammer. gpsim does support a "configuration script", which is simply a way of collecting command line commands into a file.

    I've wrestled with the idea of scripting several times now. I've implemented a expression syntax, a socket interface, and a variety of other script like mechanisms. However, when it comes down to actually using the simulator (as opposed to academically debating what it could do), I found that the most powerful scripting language is C++. gpsim already "supports" this via the module interface and gpsim API.

    Now one could argue that "scripting" as I've described it is just another way of customizing gpsim's source code for some particular application. Perhaps. But as the author of most of the API I understand it well enough to create plugins that others would rather implement by some external scripting mechanism. From this I can conclude at least two things: 1) the gpsim API for creating C++ scripts is too complicated 2) I don't know external scripting languages.

    Until someone can define a comprehensive scripting system that can provide all of the features provided by plugin modules, I suspect gpsim will not be supporting "scripting" as defined in this thread.


  • Nobody/Anonymous

    Logged In: NO

    Actually, I'd say that the most essential advantage for any sort of embedded scripting system lies actually in the fact that logics (code) can be added/modified without requiring any recompilation/rebuilding of the underlying core project itself.
    Which means that potential contributors don't have to deal with the guts of the application (once the scripting API is stable enough, that is) so that new functionality can be contributed without causing possibly immature code to affect the core engine.
    Which also means that new (scripted) modules wouldn't necessarily have to be carefully reviewed before being included, as the scripting interpreter itself would usually provide a "sandbox'ed" runtime environment.

    After all, the true power lies in the fact that essential stuff (such as processor definitions, hardware peripherals, modules and custom commands) could be added to gpsim by simply loading a corresponding script file.

    Mike (sorry, no sourceforge account yet)

  • Scott Dattalo

    Scott Dattalo - 2007-02-28

    Logged In: YES
    Originator: NO


    Thanks for providing a name. I assume you're the same "nobody" responsible for the large number of feature requests? Normally I let these feature requests (and bug reports) echo on the devel mailing list. So if someone's seeing this for the first time on the mailing list then it's because I've kept the last 30 reports from coming through.

    Anyway, the basic issue here is scripting and how best to support it. This is not a trivial issue to address. The proper approach in my opinion is to have a well defined, stable API to which any third party application can bind. Once this is done, then "scripting" becomes much easier. I strongly oppose to embedding some custom scripting language into the simulator. There are two strong reasons against: 1) it's custom 2) it's equivalent to defining the API. The "it's custom" objection is more than subjective; a custom language will be different. It will have to be learned. It has to be maintained. It has to be debuggable. It has to be implemented. I could go on. But in the long run, when writing a custom language your defining "an" API. Your custom language has to call into the simulation engine to do stuff. Why not just make that interface between your custom scripting language and the engine a public one that any script may call?

    Now that that's out of the way, I realize this is not even the issue. The real issue is how do you implement a scripting environment? And I think by this we mean a programming environment in which a user can customize things. For example, one may wish to perform unit tests on algorithms. This may involve placing the processor into a particular state, run for a while, and compare the new state against an expected state. The possibilities are endless.

    At one point I created a socket interface that would allow one to do this. The socket interface could interact with the command line and at the same time interface with a subset of gpsim's API. However, after spending a considerable amount of time on this approach I realized that it wasn't as efficient as it could be. A far superior approach is to embed a "script" into a gpsim loadable module. Once control is given to the module, it can take over gpsim and do what ever you want.

    Here's a real life scenario. In my day job I've implemented processors for our proprietary micro controllers. There's one application that communicates via a custom bit banged protocol. I've implemented a module to simulate the device on the other side of the communication link. I've created several different scripts for testing the firmware running in the proprietary processor. This code resides in a gpsim module. When this script gets control, it spawns a new pthread for running the script. gpsim continues to run in the other (main) thread. A mutex synchronizes communication between the simulator and the script. The simulation engine switches to the script thread on certain conditions that I define.

    In one test, the script will send a command down to the simulated processor. After the command is sent, the script will verify that the processor got the command. It does this by looking directly into the processor's memory (via gpsim's API). This is a simple example, but I think it's clear.

    Now, there's no reason why this scripting model can not be exposed as an "API" to which any third party scripting tool can interface. The code all exists, but there's absolutely no documentation on how to do this. As a start, I could create a few really simple examples.


  • Nobody/Anonymous

    Logged In: NO

    Hi !

    While I am certainly not responsible for anything like "30 feature requests", I did
    indeed add some comments to a few requests here, that seemed particularly interesting.
    However, those were only a handful of comments and definitely not 30: there must be some
    other lazy folks around here who didn't care to add a signature ;-)

    Well, sorry if that caused any trouble or confusion, didn't consider the fact that this stuff
    might be redirected to the mailing list. I do understand that this can be pretty annoying.

    Regarding scripting, I am not sure who -if anybody at all- mentioned coming up with a custom
    solution to scripting, specifically for use with gpsim. Personally, I would also certainly NOT
    favor any such approach (if that means, creating your own language!) - for various reasons,
    for example:

    - scripting languages can be a complex domain in and of themselves
    - adding yet another hardcoded component to gpsim would certainly increase maintenance requirements
    - there is already a large number of embedded scripting language interpreters FREELY available

    That is, the "interesting stuff" for most gpsim users (and probably developers, too?) is probably
    not reinventing things such as scripting languages from scratch, but rather concentrating on the
    MUC-specific side of things, and PICs in particular.

    Thus, coming up with any such custom implementations would inevitably mean that having those
    components in gpsim, would drain important resources (manpower & spare time) from the project,
    even though there are indeed many existing solutions available, which depending on the overall
    requirements, would certainly be more viable in the long run.

    This applies in particular to embedded solutions such as the previously mentioned Python, CINT or
    AngelScript -interpreters, which are standalone projects with sufficient momentum and community
    support, which in turn ensures that these are INDEED being ACTIVELY developed and maintained,
    which IMHO is important to make sure that any concerns are promptly and swiftly being dealt with
    by a SEPARATE project, without requiring gpsim developers to tackle issues they may either not be
    familiar with or even interested in at all.

    I do fully agree that having a stable and proven API should make it much easier to eventually
    implement scripting support for gpsim.

    Regarding the idea to enable arbitrary scripts to interface with gpsim via a well defined API, yes
    generally this is pretty interesting, and indeed exactly this is being facilitated by using
    wrapper generators such as SWIG.

    However, I have in the past witnessed both approaches in other OSS projects:

    1) settling for one specific scripting solution and supporting ONLY scripts in that language
    2) settling merely for an actively maintained interface, which arbitrary scripting languages could use

    The regular complaint for case #1 was often that people preferred using their own
    scripting language of choice and didn't really want to learn YET ANOTHER dialect of
    some weird scripting language, often fearing the learning curve required.

    The regular complaint for case #2 was often that there was literally a "scripting hell" happening:
    a relatively huge number of users provided functionality via scripts, which more than often enough
    could only be exploited by using one specific scripting language. Such contributions regularly came
    with their own dedicated set of SWIG bindings for their paritcular language of choice, including
    modifications required in the original bindings in CVS, which not rarely affected/conflicted with
    other people's scripts.

    Also, this situation inevitably lead to frustration among users who preferred using some other
    scripting language, but who would have liked to employ functionality implemented in some different
    language, as there was simply NO way to re-use such code, without actually running the other
    interpreter in parallel and executing said script there.

    Besides, the project which followed the track outlined in case #2, eventually suffered
    "dependency hell" as well:

    To be able to actually run all useful scripts that users came up with, you had to build
    the actual application with support for ALL scripting languages enabled, which required them
    to be compiled/installed in the first place:

    ./configure --enable-perl --enable-python --enable-ruby --enable-javascript

    That's what may happen if you give too much power/choice to your users...

    And don't get me started about different versions of scripting interpreters :-)

    With regards to case #1 however, people were experiencing a rather stable and mature scripting
    experience in the long run, which even though different from what they're familiar with initially,
    turned out to be absolutely sufficient for their purposes.

    So, unlike the users of the other project, these users could assume that all scripts would
    automatically work for them, too.

    And even more important: contributors could RE-USE/MODIFY/*MAINTAIN* existing code easily, simply
    because ANY script-contributor was sufficiently familiar with that one single,simple scripting

    Sorry for this rather longish posting

    I will comment on some of your ideas in particular later on ;-)


  • Nobody/Anonymous

    Logged In: NO

    Hi, some more comments:

    I am not sure whether we actually agreed on terminology: when I read about a "custom" scripting
    language implementation I figured you were talking about literally implementing a NEW scripting
    language from scratch, while the only thing I personally considered at all, was adding "bindings"/
    "glue code" to gpsim to use existing scripting languages together with gpsim.

    So, I'm sorry for any misconceptions I may have caused by elaborating on my view ;-)

    I also think that having the current (even though rather undocumented) API in place is definitely a
    good thing and -at least from my POV- it is out of question that any scripting support would not
    obsolete/depreciate said API, but rather only augment it in some places where gpsim might obviously
    benefit from functionality/code getting added and used more dynamically, rather than requiring it
    to be compiled.

    In fact, the API can probably really be considered to be the foundation for any serious effort
    to add scripting support to gpsim, and as such it would surely also make for an excellent role model
    to illustrate what sort of HLL hooks may be required for a possible interpreter to be truly
    useful in the long run.

    As such, the API is likely to resemble all of the low-level key functionality directly, so that
    exposing any scripting functionality would mainly come down to providing the corresponding bindings.

    Besides, we need to realize that "scripting" isn't necessarily the ultimate response to all

    For example, at work we are using a hardware simulator on a PC that needs to be interfaced with real
    hardware (connected via LPT & COM ports in this case), which means that there needs to be a way to
    convert/map simulator-state to actual LPT/COM-port outputs on the PC, in order to control said
    hardware from within the running simulator session (e.g. think: 7 segment LED display).

    This is only one example where conventional scripting solutions would simply not be adequate or even
    sufficient in the first place (without corresponding native support), as this is exactly the sort of
    stuff that is best done in native code (often requiring pretty platform-specific code, too).

    That is, if one intended to add such facilities to gpsim, you'd probably also be well-advised to do
    this natively with C or C++ source code (otherwise, you would also need to have the corresponding
    bindings in the API available in the first place, as most multi-platform scripting languages, that I am familiar with, do not provide such support).

    Also, I've recently had the idea to directly integrate (link to) some gpasm functionality in gpsim,
    which would hopefully enable users to directly assemble instructions to opcodes at runtime IN gpsim,
    so that you can treat and use gpsim as a PIC simulator that literally features an in-memory
    debugger & dis/assembler (similar to other tools like gdb or softice).

    Actually, very similar in scope to how the opcodes are being shown disassembled (in fact, the same
    low-level hooks would be basically required to implement the "watch" command as suggested in
    request #1624146 by "wibix", as users would then provide a mnemnonic (watch ADD 0x39) which
    would internally have to be converted to its opcode representation (=assembling) to deal with it).

    This is another example where you wouldn't really benefit all that much from a _purely_ script-based
    extension mechanism, as you would want to interface existing code (gpasm) with the gpsim
    core to read/modify specific opcodes in gpsim and lookup/translate supported mnemonics to
    their binary equivalents in order to insert them into the opcode table in gpsim.

    (So, I do also consider the API, as well as a possible scripting system, a way to easily customize and
    extend gpsim functionality. Thus, the API and scripting system would hopefully "grow hand in hand"
    as new hooks & bindings are added)

    I actually do like the idea you outlined with regards to simply using conventional modules and
    equipping these with some form of -possibly even custom- embedded interpreter individually,
    to make interfacing more powerful.

    In fact, one of the earlier mentioned OSS projects considered using a very similar approach,
    however ultimately they didn't really pursue it because they feared this would facilitate possibly
    proprietary code getting dynamically linked into the GPL'e core, thereby circumenvting "the spirit"
    of the GPL. Given that gpsim already provides a binary plugin interface, this should however not
    be an issue in this case ;-)

    Of course, this approach would entail lots of freedom for potential contributors and would not
    limit them to using any specific scripting interpreter in particular, so from that point of view
    it's indeed interesting.

    On the other hand, as soon as several people actually start providing and using
    modules implemented in that fashion, you may indeed face situations where multiple different
    (or even IDENTICAL) interpreter instances are loaded and run simultaneously, instead of only
    having one single dedicated interpreter for arbitrary modules.

    So, the overhead, footprint and dependencies may become an issue eventually.
    Still, it's an interesting idea.

    Thus, maybe, the most viable answer to the scripting issue after all these postings and
    babbling would be a compromise along the lines of:

    scripting support YES - but: IMPLEMENTED AS a SEPARATE gpsim MODULE?

    So, that scripting would not actually become directly a part of gpsim, but rather implemented
    by using the existing API (which could be extended as required). Which would ensure that the
    only connection is realized by using the API, so that'd also benefit modularization and

    Thus, an implementation would only be required to expose corresponding command interfaces to
    the gpsim shell, to facilitate loading/running of external scripts (which may not come with the
    module itself), rather than restrict any modules to module-specific scripts.

    What do you think?

    Given the lack of docs WRT to module writing, it would surely be a good idea to provide a couple
    of intuitive/well-documented examples that employ the current API, this would certainly help to
    get a better understanding of how things are hanging together.


    BTW: thanks to all gpsim/gputils contributors - very impressive projects! :-)

  • Scott Dattalo

    Scott Dattalo - 2007-02-28

    Logged In: YES
    Originator: NO


    As far as the flurry of feature requests... I know you weren't responsible for all 30 or so. There is another anonymous contributor.

    On the issue of a) creating a well defined API vs b) creating thorough support for only one scripting language, I'd weigh in on the API first followed by the scripting language second. If you understand gpsim's architecture you'd realize that the most pragmatic approach as well as probably the most logical approach. gpsim consists of three dynamically loaded modules: simulator core, cli, gui. There is a clear distinction between these. There is an API for each. However, the API is 'fat'. In other words, all of gpsim's simulation core class header files are installed whenever gpsim is installed. Any application can link to simulator core and instantiate any of the objects. This is the approach KtechLab took for example. This API is fat in the sense that *everything* that can be done with the simulator engine is exposed. However, only a small subset of this API is really required for scripting. So, rather than exposing the entire API, I'd rather define a thin interface consisting of just a few well defined virtual classes.

    So to summarize: gpsim could benefit from an IDL-type design that exposes only the important portions of the simulator engine. Once this done, any scripting infrastructure can be built around it. However, for practical reaons only one such infrastructure should be created.


  • Nobody/Anonymous

    Logged In: NO

    Hi again,

    I understand that due to the verbosity of my last postings (read: BRAINSTORMINGS) you may not yet have had the time to actually catch up on them, so just to make sure you are getting me right:
    I was proposing to use [parts of] the existing API to create a native gpsim module that EXPOSES a scripting interpreter to the core by registering a handful of relevant
    commands in the core (i.e. "load_script", "unload_script", "run").

    So that scripts -in turn- may make use of exported functionality from the core.

    Specifically, I am currently primarily envisioning modifications to enable
    the following (via scripts):

    - manipulate the command table (i.e. to add new instructions, or modify existing ones)
    - manipulate a table that contains help strings for existing & new commands

    - manipulate the variable symbol table (add, edit & remove symbols)
    - manipulate the simulated PIC's RAM/regs (change bytes at specific addresses)
    - manipulate the simulated PIC's program memory (add, edit/remove opcodes)

    Basically, scripting support would become another separate module that would only make use of
    the API itself.

    However, I realize that I should probably get more familiar with the current code base before
    digging any further.


    PS: Even though probably not immediatley useful for a scripting implementation, the above may actually help you to assess what sort of hooks would be required on the API side of things to make some of this possible.

  • Nobody/Anonymous

    Logged In: NO

    google/wikipedia: SWIG

  • Nobody/Anonymous

    Logged In: NO

    I have a support request pending
    as soon as that's solved I can start working on a scripting module based on
    GPMI (currently supports 10 script languages)

  • Nobody/Anonymous

    Logged In: NO

    regarding the possible use of VHDL to describe PIC targets, one could tackle the underlying issue (that is, describing targets in a flexible fasion that doesn't require recompilation to modify/update targets or create new ones) by looking at the existing PIC implementations and how exactly they're made up, this would make it possible to determine the requirements in order to come up with an XML/RDF based target description dialect, which in turn could be processed by a corresponding high level target composer that could dynamically create targets at runtime.

    For starters, this could possibly be also implemented as a separate gpsim module, which would then slowly evolve as support for new features is added, in order to eventually facilitate hardware description of all major PICs and their various relevant peripherals.



Cancel  Add attachments

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

No, thanks