Thread: [Amforth] amforth-shell.py bugfix

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Hello Matthias & All:

amforth-shell.py bugfix. skips already uploaded #include files.

http://pastebin.com/ePa76LmL

Regards, Enoch.

Hi Enoch,

> Hello Matthias & All:
> 
> amforth-shell.py bugfix. skips already uploaded #include files.

This would break my standard workflow: 
start shell
do
  flash controller
  upload files with #include (via cursor up-history)
  test
again

I would prefer something like
require <name> as specified
in http://www.forth200x.org/required.html
and keeping the include as it is now.

Another closly related feature would be

require 2drop

that checks, whether 2drop is already defined and includes
the file 2drop[.frt] only if not found in the dictionary. 

Just dreaming ;)

Matthias

Matthias Trute <mt...@we...> writes:
>> amforth-shell.py bugfix. skips already uploaded #include files.
>
> This would break my standard workflow: 
> start shell
> do
>   flash controller
>   upload files with #include (via cursor up-history)
>   test
> again

Hello Matthias,

I guess you better alter the above workflow a little if you wish to
prevent postpone (for example) being uploaded multiple times :-)

This #include once behavior is what Python does with import. I was
considering to remember the set of uploaded files between shell sessions
but rejected it for simplicity.

> I would prefer something like
> require <name> as specified
> in http://www.forth200x.org/required.html
> and keeping the include as it is now.

Hmm... I prefer keeping the shell's meta commands "meta"
(http://foldoc.org/meta). Once AmForth incorporates a notion of files
(anyone using AT90USB????) we can let the AVR take charge.

Regards, Enoch.

{-- Mon, 15 Apr 2013 03:03:55 -0400: Enoch <ix...@ho...> wrote: --}

  Enoch> Hello Matthias & All:
  Enoch> amforth-shell.py bugfix. skips already uploaded #include files.

  Enoch> http://pastebin.com/ePa76LmL

I can see how this would be useful in some circumstances.  However, in
interactive development I upload the same file many times after making
changes to it, often with the use of marker words temporarily embedded
in the file.  For that reason I would prefer that this not be included
in the main distribution unless there is a method to turn the checking
on/off, preferably with a default value of off.

This got me wondering if what would be more useful is a

    #require <file>

directive as an alternative to

    #include <file>

The way I would implement "#require <file>" is that it would attempt to
execute the word "<file-basename>-provided", where <file-basename> is
replaced with the name of the specified file with any path to reach the
file and any ".frt" extension stripped.  So, if the directive were:

   #require lib/marker.frt

The word that would be executed would be:

   marker-provided

If this word can be executed without error it indicates the required
file has already been uploaded and thus the upload can be skipped,
otherwise the file is uploaded just as it is in the #include directive.

I would not have the "#require" implementation do anything to create the
"<file-basename>-provided" word.  Instead, I'd let the library itself do
that so that the library can determine what the definition of that word
should be.  To guard against usage of "#require" with libraries that do
not create "<file-basename>-provided" I would have the "#require"
implementation verify it *can* successfully execute the that word on the
target after the upload and print a warning if it can not.

I suspect this would address the use case that the suggested change is
intended to support while allowing the user of the shell to clearly
express the kind of behavior they want without having to be aware of
configuration settings/flags.

On a related note, I've also considered having #include automatically
establish a markers named "marker-before-<file-basename>" before a
library is loaded and call this during it is reloaded as a convenience.
Up until now I avoided doing this because I didn't want the shell to
require any non-default words.  However, I think this could be nicely
combined with some of the infrastructure the "#require" functionality
would need in the shell, in the following way:

  1. Have the marker library create marker-provided
  2. In #include:
     a. Check for marker-provided.  If it is not present skip the
        following two steps
     b. Before uploading the file, attempt to execute
        "marker-before-<file-basename>", ignoring any error
     c. Create a new "marker-before-<file-basename>"
     d. Continue with the upload

The nice thing about this is it only takes effect if you load the marker
library.  So you can do things like load marker.frt during development
but then not load it when you are ready to create a production image in
which you don't want the markers, etc.

Unfortunately, I don't have time to work on any of this right now.
Hopefully these thoughts are at least useful to the discussion.

      --- Keith

Hi Keith,

As this is our first direct communication, thank you for contributing
the shell.

Keith Amidon <ca...@pi...> writes:

> {-- Mon, 15 Apr 2013 03:03:55 -0400: Enoch <ix...@ho...> wrote: --}
>
>   Enoch> Hello Matthias & All:
>   Enoch> amforth-shell.py bugfix. skips already uploaded #include files.
>
>   Enoch> http://pastebin.com/ePa76LmL
>
> I can see how this would be useful in some circumstances.  However, in
> interactive development I upload the same file many times after making
> changes to it, often with the use of marker words temporarily embedded
> in the file.  For that reason I would prefer that this not be included
> in the main distribution unless there is a method to turn the checking
> on/off, preferably with a default value of off.

Good point, I'll add #include-once [<yes-or-no>] with a "no"
default. You have provided a nice config nesting infrastructure.

I did read your other thoughts. It raises intersting questions regarding
the future role of the shell in AmForth development. Is it an essential
tool or "just" a convenience.

Regards, Enoch.

{-- Mon, 15 Apr 2013 12:52:45 -0400: Enoch <ix...@ho...> wrote: --}

  Enoch> As this is our first direct communication, thank you for contributing
  Enoch> the shell.

Thanks for your appreciation.  I've really enjoyed working on my amforth
projects and just wish I had more time for them. ;-)

  Enoch> I did read your other thoughts. It raises intersting questions regarding
  Enoch> the future role of the shell in AmForth development. Is it an essential
  Enoch> tool or "just" a convenience.

Personally, I think it is important that the shell is *always* just a
convenience.  My suggested implementation was intended to maintain that
property.  

My reasoning for the "#require" suggested implementation with the
special word to indicate something was uploaded was that the only way to
reliably know what has been uploaded to the microcontroller is to ask
it.  Anything else is vulnerably to shell restarts, microcontroller
re-flashes, etc.

My reasoning with the "#include" suggestions was motivated purely by my
own workflow.  I pretty much manually do all the things I suggested the
shell do and I find that process to be error-prone.  Trying to automate
it using the existing support in the shell is doable (e.g. by having it
ignore the error from the next command when calling the marker word) but
results in output errors when the markers aren't present.  I'd like to
avoid that error output since the primary expected users of the
libraries I've created are very inexperienced.

         --- Keith

I added an #include-once directive and let it default to "no" (current
behavior).  Also added was a program argument to change the default
behavior if one so desires [me]. Testing done as well.

http://pastebin.com/5Mfrvzdw

Regards, Enoch.

Keith Amidon <ca...@pi...> writes:

> {-- Mon, 15 Apr 2013 12:52:45 -0400: Enoch <ix...@ho...> wrote: --}
>
>   Enoch> As this is our first direct communication, thank you for contributing
>   Enoch> the shell.
>
> Thanks for your appreciation.  I've really enjoyed working on my amforth
> projects and just wish I had more time for them. ;-)
>
>   Enoch> I did read your other thoughts. It raises intersting questions regarding
>   Enoch> the future role of the shell in AmForth development. Is it an essential
>   Enoch> tool or "just" a convenience.
>
> Personally, I think it is important that the shell is *always* just a
> convenience.  My suggested implementation was intended to maintain that
> property.  
>
> My reasoning for the "#require" suggested implementation with the
> special word to indicate something was uploaded was that the only way to
> reliably know what has been uploaded to the microcontroller is to ask
> it.  Anything else is vulnerably to shell restarts, microcontroller
> re-flashes, etc.
>
> My reasoning with the "#include" suggestions was motivated purely by my
> own workflow.  I pretty much manually do all the things I suggested the
> shell do and I find that process to be error-prone.  Trying to automate
> it using the existing support in the shell is doable (e.g. by having it
> ignore the error from the next command when calling the marker word) but
> results in output errors when the markers aren't present.  I'd like to
> avoid that error output since the primary expected users of the
> libraries I've created are very inexperienced.
>
>          --- Keith
>
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter

Hi,

> I added an #include-once directive and let it default to "no" (current
> behavior).  Also added was a program argument to change the default
> behavior if one so desires [me]. Testing done as well.
> 
> http://pastebin.com/5Mfrvzdw

Applied. And Thanks to Keith for discussion. 

btw: I agree with Keith that the shell is a tool to make live
easier, but I won't make it a must-have tool. 

Matthias

Matthias Trute <mt...@we...> writes:

> Hi,
>
>> I added an #include-once directive and let it default to "no" (current
>> behavior).  Also added was a program argument to change the default
>> behavior if one so desires [me]. Testing done as well.
>> 
>> http://pastebin.com/5Mfrvzdw
>
> Applied. And Thanks to Keith for discussion. 
>
> btw: I agree with Keith that the shell is a tool to make live
> easier, but I won't make it a must-have tool. 

Let's make it an essential convenience then :-)

I have two more developments planned for the shell, one easy -- one
tough... The tough one I mentioned already, which is, use a library like
PyUSB to control the popular JTAG Mk II ICE the Forth way.

Let's mention the easy one, perhaps we shall have a volunteer coming
up. Let's say we need to send new Forth code to a client to introduce a
new feature or bug fix. The client, unfortunately, cannot be expected to
have a sophiticated uploader tool like our Python based shell. All
he/she has is something akin to minicom (Linux), tera-term (Windows),
etc.

A simple solution is to add a "distiller" mode to the shell where
alongside its normal work it would capture to a file all the characters
uploaded to the SUT (system under test), that is, past name substituion,
past all #include's, etc. The end result would be one compact file which
our client will be able to upload via his simple terminal program.

Anyone?

Regards, Enoch.

> To: amf...@li...
> From: ix...@ho...
> Date: Fri, 19 Apr 2013 18:51:09 -0400
> Subject: Re: [Amforth] amforth-shell.py bugfix

> I have two more developments planned for the shell, one easy -- one
> tough... The tough one I mentioned already, which is, use a library like
> PyUSB to control the popular JTAG Mk II ICE the Forth way.

And here is one idea just for fun. I thought I'll tackle this but I don't have time to learn python.

Take words and see them all. Produce dot file and make callgraph or wordgraph.
Anyway to see what gets used where and in which order.

#dot-tree for all and #dot-word for one word. 

It might be also useful for optimizing stuff.
: bar blob urf xyz ;
: baz asd = if blob then urf ;

The diagram will show that you need refactoring.

This is hand written to get a idea.

digraph foo{
  node [shape = record];
  bar [label="bar|<f0> blob|<f1> urf|<f2> xyz"];
  baz [label="baz|<f0>asd|<f1> if|<f2> blob|<f3> then|<f4> urf"];
  blob;
  urf;
  xyz;
  asd;
  if;
  blob;
  then;
  urf;
  bar:f0 -> blob;
  bar:f1 -> urf;
  bar:f2 -> xyz;
  baz:f0 -> asd;
  baz:f1 -> if;
  baz:f2 -> blob;
  baz:f3 -> then;
  baz:f4 -> urf;
}

Yes. I like to visualize my data.

Best regards,
Hannu Vuolasaho

Hi Hannu,

> Take words and see them all. Produce dot file and make callgraph or
> wordgraph. Anyway to see what gets used where and in which order.

Crazy. I think, a real forth interpreter with a modified code generator
(like the debugger in amforth) would be easier to use than a python
based heuristic parser that does not understand forth. Maybe gforth
is a better place to start with.

Matthias

I think the python script is becoming more and more complex.

Maybe it is the time to switch to an umbilical-like system.

For that, a new tool is needed and HolonU may be what your are looking for.
http://www.holonforth.com/new/holon.html

Probably you can attach a communication backend to the target system

See the tool history and use case using an old stlye Text User Inetrface in DOS
http://www.holonforth.com/using.htm

This is one of the developments that has impacted me most

Greetings
Rafael

________________________________
 De: Matthias Trute <mt...@we...>
Para: Everything around amforth <amf...@li...> 
Enviado: Sábado 20 de abril de 2013 9:30
Asunto: Re: [Amforth] amforth-shell.py bugfix

Hi Hannu,

> Take words and see them all. Produce dot file and make callgraph or
> wordgraph. Anyway to see what gets used where and in which order.

Crazy. I think, a real forth interpreter with a modified code generator
(like the debugger in amforth) would be easier to use than a python
based heuristic parser that does not understand forth. Maybe gforth
is a better place to start with.

Matthias

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Amforth-devel mailing list for http://amforth.sf.net/
Amf...@li...
https://lists.sourceforge.net/lists/listinfo/amforth-devel

Other  interesting papers by the same author
http://www.holonforth.com/holon/papers/ef89.htm

http://www.holonforth.com/holon/papers/ef94.htm

http://www.holonforth.com/holon/papers/ef95.htm

________________________________
 De: Rafael Gonzalez <ast...@ya...>
Para: Everything around amforth <amf...@li...> 
Enviado: Martes 23 de abril de 2013 12:01
Asunto: Re: [Amforth] amforth-shell.py bugfix

I think the python script is becoming more and more complex.

Maybe it is the time to switch to an umbilical-like system.

For that, a new tool is needed and HolonU may be what your are looking for.
http://www.holonforth.com/new/holon.html

Probably you can attach a communication backend to the target system

See the tool history and use case using an old stlye Text User Inetrface in DOS
http://www.holonforth.com/using.htm

This is one of the developments that has impacted me most

Greetings
Rafael

________________________________
 De: Matthias Trute <mt...@we...>
Para: Everything around amforth <amf...@li...> 
Enviado: Sábado 20 de abril de 2013 9:30
Asunto: Re: [Amforth] amforth-shell.py bugfix

Hi Hannu,

> Take words and see them all. Produce dot file and make callgraph or
> wordgraph. Anyway to see what gets used where and in which order.

Crazy. I think, a real forth interpreter with a modified code generator
(like the debugger in amforth) would be easier to use than a python
based heuristic parser that does not understand forth. Maybe gforth
is a better place to start with.

Matthias

------------------------------------------------------------------------------
Precog is a next-generation
 analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Amforth-devel mailing list for http://amforth.sf.net/
Amf...@li...
https://lists.sourceforge.net/lists/listinfo/amforth-devel

Hello Rafael,

Specialized IDEs are great but for me it's Emacs for everything. I am
using (a customized version of) gforth.el from the gforth project. As we
are all dreaming now... I hope to see Gforth specifics replaced by
AmForth specifics (sending code via serial I/O rather to an inferior
Gforth process). Time to wake up now :-)

Regards, Enoch.

Rafael Gonzalez <ast...@ya...> writes:

> Other  interesting papers by the same author
> http://www.holonforth.com/holon/papers/ef89.htm
>
> http://www.holonforth.com/holon/papers/ef94.htm
>
> http://www.holonforth.com/holon/papers/ef95.htm
>
>
>
> ________________________________
>  De: Rafael Gonzalez <ast...@ya...>
> Para: Everything around amforth <amf...@li...> 
> Enviado: Martes 23 de abril de 2013 12:01
> Asunto: Re: [Amforth] amforth-shell.py bugfix
>  
>
>
> I think the python script is becoming more and more complex.
>
> Maybe it is the time to switch to an umbilical-like system.
>
> For that, a new tool is needed and HolonU may be what your are looking for.
> http://www.holonforth.com/new/holon.html
>
>
> Probably you can attach a communication backend to the target system
>
> See the tool history and use case using an old stlye Text User Inetrface in DOS
> http://www.holonforth.com/using.htm
>
>
> This is one of the developments that has impacted me most
>
> Greetings
> Rafael
>
>
>
> ________________________________
>  De: Matthias Trute <mt...@we...>
> Para: Everything around amforth <amf...@li...> 
> Enviado: Sábado 20 de abril de 2013 9:30
> Asunto: Re: [Amforth] amforth-shell.py bugfix
>  
>
> Hi Hannu,
>
>> Take words and see them all. Produce dot file and make callgraph or
>> wordgraph. Anyway to see what gets used where and in which order.
>
> Crazy. I think, a real forth interpreter with a modified code generator
> (like the debugger in amforth) would be easier to use than a python
> based heuristic parser that does not understand forth. Maybe gforth
> is a better place to start with.
>
> Matthias
>
>
> ------------------------------------------------------------------------------
> Precog is a next-generation
>  analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> Amforth-devel mailing list for http://amforth.sf.net/
> Amf...@li...
> https://lists.sourceforge.net/lists/listinfo/amforth-devel
> ------------------------------------------------------------------------------
> Try New Relic Now & We'll Send You this Cool Shirt
> New Relic is the only SaaS-based application performance monitoring service 
> that delivers powerful full stack analytics. Optimize and monitor your
> browser, app, & servers with just a few lines of code. Try New Relic
> and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr

{-- Tue, 23 Apr 2013 12:12:50 -0400: Enoch <ix...@ho...> wrote: --}

  Enoch> Specialized IDEs are great but for me it's Emacs for
  Enoch> everything. 

It's emacs for everything for me as well.  Take a look at the mail user
agent this email comes from...  ;-)

  Enoch> I am using (a customized version of) gforth.el from the gforth
  Enoch> project. As we are all dreaming now... I hope to see Gforth
  Enoch> specifics replaced by AmForth specifics (sending code via
  Enoch> serial I/O rather to an inferior Gforth process). Time to wake
  Enoch> up now :-)

I did briefly look at modifying the forth-mode code from gforth to work
with amforth using amforth-shell.py to manage the interaction.  In
principle it shouldn't be terribly difficult to do but forth-mode seemed
fairly gforth specific and I didn't want to spend the time to try and
abstract it in addition to writing another backend.  

I haven't actually tried this but I think that a poor-man's version of
emacs integration can be achieved by using shell-command-on-region, by
default bound to "M-|", and the command "amforth-shell -p /dev/<tty>".
This should send the region to amforth-shell as standard input which
will be sent to the device.  The output will be in a buffer named
"*Shell Command Output*" which you can review to make sure the upload
was successful.

        --- Keith

Keith Amidon <ca...@pi...> writes:

> {-- Tue, 23 Apr 2013 12:12:50 -0400: Enoch <ix...@ho...> wrote: --}
>
>   Enoch> Specialized IDEs are great but for me it's Emacs for
>   Enoch> everything. 
>
> It's emacs for everything for me as well.  Take a look at the mail user
> agent this email comes from...  ;-)
>
>   Enoch> I am using (a customized version of) gforth.el from the gforth
>   Enoch> project. As we are all dreaming now... I hope to see Gforth
>   Enoch> specifics replaced by AmForth specifics (sending code via
>   Enoch> serial I/O rather to an inferior Gforth process). Time to wake
>   Enoch> up now :-)
>
> I did briefly look at modifying the forth-mode code from gforth to work
> with amforth using amforth-shell.py to manage the interaction.  In
> principle it shouldn't be terribly difficult to do but forth-mode seemed
> fairly gforth specific and I didn't want to spend the time to try and
> abstract it in addition to writing another backend.  
>
> I haven't actually tried this but I think that a poor-man's version of
> emacs integration can be achieved by using shell-command-on-region, by
> default bound to "M-|", and the command "amforth-shell -p /dev/<tty>".
> This should send the region to amforth-shell as standard input which
> will be sent to the device.  The output will be in a buffer named
> "*Shell Command Output*" which you can review to make sure the upload
> was successful.

Hi Keith,

Your amforth-shell.py is designed for importing and that's a good
starting point. I am sure that we can copy ideas from the various python
development approaches under emacs to modify gforth.el. All that we need
is ... time :-)

Did you make any significant changes to gforth.el? All I did thus far is
just adding some keywords and change indentation to follow the
openfirmware style.

Regards, Enoch.

{-- Thu, 25 Apr 2013 11:46:23 -0400: Enoch <ix...@ho...> wrote: --}

  Enoch> Your amforth-shell.py is designed for importing and that's a good
  Enoch> starting point. I am sure that we can copy ideas from the various python
  Enoch> development approaches under emacs to modify gforth.el. All that we need
  Enoch> is ... time :-)

Exactly.

  Enoch> Did you make any significant changes to gforth.el? All I did thus far is
  Enoch> just adding some keywords and change indentation to follow the
  Enoch> openfirmware style.

No, I haven't made any modifications yet.  I'd be interested in seeing
yours though.

  --- Keith

Thread: [Amforth] amforth-shell.py bugfix

amforth is an extendible interpreter on microcontrollers

amforth-devel