Re: [Amforth] Ref Card Generation quo vadis?

[Erich W. <ew....@na...>]

Dear AmForthers,

I tend to disagree on some of this ...

What you argue for is a "architecture and project-specific"
"reduced" refcard. At least this is, what I understand.


I have used the refcard extensively in the beginnings of my
AmForth journey. And I would argue any time, that the refcard
should please include /all/ /available/ things --- whether or
not they are in my build. This is akin to the "Complete
Instruction Set Summary" found in the document describing the
AVR assembly instructions.

Whether or not a given word is /in my build/ ... " words " to
rescue. Or just trying this /interactively/ on the serial
prompt. Or reading the .lst file from the build (for .asm
definitions).


The examples by Tristan
>> https://tjnw.co.uk//amforth-6.92/appl/arduino/uno.html
>> https://tjnw.co.uk//amforth-6.92/appl/arduino/custom.html
look nice and clean.

However, /I/ did not notice, the words are actually links ---
only after Marks comment. For me attributes like this are pretty
much /invisible/, although they are quite common (They can be
topped by buttons that appear only /after/ I have selected
something -- sigh).


IMHO a worthwhile extension of the refcard would include a
pointer to the source (in readable form), something like

> WORD  ( stack -- effect )
> - one-line-description
> - colon definition equivalent, if the word is in .asm
> - .../path/to/file.frt -- one entry if this is a /common/ .frt file
> - .../path/to/arch/file.asm -- several entries (one per line) if
>   this is a target-specific .asm file

This way it is clear, where to look, if the word is missing in
my build. It even indirectly tells you, on what targets this
WORD is available. And all this information should be visible on
a printed version of the refcard, too.


The pre-made .hex files for the arduino targets are a double
edged sword imho: On one hand they considerably lower the bar to
get to your first AmForth-prompt. On the other hand, they do not
teach users about fuses, about .lst and .map and include files,
etc. This is where the "can we please include words X and Y"-
wishlist comes from. But with pre-made files, users give up
control over their project --- to some extent. I would encourage
everyone get on top of the complete tool chain immediately after
the first pre-made file runs successfully.


But lets assume for this paragraph we want these "target and
build specific" refcards (which formats is another question
imho). How are we going about them? Is their production included
in the project Makefile? Are they produced by /every/ build? How
long does that take and what would be an acceptable time? And
how much tools do we add as a dependency?

The minimal toolchain is already substantial (from my Linux
centric point of view):

- the sources (via tarball, subversion/git, ...)
- the assembler (AvrAsm2 plus wine, unfortunately)
- an editor (/any!/)
- make to help with the build (could be done with shell commands
  in principle)
- a programmer (hardware)
- a program to talk to the programmer (avrdude or similar)
- a serial cable of some sort (hardware)
- a program to talk to the serial interface (minicom et al.,
  amforth-upload.py, amforth-shell.py)
- possibly some programm (evince, xpdf, ...) to read the
  documentation offline.

Even if we produce the refcard using an upgraded perl script,
imho this is going the wrong way.



One other thing I find interesting to note in this lengthy
thread: So far we have only looked at

"How to generate the documentation from the sources?"

But there is another option:

"How to generate the source from a (large) description of the
whole system?"

This is called "literate programming", as everyone and their dog
knows, of course :-)

I envision something like "every word has a description,
including any reasoning why it is implemented in such a way,
plus a description of the source code in some form. From this
description a .frt and/or .asm (variable asm formats)" is
generated, along with the Technical Handbook, the RefCard and
what not.

HOWEVER! I have tried to produce code like this. This is a not
so modest pain up the backside for a project, which is on the
move, which is not fully implemented. Literate Programming alone
is unable to track the evolution of the code, unless you rewrite
the description every time. I still think there is value to this
idea, but it is not the one answer to rule all questions.

I will not even try to convert AmForth to such a structure,
because it requires a lot of work PLUS an even larger tool
chain. :-)


Too Long; Didn't Read:

- I argue for a full refcard, not a reduced one
- I argue for a bit more information on the refcard
- I argue against pre-made .hex files
- I argue for less dependencies
- I do not argue for "produce the code from documentation" :-)

- my current priority is "how to make a release". And I'm not
  there yet.

- I'm willing to accept patches fixing the headers of .asm
  files.

- I'm willing to rewrite the refcard scripts, since I still use
  perl for my own stuff, too.


Thank you for your precious time!

Happy häcking!
Erich

PS: I'm very uneasy about the "answer above the thread text"
format. But I didn't see a convincing "merged form".

Mark Roth writes:

> Hello Tristan, Erich and AmForthers around the world.
>
> Tristan, I really like the direction you have gone in. Coupling the build
> process with the local refcard really is a solution that I can put my
> support behind. When push comes to shove what really matters to the end
> user? The words that have been installed into their system. From what I can
> see, it just works. AHHH and I just noticed that you linked the names to
> the source file. That is the solution to having the info you might need
> easy to find. Super.
>
> Since the build process would create individual refcards, it would just be
> a matter of making group of refcards based on the builds. But that begs
> the question of what other platforms is AmForth being used on? Is it enough
> to just make a project that loads every single common and avr8 file and use
> that for the website. I think maybe yes. Sort of the one refcard to rule
> them all.
>
> In any case, I really like what you've done so far Tristan. It is a big
> improvement upon the oundated version. Cleaning up the source to give a
> better card will be a minor issue in the long run since it will only have
> to be done once. When it is done locally during the build, the created html
> file should be dumped somewhere convenient, perhaps even in the build
> directory so one could just click on it to load it into a browser.
>
> All the best,
> Mark
>
> On Sat, Sep 19, 2020 at 6:30 PM Tristan Williams <ho...@tj...> wrote:
>
>> Hello Mark, Erich, AmForthers,
>>
>> I made some more modifications to the perl reference card script so
>> that it will write out an AVR8 build specific reference card in
>> html. Below are example outputs for the stock UNO build and
>> a custom build
>>
>> https://tjnw.co.uk//amforth-6.92/appl/arduino/uno.html
>> https://tjnw.co.uk//amforth-6.92/appl/arduino/custom.html
>>
>> The issues relating to the presence and correctness of the
>> documentation in the .asm files still remain.
>>
>> Best wishes,
>> Tristan
>>
>> On 09Sep20 08:38, Tristan Williams wrote:
>> > Hello Mark, Erich, AmForthers,
>> >
>> > Yes, I completely agree the format of my refcard excerpt has some
>> > issues. I just wanted to show that, with the hard work done by the
>> > perl script, all of the documentation data fields for AVR8 built-in
>> > words with compliant doc headers are readily available for
>> > output using the .lst file. That data could be formatted as desired
>> > and then written out in html, rst, LaTeX, Lout, etc.
>> >
>> > For the above to be useful, the .asm source files need to have
>> > compliant (and correct) doc headers. Lots of files are, but sufficient
>> > are not that some coordinated way of doing this is needed.
>> >
>> > How is this best done?
>> >
>> > Part of my motivation for pursuing this is that I think there is some
>> > value in having "fuller" pre-built AVR8 hex files in the distribution
>> > and giving them greater prominence in the documentation[1]. A build
>> > specific reference card would be helpful in such cases and it ought to
>> > be created by the same process that creates the hex files. Whilst I
>> > agree with [2] that it would be impractical to build hex files for an
>> > extensive combinations of clock frequency/mcu/baud rate, appl/arduino
>> > already exists and caters for arguably the most popular AVR8
>> > development boards plus a few other interesting ones - perhaps ~6
>> > pairs of hex files in total. Adding assembler words such as bm-set,
>> > bm-clear, bm-toggle, sleep, spirw, store-wdc etc. to the default build
>> > for these larger flash devices would just make the default build more
>> > useful.
>> >
>> > > This does drastically change what the current refcard is. That is to
>> say,
>> > > right now it was just a dump of the base system without regard to
>> usage,
>> > > more to availability.
>> >
>> > It does not have to be a replacement, just something that I think fits
>> > well with "fuller" pre-built AVR8 hex files and I see as achievable.
>> >
>> > > For the website there would have to be cards generated for different
>> > > architectures and perhaps even NWWR sizes. So, what to do?
>> >
>> > Limiting this to AVR8 and those boards that already are in
>> > appl/arduino (or perhaps should be) etc. makes it simpler.
>> >
>> > For existing non AVR8 pre-built hex/binary then having a matching
>> > refcard would make sense. However, I don't know enough about the details
>> > of the non AVR8 build process to say whether the same approach would
>> > work. Also risc-v/words/1-minus.s does not use the same doc headers as
>> > avr8/words/1minus.asm which suggests problems.
>> >
>> > > Maybe the easiest would be to have some generic setups in the appl
>> > > dir (most likely many already exist) and run the refcard script against
>> > > them while building for the website (or perhaps even locally, it won't
>> take
>> > > much time) then using that to create an array of refcards that could be
>> > > grouped together as web pages. The point is, that amforth.asm will
>> dictate
>> > > what a more or less default system will be and that can be used for the
>> > > site.
>> >
>> > Yes, I would agree - with board level customisation in uno.asm etc as
>> > it is currently. A refcard reflecting the built-in words of hex file
>> > created, be that built locally or pre-built on the website.
>> >
>> > With the current AmForth and AVR8 it is likely the built-in refcards for
>> > appl/arduino boards would be very similar - if not identical.
>> >
>> > So for an AVR8 builtin-ref card I think this is needed
>> >
>> > 1. Compliant doc headers for all the .asm files
>> > 2. Modify the perl script to write out refcard as desired in the desired
>> formats
>> > 3. Connect this to the build system
>> > 4. Connect the pre-built hex files and their refcards to the main
>> documentation
>> >
>> > All can be done for a local build setup, but most value would be with
>> > 1 and 4 done at the distribution/website level.
>> >
>> > Best wishes,
>> > Tristan
>> >
>> > [1] https://sourceforge.net/p/amforth/mailman/message/37054617/
>> > [2]
>> http://amforth.sourceforge.net/faq.html#there-are-no-hexfiles-in-the-distribution-archive
>> >
>> >
>> > On 08Sep20 13:14, Mark Roth wrote:
>> > > Hello Tristan, Erich and fellow lurking AmForthers (I really do like
>> this
>> > > intro Tristan) :)
>> > >
>> > > It really does seem that you may have caught the tiger by the tail
>> Tristan.
>> > > For better or for worse even!
>> > >
>> > > For the better (hey, you caught the tiger) :
>> > > I think your layout really goes a long way toward documenting the used
>> > > words. The last few days before I saw your mail I had been thinking of
>> this
>> > > very thing, to use the local build for the refcard. I hadn't, however,
>> seen
>> > > the obvious which was to use the list file. I used it when I was
>> trying to
>> > > find what was compiled into the base system and then finding useful
>> > > assembly files to build into my hex. It's funny how a new set of eyes
>> sees
>> > > the obvious. Well played indeed.
>> > > A few things of note from your examples. Since the items will most
>> likely
>> > > be listed by category the "category:" would be redundant. It also would
>> > > seem pretty trivial to remove the "C:" or "R:" from the stack effects
>> if
>> > > they were to be listed in the way you have. It makes it very clear
>> what is
>> > > going on as well as being consistent. Having the location of the asm
>> file
>> > > is super. Too many times I'm searching my local system for just that
>> very
>> > > thing. Of course it is listed in the lst file which makes perfect
>> sense to
>> > > show it. It could even open the door to having a location to forth
>> source
>> > > files there as well. It would just be a matter of putting the location
>> in
>> > > the comments in some way. I know you have shown interest in having that
>> > > easy to find as well Erich. Perhaps something like that could be a good
>> > > compromise?
>> > > I have a file in my appl directory that I use to upload everything I
>> add
>> > > after burning a new hex into the controller. It is just a list of
>> locations
>> > > for files that get added after the fact. It would seem that something
>> like
>> > > this could even be added into the documentation process. Say perhaps,
>> to
>> > > give an upload file name to the script before running so it could
>> parse out
>> > > the frt files that will be added after. Of course, that would mean they
>> > > would have to have some consistent system of comments as well... In any
>> > > case, it does open that door.
>> > >
>> > > And for the worse:
>> > > This does drastically change what the current refcard is. That is to
>> say,
>> > > right now it was just a dump of the base system without regard to
>> usage,
>> > > more to availability. For the website there would have to be cards
>> > > generated for different architectures and perhaps even NWWR sizes. So,
>> what
>> > > to do? Maybe the easiest would be to have some generic setups in the
>> appl
>> > > dir (most likely many already exist) and run the refcard script against
>> > > them while building for the website (or perhaps even locally, it won't
>> take
>> > > much time) then using that to create an array of refcards that could be
>> > > grouped together as web pages. The point is, that amforth.asm will
>> dictate
>> > > what a more or less default system will be and that can be used for the
>> > > site.
>> > >
>> > > Overall, I really like the direction of this proposal from Tristan. I
>> think
>> > > it does capture what the refcard should do for the end user.
>> > >
>> > > All the best,
>> > > Mark
>> > >
>> > > On Mon, Sep 7, 2020 at 3:00 PM Tristan Williams <ho...@tj...>
>> wrote:
>> > >
>> > > > Hello Mark, Erich, AmForthers,
>> > > >
>> > > > I agreed with Mark's comment below
>> > > >
>> > > > >> It seems that the intent of the refcard was to document the
>> things that
>> > > > >> are compiled into the system.
>> > > >
>> > > > and commented
>> > > >
>> > > > > For me, the scope of the/each refcard is defined by the
>> > > > > distribution build for each architecture (AVR8, msp430, etc.). If
>> the
>> > > > > refcard script were part of the hex build process then a custom
>> > > > > refcard could be a product of the build process also.
>> > > >
>> > > > For AVR8, the .lst file produced as part of the build process lists
>> > > > all the .asm files used in building the hex files. Modifying Mark's
>> > > > perl script from
>> > > >
>> > > > https://sourceforge.net/p/amforth/mailman/message/37060541/
>> > > >
>> > > > so that it uses only the included files listed in the .lst file, a
>> > > > list of words with the their documentation fields can output. These
>> > > > are specific to the individual build, rather than to the general
>> > > > assembly source tree.  Giving something like this (see end of
>> > > > message). As Mark pointed out
>> > > >
>> > > > >> There is still the issue of files that don't have the 3 lines at
>> the top
>> > > > >> of stack effects, category and description.
>> > > >
>> > > > Making these (assembly) files compliant would require some
>> coordinated
>> > > > effort - some of which has been already done I believe, but after
>> that
>> > > > a build specific ref card documenting built-in words could just
>> > > > be another automated part of the hex build process.
>> > > >
>> > > > Not perhaps the perfect ref card - whatever that is, but achievable
>> > > > and with a clearly defined scope. Certainly something I would make
>> use
>> > > > of.
>> > > >
>> > > > Best wishes,
>> > > > Tristan
>> > > >
>> > > > Edited example (text) output of the modified script for uno.lst
>> > > >
>> > > > ....
>> > > >
>> > > > Arithmetics
>> > > > -----------
>> > > >
>> > > > VOC      : 1-
>> > > > DSTACK   : ( n1 -- n2 )
>> > > > RSTACK   :
>> > > > CSTACK   :
>> > > > DESC     :  optimized decrement
>> > > > CATEGORY :  Arithmetics
>> > > > ASM_FILE : amforth-6.92/avr8/words/1minus.asm
>> > > >
>> > > > VOC      : 1+
>> > > > DSTACK   : ( n1|u1 -- n2|u2 )
>> > > > RSTACK   :
>> > > > CSTACK   :
>> > > > DESC     :  optimized increment
>> > > > CATEGORY :  Arithmetics
>> > > > ASM_FILE : amforth-6.92/avr8/words/1plus.asm
>> > > >
>> > > > VOC      : 2/
>> > > > DSTACK   : ( n1 -- n2 )
>> > > > RSTACK   :
>> > > > CSTACK   :
>> > > > DESC     :  arithmetic shift right
>> > > > CATEGORY :  Arithmetics
>> > > > ASM_FILE : amforth-6.92/avr8/words/2slash.asm
>> > > >
>> > > >
>> > > > Compiler
>> > > > --------
>> > > >
>> > > > VOC      : 2literal
>> > > > DSTACK   : ( -- x1 x2 )
>> > > > RSTACK   :
>> > > > CSTACK   : (C: x1 x2 -- )
>> > > > DESC     :  compile a cell pair literal in colon definitions
>> > > > CATEGORY :  Compiler
>> > > > ASM_FILE : amforth-6.92/common/words/2literal.asm
>> > > >
>> > > > VOC      : again
>> > > > DSTACK   : ( -- )
>> > > > RSTACK   :
>> > > > CSTACK   : (C: dest -- )
>> > > > DESC     :  compile a jump back to dest
>> > > > CATEGORY :  Compiler
>> > > > ASM_FILE : amforth-6.92/common/words/again.asm
>> > > >
>> > > > VOC      : ahead
>> > > > DSTACK   : ( f -- )
>> > > > RSTACK   :
>> > > > CSTACK   : (C: -- orig )
>> > > > DESC     :  do a unconditional branch
>> > > > CATEGORY :  Compiler
>> > > > ASM_FILE : amforth-6.92/common/words/ahead.asm
>> > > >
>> > > > ....
>> > > >
>> > > >
>> > > >
>> > > >


-- 
May the Forth be with you ...