Announcing forthcon another forth console

[Jim Morris]

Hello,

I have produced another forth console similar to e4thcon but written in go.

It is here (with precompiled binaries for various platforms)
https://github.com/wolfmanjm/forth-console

It adds a bunch of convenience features over and above what e4thcon provides...

• persistent history
• fast download
• filename completion on \i (and \d)
• paste code from the clipboard

It is written specifically for quintus, but should work with any mecrisp forth.

The fast download (\d filename) is accomplished at the expense of a
large RAM buffer, and the forth code is included in the repo.
Basically it streams the file into the RAM buffer then evaluates that
buffer. I found this will download at the full baudrate (I use 3000000
baud!). The problem with waiting for each line is that the USB/CDC
turnaround time is very slow, and increasing the baudrate does not
actually help the download time.I found that this method downloads as
fast as (if not faster) than swdcom (which I love but doesn't work for
rp2350 or rp2050/pico).

\i filename will do the usual ping pong waiting for ok.

Feedback welcome as I have not tested on platforms other than linux on
x64 and arm64. Binaries for mac and windoze are included (but
untested) courtesy of go.

[Igor M]

It seems it works in win11, it has uploaded a large file fine.. I wonder what is the line delay hardcoded in there (if any)?
PS: with "words" I got the "line too long"

PS C:\Users\user> C:\Users\user\Desktop\forthcon.exe -b 115200 -d "COM3"
  ok.
  ok.
  ok.
» line too long, discarded
 2! 2@ cmove> cmove fill sfind align aligned words link@ here tib init forth >in base state /string type count .x .x2 bl cr space c! c@ emit key key? emit? pause nop um/mod * um* d2* d0= m+ s>d dabs dnegate d- d+ depth io@ io! nip over dup swap u< < = invert not or and xor - + ! 2/ 2* cells abs bounds umax umin max min 2over 2swap +! 2dup ?dup 2drop tuck -rot rot true false drop u> 0> 0< > 0<> <> cell+ 0= rdepth @ 1- negate 1+ arshift rshift lshift execute  ok.
  ok.

[Jim Morris]

Yea that is the only command that gives that error. There is a line length limit of 1024 characters, I modified my copy of list to break up the lines to that length. words4 works though.

Thanks for testing on windows!

There is no line delay for \d it just streams as fast as it can. \i also has no line delay it just waits for ok. or an error.

The reason I did the \d was that there IS a delay in the USB drivers on most operating systems (windows and linux) that make the ping pong very slow, at least a lot slower than it should be. (ping pong being the action of sending a line then waiting for a response then sending the next line). I ran into this issue a long time ago with gcode senders and never found a solution as it appears to be a USB thing. FWIW on Linux the /dev/ACMx driver does not have that issue so much, but the /dev/USBx driver does.

[Jim Morris]

I have updated the forth/dl.fs words to not allocate the large download buffer, it basically uses as much free memory as it can between the top of the dictionary and the end of RAM (basically in the middle of that), it uses 1/4 of the unused amount, on the RP2350 that is about 50K, on smaller systems it will be less. This should work fine so long as the files you download with \d are within that size. The only way I was able to get fast downloads was to just stream the files as fast as they would go with no delay and copy the data directly into the buffer, at least on an RP2350 this seems to be able to keep up. I have not tested it on slower MCUs yet. So if you get corruption on fast download just use \i instead.

[Igor M]

Hi Jim, one thing I always wanted to have is to assign a special char or char sequence or a special color when the mecrisp returns "?" (because of an error in words defs) as to find the returned "?" within a large upload is difficult..
I've been using TeraTerm in Win11 (my fpga wired over serial/ usb chip) and I need at least 15ms line delay there. Interestingly it uploaded with your terminal fine. Hopefully not matter of luck :) :)

[Jim Morris]

Yea color coding can be done, there is a mecrisp version that adds color to those prompts as well. In the case of \i it stops as soon as it gets an error anyway so it will be the last thing on the screen.
\d however is really only for files you already know have no errors. Adding color coding to forthcon will be something I'll look into, the prompt is already color coded. The line delay is only really needed if the returned ok. is not checked for every line. This is what e4thcom does as well as the xfr.py program that is somewhere on the mecrisp site. Also color coding the error is really only useful if it does not stop sending when an error is received.

[Jim Morris]

Ok I have fixed the "line too long" issue in forthcon, I just write out the long line. No need to fix the list word now :)

[Igor M]

I rebuilt my fpga and the 921600 upload works fine. The syncing with "ok" is/was the way to go, sure..

[Igor M]

The words shows more words now, but not all.. My words are 2456 bytes, and not all uploaded yet..

  ok.
words mandel mag next 2sq fsq nexty nextx ys! xs! zy zx y x falog fpow flog fln (log) fraction integer> >integer lbase epsilon fexp ^fraction ^integer (!) d>u d0< f** fatan2 facos fasin (fasin) fatan dom2|3 dom3 dom2 (fatan) (taylor2) 2degrees ftan fsincos fcos fsin >range -taylor +taylor (taylor) >taylor rad>deg deg>rad e pi F# (f#) _f# F.S fd abort" FVARIABLE FCONSTANT >FLOAT -trailing flfrac flexp flint fdigit? flgood fsign FSQRT d< FE. FS. (E.) R. F. (F.) fsplit F~ F/ F* FMAX FMIN FROUND FLOOR F<= F<> F>= F> F= F< F- F+ tneg FROT FNIP FPICK FOVER FSWAP FALIGNED FALIGN FABS F0> F0< F0= F0<> FSIGN? F! F@ FLOATS FLOAT+ S>F F>D D>F FNEGATE FDROP FDUP FDEPTH FCLEAR SET-PRECISION PRECISION F2/ F2* normalize fshift >exp1 ftemp' expx2 expx1 m1sto m1get fmove 'e2 'e1 'm3 'm2 'm1 longdiv *norm lstemp t+ t2/ t2* sign>exp exp>sign frshift ud2/ d2/ du< m*/ t/ t* 2r@ 2r> 2>r tnegate pick roll ferror &esign &sign &unsign mxdig bicl ftemp fstak fsp digits -bme. disasm-cont disasm-$ insight .s dump g/ ud/mod (ud/mod) divisor shift dividend g* g>s s>g 2constant 2variable g. g.n g# hold< du<= du>= du> du< dmax dmin d<= d>= d> d< 2rot 2nip 2arshift 2rshift 2lshift dshr d2/ d<> d= d0< 2xor 2and 2or save erase spiwe waitspi esc? nextirq ticks random randombit ms us delay-cycles cycles/ms cycles/us cycles leds unused cornerstone endcase endof of case s" within pad ." mod / /mod move u.r .r d.r rtype u. . d. ud. (d.) #> #s # .digit sign hold <# hld BUF BUF0 spaces */ */mod fm/mod sm/rem sgn constant variable m* >body buffer: create repeat while else <= >= u<= u>= ( [char] ['] load spi> >spi spix idle welcome xor! bic! bis! eint? dint eint quit evaluate refill accept number digit \ char ' postpone literal abort rdrop r@ r> >r hex binary decimal unloop j i +loop loop ?do leave do recurse does> until again begin then if ahead ; exit :noname : ] [ immediate foldable sliteral s, compile, c, , allot parse parse-name source 2! 2@ cmove> cmove fill sfind align aligned words lin»

[Jim Morris]

ok I'll take a look... can you post the list word you are using?

[Igor M]

It is now 2456 bytes (incl. spaces between words), I think 4kB buffer/line would be ok..
Hmm, that is for 16kB mecrisp in fpga, people with large 32bit mcus might get it even bigger.
So 64kB line would be perhaps ok.. :) :)

[Jim Morris]

There is no buffer limit now it just prints what it gets. However if the line is not terminated with a \n then it may not get flushed, I'm looking into it. There does not appear to be a stdout flush in go, I'll look harder.

[Igor M]

Another interesting observation - when printing out the mandelbrot demo (made of chars) it waits till the \n and then prints out the entire line. In TTerm I see the individual chars coming (there are large diffs in the speeds it prints out the individual chars because of math). So in the first moment I thought it got frozen.. :-)

[Jim Morris]

Yes it is line based, it reads a line at a time a line being terminated by \n, if it does not have a \n it gets buffered until it either gets a \n or it gets 1024 bytes.

[Jim Morris]

Ok I found the error, it will take a while to fix it... it only happens if there are very long lines that are not terminated by \n. I'll update github when I have fixed it.
Thanks for the help :)

[Jim Morris]

Ok I think I fixed it at the expense of adding a \n to a partial line that exceeds 1024 bytes. Now list seems to return all the words, albeit some are split by a newline. I think getting lines over 1024 is an edge case anyway and list is the only thing that seems to do it. Give it a try now.

[Igor M]

words - here with the long list I have now - it starts ok, then it misses X words periodically after 1025bytes, finishes ok..

CORRECTION: it does not miss but inserts \n - so my bad..

[Jim Morris]

Did you try the latest version?

[Igor M]

This is with my longest words list..
It periodically cuts making \n after aprox 1025 bytes (the yellow arrows show where ).
It does not miss words.
Your bin from 13:13..

[Jim Morris]

They do not appear to be missing, they are split with a newline as I said I had to insert a newline at 1024 bytes. Can I see what the list looks like with your other terminal? Thanks

[Igor M]

I corrected my findings above - it does not miss but inserts newline as you wrote.. The full list you may find above in the jpg picture

[Jim Morris]

I initially just sent the buffer with no \n, but for some reason it then does miss several hundred characters. I think it is a flushing or buffering issue, I am not sure, but there is no flush option on the write. Go says that output to stdoout is non-buffered, so it maybe the readline library. I'll let you know if I can fix that. for now inserting a \n is the easiest solution.

[Igor M]

Could it be because the serial_COM to usb dongle (ftdi in my case). Those work with such chunks/packets.