StringTables (chr indexed)

[ToniG]

I have modified my GCB preprocessor to enable input of Table string data directly in GCB source code.
The TableString sub converts Table data "string" lines to Table data byte lines.
The preprocesssor/compiler then process as a normal byte table.
I haven't noticed any unusual behavior prior to the convert (the preprocessor is not easy to follow in parts), the dev's would have more of a idea.

To get strings from the table I have some example GCB code to grab a block of ASCII bytes, the code is structured to be easy to read.
Attached ReadTableStrings1.gcb code(tested in sim) & files if anyone wants to give it a try.
If you want to test the string extract code with your current compiler version, just un-comment / comment the appropriate table.

The following Table examples produce the same same byte table in asm.

Table Str1
 "ABCDEFGHIJ0123456789" '1 String line
End Table

Table Str1
 "ABCDE"  'String1
 "FGHIJ"  'String2
 "01234"  'String3
 "56789"  'String4
End Table

Table Str1
 "ABCDE _
  FGHIJ _
  01234 _
  56789"
End Table

Table Str1
65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57
End Table
-----------------
           Asm byte table:
TABLESTR1
db  20,65,66,67,68,69,70,71,72,73,74,48,49,50,51,52,53,54,55,56,57


' Any ASCII characters between "  " will be converterd to byte.
"01234", "56789"          '<-  will produce a Table of 14 bytes.
"Switch 'ON' " ' Test"1"  '<-  will produce a Table of 22 bytes.

So as not to not add 'clutter' in PreProcessor the TableString code is a separate Sub.

In GCbasic.bas: Declare Sub TableString, In SUB PreProcessor: Dim TF, TF = "0"

   TableString (DataSource, TF)

TableString Sub (FB code)
' --------------------------------------------------------- 
      'Convert Table string data lines to Table byte data lines  2021-03-02
Sub TableString (DataSource As String, TF As String)    '( TF must persist!)
      Dim As String TempDS, TempTD, CSV
      Dim As Integer Lp1        
      TempDS = UCase(DataSource)

      'Replace tabs, trim & exit if line a comment
      DO WHILE INSTR(TempDS, Chr(9)) <> 0: ReplaceAll TempDS, Chr(9), " ": Loop
      TempDS = LTrim(TempDS): TempDS = RTrim(TempDS)
      If Left(TempDS, 1) = "'" Or Left(TempDS, 1) = ";" Or Left(TempDS, 1) = "//" Or Left(TempDS, 1) = "Rem " Then Exit Sub

      'Delete after string terminator
      Lp1 = InStrRev(TempDS, (Chr(34)))
      If Lp1 > 2 Then TempDS = Left(TempDS, Lp1)

      'Detect table 
      If Left(TempDS, 6) = "TABLE " Then TF = "1"
      If Left(TempDS, 9) = "END TABLE" THEN TF = "0"

      'Build Table data
      If TF = "1" Then
        If Left(TempDS, 1) = Chr(34) and Right(TempDS, 1) = Chr(34) And Len(TempDS) > 2 Then 
          TempTD = "": CSV = ""
          For Lp1 = 2 To Len(TempDS)-1  
            TempTD &= CSV + Str(Asc(Mid(TempDS, Lp1, 1)))   
            CSV = ", "
          Next
            DataSource = TempTD ' <Put back in DataSsource
        End If
      End If
End Sub
/*
'   TooDo: Add support for multiple strings per line 
           Table Str1
             "str1", "str2", "str3"
             "str4", "str5", "str6"
           End Table
*/
' ---------------------------------------------------------

[Anobium]

Nice job. A nice change.

Do you want to complete the multiple strings per line before we integrate into the main code base?

---

:-)

If you are up to it .. there are other changes....

[ToniG]

If you don't mind to wait a bit, I am not a fast coder(especially when I get stuck on a stupid things).
It would be good to have some tests (other than me), good to catch any bugs early.

I have started working on the multiple strings per line...

Do you mean 'other changes...' to PreProcessor ? as long as the scope is not to wide I could handle it.

[Anobium]

We can always be patience. No rush.

What baseline are you using for these changes? LIke.. what is the date of the compiler source? I want to ensure we support your changes.

And, the 'other changes'. For-Next loop needs an improvement. We need to add a step exit clause when 'step' is used. At the moment the For-Next -Step loop can cause confusion as it does not exit when the max value is exceeded.

[ToniG]

Yes, just what I need, review & comments...

If you can advise where to identify the Version I can verify.
I just downloaded the latest? from SF a few days ago "gcbasic-code-r1186-GCBASIC.zip"
There is a version in the .bas file
'Set version
Version = "0.98.<<>> 2021-02-28"

For loop Step
not sure what you mean...
The for loop step is default 1 , the "-1" is 1 less than line length, could be written (Len(TempDS)-1)

could also be

 LP1=2
 Do While Lp1 < Len(TempDS)

     LP1++   
 Loop

[Anobium]

If you took from SF then all should be good!
I will stay away from the pre-processor code. 

The For-next issue is where the test overloads and the repeat goes on and on and on..... see https://sourceforge.net/p/gcbasic/discussion/596084/thread/59d24215/

[ToniG]

I can put a test-then-Exit For .
The For loop overrun condition should not occur with a (default) Step of 1, & considering the To value is (string length - 1)
FB seems to handle Step overrun in For loops (from my observation)

Multiple string lines...
Now, I just need to make a test for the string separator... (only "," & space chr's in substring)

[Anobium]

For-Next-Step

I took a look at the source.

It may need additional code to resolve the case where a the first parameter is higher than the second parameter and the step is ( NOT negative and NOT an integer). I recommend we throw a syntax error for this case.

I think it is safest to leave the code as-as and add code to handle the case where the step <> 1. This way we ensure that we dont add addition code to the simple step 1 case, and, we know where the new code is.

I can start this action. But, I may need help. So, I will add the syntax first, then, add the the step <>1 code.

---

Re String Lines.

Sounds good!

[Anobium]

Re For-Next-Step.

bot = 1
top = 252
For i = top to bot step 5
next

This fails. Why? Top and Bot cannot be validated in the pre-processor. So, to ensure this works in all case we will have to add ASM to check that TOP is greater then BOT and negate the step value if required. Something like

if top > bot then step_variable (potentially created by the compiler from the specified constant).15 bit = 1

if top < bot then step_variable (potentially created by the compiler from the specified constant).15 bit = 0

I think this will resolve one problem.

[ToniG]

I am still confused (not unusual for me), I understand the problem but not where it is...

" I took a look at the source."

Can you clarify which source code & paste the code block where the potential issue is.
cheers

Table multi string CSV line.
This is a bit more involved than a numerical CSV line where a simple test for chr(44) will detect.
Strings {that may contain either chr(34) or chr(44) } do make it a bit more involved to detect the separator pattern & there can be any combination of spaces around the comma including none to be consistent to existing numeric CSV Table data.
I have preliminary working code & need to test combinations of string data & separator detection.

[Anobium]

Goal in these changes is to resolve two issues. These changes do not address the overflow issue - this is next.

-1 When the StartValue or EndValue of the 'For' parameters are variables and the step is a Constant then things can go wrong.

An integer must be used when either of the StartValue or EndValue are a variable. Users cannot expect the compiler to handle the dynamic nature of StartValue or EndValue variables changes during runtime - the code growth would be large. So, the compiler now issues a 'bad step' message as follows - "For-Next 'step' must be integer when start or end values are variables"

This means the user needs to ensure the step integer is a positive value or negative value to manage the the runtime StartValue or EndValue variables values.

Consider the cases.

dim StartValue as byte = 0
dim EndValue  as byte = 254
dim StepValue as Integer = 2
for ccount = StartValue to EndValue Step StepValue

or

dim StartValue as byte = 0
dim EndValue  as byte = 254
dim StepValue as Integer = -2
for ccount = StartValue to EndValue Step StepValue

or

dim StartValue as byte = range 0 to 255
dim EndValue  as byte = range 0 to 255
dim StepValue as Integer = 2

if EndValue > StartValue .. StepValue.7 = 0 'ensure positive
if EndValue < StartValue .. StepValue.7 = 1 'ensure negative

for ccount = StartValue to EndValue Step StepValue

-2 When the StartValue or EndValue of the 'For' parameters are constants and the SetValue is a constant the compile automtically resolves the Step constant as a positive value or negative value This change adds a comment into the ASM to show when a Constant is used and there was a need to change the constant to a negative…. otherwise the compiler will be making silent changes to the users source program.

Changes in SUB CompileFor (CompSub As SubType Pointer) are

SUB CompileFor (CompSub As SubType Pointer)

                …..
      'Negate step value if necessary
      IF VAL(EndValue) < VAL(StartValue) AND IsConst(EndValue) AND IsConst(StartValue) AND (InStr(StepValue, "-") = 0 And Not StepIntVar) THEN
        CurrLine = LinkedListInsert(CurrLine, ";For-next step value was automaticaly negated by compiler")
        CurrLine = CurrLine->Prev
        StepValue = "-" + StepValue
      End if

      If ( Not IsConst(EndValue) Or Not IsConst(StartValue) ) AND IsConst(StepValue) then
        LogError(Message("ForBadStep"), Origin)
      End if

[ToniG]

So if Start & End are integer variables & step is 1 then all should be ok ?

An example is getting a range of values from a table or array StartVal --> EndVal
Dim StartVal as word
Dim EndVal as word
' Step is default (1)

For Count = StartVal to (EndVal -1)

[Anobium]

Recommend that we use the new For-Next routines for this string handler. It will be a good test.

[jackjames]

"CurrLine = CurrLine->Prev"
???????????????????????????????????????????????

[Anobium]

Where are you seeing this ? Some context please.

[jackjames]

SUB CompileFor (CompSub As SubType Pointer)

                …..
      'Negate step value if necessary
      IF VAL(EndValue) < VAL(StartValue) AND IsConst(EndValue) AND IsConst(StartValue) AND (InStr(StepValue, "-") = 0 And Not StepIntVar) THEN
        CurrLine = LinkedListInsert(CurrLine, ";For-next step value was automaticaly negated by compiler")
        CurrLine = CurrLine->Prev
        StepValue = "-" + StepValue
      End if

[Anobium]

Yes, I still do not understand the question. But, I used the ->Prev to maintain the pointer to the current line after the additional line of code was added. It was the only method that worked.

[jackjames]

I'm not familiar with freebasic.
So "-> Prev" is the pointer.

[Anobium]

:-). So, the question is 'what does ->Prev do?

The compiler uses LinkedList rather than arrays to improve performance. A linked list is a structure that is easily expandable by using a single function, and it comes in very useful when you need an 'array' of something but you have no idea how many 'arrays'. The concept behind a linked list is that each node structure has a pointer to the next and previous node structure. This is called a double linked list, as it links to two different nodes. By using a pointer to a structure, you can specify a null pointer if there is no next or previous node, and since the pointer stores a memory address, the amount of nodes you can store is limited only by memory.

Hugh did use arrays then moved to LinkedLists the compile times decreased hugely.

[jackjames]

Thanks for the exhaustive explanation.

[Anobium]

Pleasure

[ToniG]

Recommend that we use the new For-Next routines for this string handler. It will be a good test.

Anobium, I very much appreciate the detail,
1. For me to get a clear understanding, if you could modify(or identify code block) & return the code I have created & I can explore the difference .
2. Can you please clarify 'this string handler"...
You mean the 'Sub TableString' or the GCB code that handles string tables(that are integer tables when compiled)
For 'Sub TableString' it would be out of scope as I understand as it deals only with GCB source lines between "Table " --> "End Table" & those that are a string "......."

I have finished the inclusion of processing Table CSV string data lines.
For the attached code 'Sub TableString' , it can be considered as proof of concept in that it produces the desired result from required input, so can be tested as is. But it will most likely need some change.
I have already redone the CSV string detection/process block, but it can be improved.

See end of file 'ReadTableStrings2.gcb' for detail of Table string line rules.

(only changed files attached, replace previous submitted files)

[Anobium]

Thank you. Nice. Good code!

Forget about my 'this string handler" comment. I am referring to your table handler.

---

I have tested, I did hit the issues you raised and then I updated. To resolve a few of the issues you raised.

The processor.bi attached now support escape characters which means that we have the following capability - this in the output on the terminal.

123678,9AB
"DQuotes"
'SQuotes'
calculate 4\2 = 2
Use pattern ","

Accepted escape strings are shown in the table below.

Escape sequence	Meaning
\a	beep
\b	backspace
\f	formfeed
\l or \n	newline
\r	carriage return
\t	tab
\&nnn	ascii char in decimal
\\	backslash
\"	double quote
\'	single quote

I have also added a DQuote checker. If the number of DQuotes are unbalanced then an error is issued.

---

Example code:

#chip 16f88

#define USART_BAUD_RATE 9600
#define USART_TX_BLOCKING

#option explicit

dim tablelen, tabledataptr, tabledata as Word
ReadTable Tdata, 0, tablelen


for tabledataptr = 1 to tablelen
  ReadTable Tdata, tabledataptr, tabledata
  HSerSend tabledata
next

do
loop


Table Tdata
"123"
"678,","9AB"
"\&013\&010"
"\"DQuotes\"", "\&013\&010","\&039SQuotes\&039"  'This is Dquote in the comments
"\&013\&010"
"calculate 4\\2 = 2","\&013\&010"
"Use pattern \"\,\""
"\&000"
End Table

---

You can now add \&000 as a string terminator. This means you can index the strings without the need for an external application. Means you can walk the table of string (or you can preindex) - I am showing walking the table of strings - I will you sort preindexing.

' --- This is test code for Table strings ---



#chip 16f88
#config Osc = Int
#define USART_BAUD_RATE 9600
#define USART_TX_BLOCKING '< wait for tx done
#Define TblStrNum 20      '< num of strings

#option explicit
Dim MnuTblLen as Word     '< length of table
Dim PtrTblLen as Byte     '        "
Dim Result as String
Dim Result0 as String     '< for bug in older compiler <= V0.098.00
Dim StrNum as Byte        '< num of string in table
Dim StrIndex as Word      '< String to get
Dim _Index as Byte
Dim Lp1 as Word           '< loops
Dim Item as Byte          '< byte from table
Dim StrStart as Word
Dim StrEnd as Word
Dim BtnPressed As Bit          '



' In a real menu the Menu Level, Item Index, Data Table are selected based on menu navigation.
' This code is just getting a selected string from a table of ASCII bytes & print on serial
' to test working.                    LCD_Menu is a WIP

    Gosub Init1
'---------------------------
Main:
    Gosub ScanButn
'    Gosub MenuNav
    Gosub MenuItem
'    Gosub Do other stuff
'    Wait 1500 ms   '<Comment for sim
goto Main
'---------------------------


MenuItem: 'TX table strings (StrNum as byte),
          If BtnPressed = 0 then Return
          For _Index = 1 to TblStrNum
            Gosub GetTblStr0
            HSerPrint leftpad( str(_Index ), 2 ," ") +  ")"
            HSerPrint Result
            HserPrintCRLF
          Next
          HserPrintCRLF
          HserPrint "----------------"
          HserPrintCRLF
'          Wait 5 ms   '<Comment for sim
Return


GetTblStr0:
          Result = ""

          Dim TableSize as word
          ReadTable MnuTxt1, 0, TableSize
          StrIndex = 0
          StrStart = 1

          For Lp1 = 1 to  TableSize
              ReadTable MnuTxt1, Lp1, StrEnd

              If StrEnd = 0 then
                  StrIndex =  StrIndex + 1
                  if StrIndex = _Index then
                      StrEnd = Lp1
                      Exit For
                  else
                      Lp1++
                      StrStart = Lp1
                  End if
              End if

          Next

          If StrEnd = 0 then return

          For Lp1 = StrStart to StrEnd - 1
             ReadTable MnuTxt1, Lp1, Item  '< Readtable data item
             Result0 = Result + Chr(Item)
             Result = Result0
          Next

Return


ScanButn:
  ' Button(s) handler here...
  BtnPressed = 1
Return


Init1:
     ' Get Table(s) size
     ReadTable MnuTxt1, 0, MnuTblLen
Return


    '==========  TABLES ===============

    Table  MnuTxt1  ' Sub1 Menu
       'These are csv string lINES (use with modified Preprocessor.bi (Sub TableString))
       "\"This is Master Menu Item 1\"\&000", "Menu Item 2\&000", "Menu Item 3\&000", "Menu Item 4\&000", "Menu Item DropDown 5\&000"  ' SUB1
       "Config1 Itm 1\&000", "Config1 Itm 2\&000", "Config1 Itm 3\&000", "Config1 Itm 4\&000"         ' SUB2
       "DataLog Itm 1\&000", "DataLog Itm 2\&000", "DataLog Itm 3\&000"                          ' SUB3
       "Diag Itm 1\&000", "Diag Itm 2\&000", "Diag Itm 3\&000", "Diag Itm 4\&000"                      ' SUB4
       "Help Itm 1\&000", "Help Itm 2\&000", "Help Itm 3\&000", "Help Itm 4\&000"                    ' SUB5

    End Table

This will yield the following. You simply terminal a string with escape \&000 and the revised SUB GetTblStr0: now walks the table - there are faster ways of walking the code be preindexing still using the \&000 as index key.

You can simply edit the table and the code sorts the string length for you.

Master Menu Item 1
Menu Item 2
Menu Item 3
Menu Item 4
Menu Item DropDown 5

Seems to work nicely.

:-)

For preprocessor.bi see https://sourceforge.net/p/gcbasic/discussion/579125/thread/00242c7bee/3deb/attachment/preprocessor.bi

[Chris Roper]

Nice addition.
may I suggest that you Escape Zero too such that:
Escape sequence Meaning
\0 NUL Terminator - Allows ASCII Strings to be terminated in ASCII

[Anobium]

See https://sourceforge.net/p/gcbasic/discussion/579125/thread/00242c7bee/?page=1&limit=25#3deb for the latest post - but, this is a good idea.