Menu

SQL Preprocessor, What is that exactly?

2022-10-04
2022-10-28
1 2 > >> (Page 1 of 2)
  • Aoirthoir An Broc

    looking at all the posts and looking at Vincent's applewood software I see discussions about an SQL preprocessor. I haven't quite figured out what that is supposed to be the best I can guess is from a statement I saw regarding that in the FAQ where it said that the preprocessor removes the tedium of having to enter SQL statements.

    and I'm not sure what that means because SQL statements for me and you have always been easy to write. I tend to create stored procedures when I was a database administrator and not let programmers have direct access to the database except occasionally perhaps I would give them select capabilities but other than that I would create the stored procedure and then they would call it.

    so if I am comfortable creating the SQL statements themselves do I even need a pre-processor or is there a way to simply have SQL statements that I just write myself and call the MySQL or postgres or sqlite or firebird and so on libraries?

    I looked at the sample for open Cobol Esql and the way that looks like it's processing it's just using normal SQL statements as if they were Cobol statements without having to perform any of the quoting like you do in PHP when you create your statement.

    So in php $sql = "select from customer where primarykey = $id"

    and I realize that it's not exactly correct and there's lots of ways you can do SQL and PHP but the point is that you usually wrap your SQL statement in quotes and then that's how code injection works which is why you had to do all of this escaping but in the example from esql we get this...

               EXEC SQL 
                   DECLARE C1 CURSOR FOR
                   SELECT EMP_NO, EMP_NAME, EMP_SALARY 
                          FROM EMP
                          ORDER BY EMP_NO
               END-EXEC.
               EXEC SQL
                   OPEN C1
               END-EXEC.
    

    and to me that is what an SQL statement should look like. so my question is do I need the preprocessing to do any SQL or do I just do statements similar to that and I know you have to connect to the library and all of that. or do I need to do the SQL preprocessing in order to be able to do statements like that above.

    and of course if so which preprocessor do I use.

    I put this in the lounge rather than other places because it's not super important right now and if you guys think it should be moved to another then feel free and hopefully I get the code thing right.

     

    Last edit: Simon Sobisch 2022-10-04
    • Vincent (Bryan) Coen

      Take a look at dbpre available via :

      https://sourceforge.net/projects/dbpre

      This supports Mysql / Mariadb.

      Documentation is a bit weak as it does not show the SQL commands supported so you have to read the C source.

      Now I did start to work on the ACAS supplied JC pre-compiler to support a $SET SQL command starting with MYSQL i.e., $SET SQL MYSQL

      (but also adding other RDB's as well as ODBC)

      and generate the needed calls etc but got waylaid on some thing else but I have since found out that there are other tools that do support Mysql although may be not written in Cobol.

      My problem is I need other written tools to get the call syntax used for the various others in order to hook them in but there again with other tools should I do so ?

      Bit of a case of the blind may be leading the blind :)

      or is it the unknowing ?

      Vincent

       

      Last edit: Simon Sobisch 2022-10-04
      • Aoirthoir An Broc

        @vcoen that example helped a lot also. Thank you.

         
    • Brian Tiffin

      Brian Tiffin - 2022-10-04

      Hello, Aoithoir. It's been a while. The time has come the forum said to talk of many things, of shoes and ships and SQLing wax, and other COBOL bling.

      This note isn't just for you, Aoithoir, but I'll be writing for the whole forum to read through too.

      ESQL is an old standard? for Embedded SQL. I've never actually seen a spec for it, and whether it is codified or just a defacto standard based on an original implementation that stuck.

      GnuCOBOL programmers have a few choices for preprocessors. One specific to PostgreSQL out of Japan, Sergey's for ODBC/MySQL/MariaDB, Marco's as part of GIX, Firebird gsql, one for Oracle that has been proven to work with GnuCOBOL, DB2 processors ( László has great examples of this using DB2 Express (and PostgreSQL) in the Contributions source tree here), and probably one or two I'm missing at the moment. These take sources that include EXEC statements and generate COBOL compiler friendly outputs that include a lot of CALL verbs. ESQL -> COBOL -> Program.

      The common preprocessor that most programmers bump into is the C preprocessor #include. C, the language, knows nothing about #include, but the toolchains for a C compiler include the cpp pass as a step in the compile. COBOL and GnuCOBOL have similar with COPY and REPLACE. COBOL, the language, knows nothing about COPY, it is a step in the compile called the Compiler Directing Facility in the spec, which includes Text Manipulation and new conditional compile directives like >>IF.

      This is not the same as ESQL, which is usually a completely separate step in preprocessing.

      An alternative to ESQL is simply using CALL in the COBOL sources. I'll use SQLite as an example. CALL "sqlite3_exec" USING BY VALUE db BY REFERENCE sql-statement ... This is my favoured way of using GnuCOBOL with external resources like database engines, but, not the way most COBOL programmers are shown how to embed SQL. Most real world COBOL will use EXEC, END-EXEC and an extra external step before invoking the COBOL compiler.

      The recommendations that follow are personal opinions. I usually choose PostgreSQL for larger efforts, and the preprocessor out of Japan. SQLite3 for all the other things and no preprocessor, but just COBOL and CALL. For everybody else, I recommend reading through

      https://sourceforge.net/p/gnucobol/contrib/HEAD/tree/trunk/samples/DBsample/DB2/
      and/or
      https://sourceforge.net/p/gnucobol/contrib/HEAD/tree/trunk/samples/DBsample/PostgreSQL/
      mainly because László does such an awesome, comprehensive job explaining things, in small steps. And his contributions can end up being quick start template source kits for hitting the ground running.

      Can't really recommend a best ESQL for anyone, as they usually happen to be either personal preference or shop mandates. I have no experience with Marco's Gix integrated ESQL layer, but I get a sense that it'll be the way to go for anyone using the Gix IDE as you can learn all the things in the one thing.

      Have good, make well,
      Blue

       
      • Brian Tiffin

        Brian Tiffin - 2022-10-04

        Oh, and I should mention that Ron Norman has been adding features to trunk for XFD fields, built into cobc/libcob for GnuCOBOL 4. I've not yet dug in and documented the new things, but GnuCOBOL 4 will be more SQL ready (of the ODBC type) than previous versions. XFDs are an extension used in another small tin COBOL system, which I never remember the name of until I go look.

        Cheers,
        Blue

         
        • Simon Sobisch

          Simon Sobisch - 2022-10-04

          The XFD directives are used as in ACUCOBOL-GT; the resulting xfd files aren't.
          This really cool GnuCOBOL4+ only feature allows you to code ORGANIZATION IS INDEXED but have libcob connect to an RDBMS instead of one of its libraries for INDEXED handlers (another 4+ only feature is that you can have multiple INDEXED handlers configured and then configure down to each file which indexed handler to use).

          This is especially useful if you only have "some" parts stored in an RDBMS or need to switch and then want to change some of the "old" access ways to EXEC SQL for "direct, fast SQL access" [there's currently no way to use a single connection, but that is likely to change some day, at least for GixSQL and esqlOC, so some care must be taken when it's about transactions].

           
          • Aoirthoir An Broc

            So if I am understanding the XFD you mention for GnuCOBOL 4+ basically I would just have my normaly COBOL file definitition, and point it to an RDBMS (mysql, postrges, db2 etc) and then my OPEN, READ etc just proceeds as normal? So then I do not actually write SQL statements and instead write normal cobol and libcob takes care of the interface?

             
            • Simon Sobisch

              Simon Sobisch - 2022-10-09

              Yes, as mentioned by Vince the PG has the necessary content.

              ... and of course you understand that RDBMs, accessed in ISAM ways are quite slow - doing this with an ISAM implementation on a disk (especially on NVMe) without the network (or shared memory) io, without multiple processes and especially: without translation is nothing you can hope for.
              But as noted: "native" access via EXEC SQL and doing queries/joins/calculations/mass-updates directly on the database for all places that matter will often provide a speedup. too.

               
              • Aoirthoir An Broc

                And considering that SQL is simple honestly, that is just as fine. I suppose it might be an advantage if you are using SQLite or ISAM and you want to switch for some reason, that has an advantage. But otherwise SQL just makes more sense.

                Especially if the ~ EXEC SQL ~ option lets you switch between SQL flavors.

                 
        • Vincent (Bryan) Coen

          See PG content, it is in there for a month or few.

           
          👍
          1
          • Simon Sobisch

            Simon Sobisch - 2022-10-09

            Vince, do you think it would be reasonable to add something about EXEC SQL to allow people understanding the idea in general, possibly with referencing esqlOC and GixSQL?

             
            • Vincent (Bryan) Coen

              On 09/10/2022 01:05, Simon Sobisch wrote:

              Vince, do you think it would be reasonable to add something about
              |EXEC SQL| to allow people understanding the idea in general, possibly
              with referencing esqlOC and GixSQL?

              Not really as that is the responsibility of the authors of the
              documentation for 'that' specific pre processor and many have differences.

              The best one may be could provide is the sources of where to get such
              but there again it is outside the scope of a Cobol Programmers Reference
              manual as the compiler has no direct support for any pre processors.

               
            • Eugenio Di Lorenzo

              I believe this is a topic that could find a much better place in Brian's FAQ.

              IBM has always published both the LANGUAGE REFERENCE manual which simply describes the language and the PROGRAMMER'S GUIDE manual which instead contains examples, comments, suggestions (such as the FAQ) ....

              By analogy, one might think that Vincent's updated manual is more of a LANGUAGE REFERENCE and its title should be changed.
              Brian's FAQ is more of a real PROGRAMMER'S GUIDE.

               
              • Aoirthoir An Broc

                I tend to agree with you here.

                Most FAQ's are not actually questions that people have asked. There are a lot of things in the current FAQ that began as proofs of concept that @btiffin did and that showed us just how powerful the GnuCobol compiler really is.

                Now I believe is the time for an Quick Start Guide, a simpler FAQ that focuses more on questions and that Brian's FAQ shouldl be moved to the name Programmer's Guide and Vincent's is definitely the Language Reference.

                @vcoen and @btiffin might disagree I do not know.

                But it would be nice to have a couple of short documents that get us going (quickstart.. which includes how to do all the basics.. cobc, cobclib, berkeley, mysql/posgres, vcodium/gix)

                I had suggested elsewhere the idea of appimage, but frankly if I had a good quickstart to get all these going, and up and running that would have been it a bit easier.

                The problem is...or is it a problem, do we make a quickstart guide for different operating systems, or one, with multiple sections. .....ok i am getting offtopic so I am going to create a separate topic in the lounge...

                 
                • Vincent (Bryan) Coen

                  Yes, I agree that the PG should be changed (copied over)  to Programmers Reference and will do so on the next update and this is something that I wanted to do for a few years now..

                  I will retain copy of the current Programmers Guide to be use as a basis for some one to change to be the guide and start to remove examples etc from the Reference as and when time permits.

                  Vincent

                   

                  Last edit: Simon Sobisch 2023-12-04
                  • Aoirthoir An Broc

                    Excellent. Is that the same as the language reference? Or would that be something else entirely?

                     
                    • Vincent (Bryan) Coen

                      I will use the existing PG (Programmers Guide) to be the PR (Programmers Reference) then remove the examples type stuff from it.

                      That way someone can take on the work for the PG. even if it was transferred to the LibreOffice .ODF format and tool to create - at least that way it would be the modern WYSIWYG word processing etc.

                       
                      • Aoirthoir An Broc

                        Excellent. Thank you. I have a couple people that might be able to help us with a GnuCOBOL Series Documentation. I think it would be good when we all have time to decide what we need (I tend to like short guides..or topic specific guides as mentioned on the other thread.. in addition to large references)

                         
                      • Eugenio Di Lorenzo

                        Hello everyone, I would like to clarify the issue related to the manuals.

                        IBM that we can take as a reference makes available:
                        - a LANGUAGE REFERENCE
                        - a PROGRAMMING GUIDE
                        See screenshot attached.
                        The difference between the two manuals is not simply the presence or absence of COBOL code !
                        The first LANGUAGE REFERENCE document explains how the language is structured and its syntax. In some cases, when it is useful, it also presents some pieces of COBOL source code as an example.
                        The second PROGRAMMNG GUIDE document presents general topics such as the use of indexed files, the use of SORT mechanisms, debugging techniques, etc.

                        For the above, simply the manual we now call GnuCOBOL Programmer's Guide should be called GnuCOBOL Language Reference.
                        Without any changes regarding its content!
                        Please don't delete the small COBOL code examples that it contains and that are used to clarify the language syntax!

                        Subsequently, the Brians FAQ could be the basis for creating a PROGRAMMING GUIDE. But this would be a longer job.

                        About the mnual source format I agree that the current format is not very attractive but TEXINFO has the objective and the advantage of allowing the development of PDF, HTML etc from the same source.
                        For this reason I think it would be better now to leave the source in TEXINFO format. The lack of skills that Vincent declares on TEXINFO should be remedied with the support of those who can explain how to automatically renumber the chapters, for example.
                        Gary, I suppose, might be able to explain how it should be done very simply.

                         
                        • Vincent (Bryan) Coen

                          Usage of LibreOffice writer allows for creating .pdf and html files but I have never tried the later.

                          The Tex info system and previous guises dates back to the early 70's and yes I have some manuals written in that i.e., original docs for my ACAS system although I cannot locate the software to process them but there again these were changed to use wordstar and that also died  a wee death.

                          The only issue I see with using writer is how to split up the input documents in a similar manner to existing in order to reduce overall size of the input manual as my import to writer is massive but there again it creates different styles for page, line, para, headings etc for each page and changing them is NOT a one day job !

                          The information to be removed from the moved over Prog. language manual will be contained within what is needed and what is not - i.e., an
                          introduction to what COBOL is is totally irrelevant and no use to a COBOL programmer and is NOT intended to replace a teaching tool such as a book etc  -  it is not that.

                          That said when I learnt COBOL in the early 60's I used the 50 page manual for IBM 1401 Cobol and when from there.
                          The next thing was looking at other programmers work - although that is another story :(

                           

                          Last edit: Simon Sobisch 2023-12-04
                          • Eugenio Di Lorenzo

                            https://devdocs.io/gnu_cobol/
                            this is an example of what can be generated from the same TEXINFO source. I don't remember who did it ...
                            Always from the same source I remember that a "HELP FILE" of the language can also be generated, That is one of those with the extension .chm which are basically navigable manuals and which could therefore be linked by an IDE e.g. GIX-IDE.

                             
                          • Aoirthoir An Broc

                            @vcoen

                            When I mentioned Libreoffice, I thought that was what you used. I would be glad to use any format of your preference.

                             
                            • Vincent (Bryan) Coen

                              No, you have the wrong end of the stick !

                              LibreOffice is for word processing Only.

                              If you are referring to what is used for ACAS please read the Building ACAS manual for details but it should be indicating the JC SQL type pre-compiler.

                              How ever it is a hybrid type product and does not use the sequence SQL EXEC etc but does have extra features that are not present in the SQL EXEC group.

                              Vince

                               

                              Last edit: Simon Sobisch 2023-12-04
      • Simon Sobisch

        Simon Sobisch - 2022-10-04

        Yes, ESQL is at least standardized, and there is a formal test specification for embedded SQL, too - https://www.itl.nist.gov/div897/ctg/sql_form.htm (took a while to get that back to NIST servers ;-)

        The biggest benefit of embedded SQL (you can have that in other languages, too) is that you don't have to take care of the possibly necessary conversions of data types and are "nearly" independent from the SQL dialect (as long as you don't do something very specific and add nonstandard SQL - there's some similarity to COBOL dialects...).

        If this is coupled with a preprocessor that allows you to use multiple different RDBMS (either "direct switch" like GixSQL with its drivers or by an ODBC driver) you can write 1 COBOL program and then switch between the RDBMS in the background (even without recompile) - writing hard wired CALL makes this quite hard.

        My personal (in this case even "professional") experience with the available preprocessors:

        Pro*COBOL from Oracle is very mature and fast - but it is also totally closed source (and thinking back how that was with JDBC: the official maven driver source gets you a source that says "you should not need that, get a contract and open a support ticket" - so "no thanks").

        Open-COBOL-ESQL seems to work with small environments, at least as long as you don't use COMP-3 (there are a bunch of memory errors with these that likely crash your process..., while it seems that I've fixed those [test environment runs] I've not found the time to upstream those changes yet [part of the reason behind that is when I've did that there was zero response from its developers, but that got better]), but (aside from these memory issues) it is limited to PostgreSQL and forces you to have all DECLARE statements in the PROCEDURE DIVISION [it is the only sql preparser I've seen that has this limit, so if you have "legacy sources" you'd have to move these in the code]).

        esqlOC was working without any memory issues or strange incompatibilities - but as it only supports ODBC needs a good ODBC driver in the background - an an application that uses "SQL as it should be" (if it uses it "very ISAM like" with a lots of small "get this one record", then the application and ODBC may not work well together - it was too slow in the main application I've tested - but that's really an issue of the application logic [more or less replaced ISAM by EXEC-SQL with no big adjustments to the logic], that shouldn't happen when you "start fresh").

        GixSQL is awesome in all places I've tested ... but so far always missed an important piece that prevent it to be used for that legacy COBOL application (again: not necessarily an issue if you can start "fresh").
        In general its pro's are: best maintained, best error handling during pre-parsing, best configurable logging, option to switch between its backend drivers via configuration.
        GixSQL also recently got an Oracle backend, so using this when you are for whatever reasons "forced" to connect to an Oracle DB seems like an interesting option.

        Currently I'm using a heavy patched Open-COBOL-ESQL version with the goal to support the legacy application to move from Oracle and Pro*COBOL to PostgreSQL, but I hope be able to switch to GixSQL and its PostgreSQL backend this year.

         
        • Aoirthoir An Broc

          If this is coupled with a preprocessor that allows you to use multiple different RDBMS (either "direct switch" like GixSQL with its drivers or by an ODBC driver) you can write 1 COBOL program and then switch between the RDBMS in the background (even without recompile) - writing hard wired CALL makes this quite hard.

          Everything you said in this post I understand, including this. But what I do not know is the mechanism. So the EXEC SQL/END EXEC still happens? Then the Preprocessor converts that to COBOL calls? or skips that entirely somehow? (A side question would all this work with the new gCobol that you linked me too?)

          Currently I'm using a heavy patched Open-COBOL-ESQL version with the goal to support the legacy application to move from Oracle and Pro*COBOL to PostgreSQL, but I hope be able to switch to GixSQL and its PostgreSQL backend this year.

          Then that is what I am going to do. Call me a follower.

          In general I would prefer the Exec SQL model without preprocessing, like the Cobol compiler just saw that as normal cobol and took care of the rest without a precompile but I am selfish, and like my cobol looking all coboly smooth. I dont object to CALLs but each time a CALL can appear to just be cobol.. yay!.

          Anyhow now I think I understand and it turns out that its kind of the opposite of what I originally understood. And that is fine.

           
1 2 > >> (Page 1 of 2)

Anonymous
Anonymous

Add attachments
Cancel