Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

buil

Developers
Ken Monks
2010-08-16
2012-11-20
  • Ken Monks
    Ken Monks
    2010-08-16

    We are developing another Open Source mathematics program called Lurch (http://lurch.sourceforge.net) and are looking for an Open Souce CAS to build into our project.  My collaborator and I are wondering:

    1. Is it easy to compile REDUCE into a larger c++ project, or is there a really complex build process?
    2. Is the REDUCE modified BSD license compatible with our GNU GPL 3 license so that we can use it?

    I have fond memories of using REDUCE in my days in graduate school.  Thanks.

    -KEN

     
  • Arthur Norman
    Arthur Norman
    2010-08-16

    First observation: good luck with Lurch - I had not spotted it before but will now look.

    With regard to your questions:

    1. Taking ANY twio substantial pieces of software and linking them so they interact reasonably seamlessly is a serious undertaking! But the CSL version of Reduce is sort of "just" a 100,000+ line C program and it is built by just compiling all the source files in a siple flat manner. That then needs to have an "image file" that comtains the guts of the Reduce algebra bit, but our existing scripts let you make that. And one image file (reduce.img) shoudl work with pretty well any C platform.

    The trunk/csl/embedded directory you will find if you check out the Reduce sources from subversion are not up to date with respect to the main sources, but can be re-synced every so often. The Makefile there reads merely

    # Makefile for this SIMPLE setup for Reduce building

    CC=gcc
    CFLAGS=-m32 -O2 -DHAVE_CONFIG_H=1 -DEMBEDDED=1

    all:    reduce

    reduce:         arith01.o arith02.o arith03.o arith04.o arith05.o \
                    arith06.o arith07.o arith08.o arith09.o arith10.o \
                    arith11.o arith12.o bytes.o char.o \
                    csl.o cslmpi.o cslread.o eval1.o eval2.o eval3.o \
                    eval4.o fasl.o fns1.o fns2.o fns3.o fwin.o gc.o \
                    preserve.o print.o restart.o sysfwin.o termed.o \
                    u01.o u02.o u03.o u04.o u05.o u06.o \
                    u07.o u08.o u09.o u10.o u11.o u12.o
            $(CC) $(CFLAGS) arith01.o arith02.o arith03.o arith04.o arith05.o \
            arith06.o arith07.o arith08.o arith09.o arith10.o \
            arith11.o arith12.o bytes.o char.o \
            csl.o cslmpi.o cslread.o eval1.o eval2.o eval3.o \
            eval4.o fasl.o fns1.o fns2.o fns3.o fwin.o gc.o \
            preserve.o print.o restart.o sysfwin.o termed.o \
            u01.o u02.o u03.o u04.o u05.o u06.o \
            u07.o u08.o u09.o u10.o u11.o u12.o \
            -lm -o reduce

    clean:
            rm -f arith01.o arith02.o arith03.o arith04.o arith05.o \
                    arith06.o arith07.o arith08.o arith09.o arith10.o \
                    arith11.o arith12.o bytes.o char.o \
                    csl.o cslmpi.o cslread.o eval1.o eval2.o eval3.o \
                    eval4.o fasl.o fns1.o fns2.o fns3.o fwin.o gc.o \
                    preserve.o print.o restart.o sysfwin.o termed.o \
                    u01.o u02.o u03.o u04.o u05.o u06.o \
                    u07.o u08.o u09.o u10.o u11.o u12.o

    # end of Makefile

    That version does not support a GUI but is provided specifically so people like you could look to see if you wanted to hack stuff into a shape that would help you!

    What may be harder is interfacing between your data and Reduce data. The most trivial way is to invoke Reduce via a pipe, or after looking at trunk/libreduce. Eg that is how it can be used from TeXmacs.  By hacking the C code I have in the past made the link not via a pipe but via direct procedure calls (see SAMPLE_OF_PROCEDURAL_INTERFACE in trunk/csl/cslbase/csl.c). A key decision is whether Lurch was the main program wanting to call bits of Reduce as subroutines, or Reduce was to be the main program calling bits of Lurch, or if they run in parallel with all the pain and fun of inter-process communication!

    2. Every proper person denies being a lawyer when any question about licenses arises! So in the end it is YOUR call what code you accept. But the BSD license is the ultimate permissive one and you can use it anywhere including with code subject to all the constraints and restrictions (or protections if you insist!) of GPL3. But "compatibility" is not reflexive. We have licensed Reduce in a way that lets you use our code without compromising your GPL3 sensitivity, but we can not accept back GPL3 code. So if you make extensions to CSL or Reduce (or if you find that the PSL Lisp platform suits your needs better) you are 100% free to make your extensions GPL3 while leaving our code BSD. But it would be nice if you were willing to make any changes you design for Reduce and its Lisp available under our license so we can happily merge them into our sources. You could do that and it would not prevent the rest of Lurch being solely under GPL3.

    But the short answer is that the BSD license allows you to make pretty well whatever use of our code in whatever commercial, GPL or other context you want - you can merge in bits of your own code and they can be licensed any way you like, But of course that will not change the license on the code we wrote.

    On www.gnu.org it says

    Modified BSD license

        This is the original BSD license, modified by removal of the advertising clause. It is a simple, permissive non-copyleft free software license, compatible with the GNU GPL.

        If you want a simple, permissive non-copyleft free software license, the modified BSD license is a reasonable choice. However, it is risky to recommend use of “the BSD license”, because confusion could easily occur and lead to use of the flawed original BSD license. To avoid this risk, you can suggest the X11 license instead. The X11 license and the revised BSD license are more or less equivalent.

        This license is sometimes referred to as the 3-clause BSD license.

    Hope this helps. Please first try  downloading and building the full regular Reduce in both CSL and PSL versions, and then maybe look at the "embedded" option. We will (I hope) all be willing to help a bit if you do decide to go ahead.

              Arthur

     
  • Nathan Carter
    Nathan Carter
    2010-08-16

    Hello!  I'm the collaborator Ken referred to in his original post.  Thank you for the quick reply, Arthur.

    Regarding building and interfacing, I think what I want to do is what you call the embedded version.  We use Qt and qmake, so translating that Makefile over is very straightforward.  I could even link to your code as a subversion external, and just add it to our project by writing one short .pri file.

    I was able to successfully build trunk/csl/embedded by just issuing "make" in that folder (on my Mac).  I ran the program and did a few simple things to test and it seems to work!

    I'd like to write a conversion function in C++ from our expression trees to yours, and another that goes the other way.  Then to do some REDUCE operation on one of our data structures, we just convert, ask REDUCE to do it, and convert back.  I've had success doing this with another CAS, but its inferiority is becoming painful, and now we're hoping to upgrade to REDUCE.  I'll do this instead of a pipe, because it seems more elegant than creating and parsing representations of things through a pipe.

    I assume there are functions for building REDUCE expressions, and traversing REDUCE expression trees, and if I have those, then I think I'm all set, unless you foresee any pitfalls to this approach.  Can you point me to any source code docs where I could find out how to build REDUCE expressions, and call standard routines on them, like simplify/solve/expand/differentiate/integrate/etc.?

    Thank you very much for the help so far, and for the foresight to make the embedded directory in the first place!  And thank you for the license info, which was just what I'd hoped.

    Nathan

     
  • Arthur Norman
    Arthur Norman
    2010-08-16

    One thing I will note is that the files in the embedded dircetory are a snapshot, so I update them from time to time - but CERTAINLY for your initial experiments the fact that the build sequence there is really easy is maybe best.

    Asking Reduce to do something and passing in tree structures etc will take some setting up. For instance Reduce has its own garbage collector and any data it is ever looking at MUST fit in with that.

    There is no single neat manual documenting internal structure. Well many years ago there was a "Reduce Newsletter" and if anybody who reads this forum has any old copies somewhere it would be really good to collect scans of them and make them available, since there were certainly some articles back then that might help!

    In the file "csl.c" see SAMPLE_OF_PROCEDURAL_INTERFACE. This is s scheme that turns Reduce input and output into callbacks to functions you provide and lets you invoke an arbitrary Lisp/Reduce function by going

        cslstart(argc, argv);    // just once to allocate store, load image, initialise
        execute_lisp_function("function name", input_callback_fn, output_callback_fn); // several times?
        cslfinish(NULL);  // tidy up at end.

    That gives you some flexibility beyond that which you get with pipes, but is still a textual interface. The idea is that you then write a function and built it into your Reduce image that does what you want, then call it. Depending on how and what you do this may or may not want to do I/O.  I put a silly function called oem!-supervisor in calbase/extras.red and extras.lsp to show this in action.

    You can also add your own functions to CSL (coded in C) so eg you could have
        Lisp_Object my_result;
        Lisp_Object save_my_result(Lisp_Object nil, Lisp_Object a)
        {   my_result = a;
             return onevalue(nil);
        }
    and put a line in a setup table so this was available from Lisp. Then you make your Lisp?Reduce code call it at its end. Then after that you have a pointer to whatever raw data structure there is. You should view it as volatile, since any garbage collection might move the data around in memory!

    I think I suggest you start with a prototype or experiment that uses a textual interface. if you send "off nat;" to Reduce as a command then its output can be 1-dimensional (or you could think in terms of the tex-like output modes - but the simple ones are simpler, and would probably not be much of a pain for you to re-parse. Once you have some easy cases going there and if it looks worthwhile you can try to hook in in a more intimate manner.  Naturally text mode involves overhead with both sides parsing etc but …

    I bet some people would advise use of the mathml (openmath) package but I have never used it so do not understand it. Maybe you do.

    You may email me as acn1~cam.ac.uk if you are going to try serious hacking so we can discuss and work out ideas using the easiest for of email - but then at the end we can post a summery of conclusions for broad readership.

               Arthur

     
  • Arthur Norman
    Arthur Norman
    2010-08-16

    Hmmm - another thought on what I could provide fairly easily. I will post these thoughts here before I just do it in case somebody can suggest improvements or in case you think it would be no use to anybody at all!

    Imagine in the CSL a procedural interface that provides an interface styled after an RPN calculator, The calls I would imagine woul include
        push_symbol("x")
        push_32_bit_integer(123);
        push_big_integer("12345678901234567890");
        push_native_float(2.7);
        push_bigfloat("3.3333333333333333333333333333333333333333e333");
        add(), subtract(), multiply(), divide(); square(), power();
        differentiate(), integrate(); // as in push_symbol(x); push_32_bit_integer(6); power(); push_symbol("x"); differentiate();
            for df(x^6, x)
        make_function(n_args, "name"); as in push_symbol("theta"); make_function(1, "sin");
        save(n); load(n);  // with a bunch of "memory locations"
        option_on("option-name"), option_off("option name") // for Reduce switches
        pop(); // discard top item on stack
        dup(); // duplicate top item on stack
    That style should support the ability to get Reduce to do most things, and the user is kept away from worries about parsing
    or internal representation.
    where
    Then there is a need to get results back. We  Reduce has a simple prefix form where eg (sin alpha + 2)^2 ends up as
    (plus (expt (sin alpha) 2) (times 4 (sin alpha)) 4)
    I could provide functions such as say
        atomic(p)                     bool
        is_integer(atom)             bool
        integer_value(integer)   32-bit int
        name_of_sym(p)            char *
        operator(p)                
        number_of_operands(p)
        operand(n, p)
    to allow one to traverse such, and a call
        p = get_prefix_form()
    to get the prefix form for the top item on the stack.

    If I do this the C names for these functions will start with some prefix such as PROC_push_symbol(char *) to reduce
    name clash prospects.

    Is it likely that me spending a few hours doing this would count as helpful?
    Can anybody suggest a good alternative, or useful enhancements, or point out risks and pains????

                      Artthur

     
  • Nathan Carter
    Nathan Carter
    2010-08-16

    Lurch is actually heavily steeped in OpenMath, and all our data structures use it.  I see that REDUCE has a MathML package and a routines for translating from MathML to/from OpenMath.  We've already got OpenMath input and output routines built, so that combination seems perfect.

    I'm a little confused, though, because the REDUCE user manual just mentions one of these two packages, the MathML one, for converting expressions to and from MathML.  But then the trunk/packages/mathml folder just has the package for converting from MathML to OpenMath.

    Ah, as I was writing this, you posted the RPN message.  The RPN interface you suggest would naturally fit smoothly into a traversal or construction of one of Lurch' syntax trees.  So now I have two good choices!

    But I don't want to make work for you, so let's see if we can do it the MML/OM way first.  Thus I have two questions remaining.  The first is my confusion above about exactly which package is which, or if there are two.  The second is how I go about getting the embedded version to compile that package, too.

    Nathan

    PS: Thank you again for this very helpful discussion!

     
  • Arthur Norman
    Arthur Norman
    2010-08-16

    mathml/openmath is not an area I understand. The source directory packages/mathml will show you the contact point for the authors of that stuff (in around 1999 - I do not know if they are still interested, but I hope they would help!). So I do not know just how well it works or whether it will help, but pacjages/mathml/mathml.tex (and indeed *.tex) there seem a place for you to start.

    There is always a denger that some later change has hurt a package written a while back - but if that is the case with all the source files on sourceforge everything ought to be fixable.

    The "embedded" version comes with a reduce.img file that should have ALL the packages in in - so you do not compile any new Reduce for that. To access a package you often need to say
        load_package mathml;
    to the Reduce supervisor to load it, or call the Lisp function (load!-package 'mathml).

    You may wish to spend time playing with the regular version interactively just to see what is liable to be easiest for you to work with.

     
  • Nathan Carter
    Nathan Carter
    2010-08-17

    I have successfully used the QProcess API to start a REDUCE process, send it commands, and get output.  I can even load and use the mathml and mathmlom packages just as you described.  This is looking good so far!

    My problem is that I get all the process's output when it terminates, not after each command I send, as if it were being run interactively.  Several internet sources suggest that this happens because stdout is buffered, and that it can be solved by inserting fflush(stdout) calls after each line of output.

    I started to look in the sources for where to put such calls, but the printing system is quite intricate, not so documented, and spread over several files.  I experimented with a few attempts, but none of them succeeded.  Can you point me to the code that prints the result of each evaluation?

    Of course, it's a little more complicated of an issue than that, because there are any number of errors that may be printed and should also be flushed, but let's start with the result of each evaluation.  (Or is this perhaps a reason to go the RPN route?)

    Nathan

     
  • Arthur Norman
    Arthur Norman
    2010-08-17

    I think that the first thing I would try would be a call to setvbuf on the stream linking Reduce and your code to set it into unbuffered mode? If you are LUCKY you can do that from your end of the pipe!

    The next thing I would do would be to write a trivial C application that reads from stdin and writes to stdout and call that via QProcess to see what buffering you observe.

    I had THOUGH that the code did and ensure_screen() before each new prompt for input, and that goes to fwin_ensure_screen which does a fflush(stdout).

    Yes one of the pains with using a text-mode interface is that of interactive queries to you, error messages and stray "informative" output that you may or may not want….

    Arthur

     
  • Nathan Carter
    Nathan Carter
    2010-08-17

    Let's see if I can use this SourceForge posting interface a little more sophisticatedly this time…

    I think that the first thing I would try would be a call to setvbuf on the stream linking Reduce and your code to set it into unbuffered mode? If you are LUCKY you can do that from your end of the pipe!

    The discussions I've seen online, coupled with the API available to me in my application (on my end of the pipe, as you say) do not seem to have any effect in this regard.  I can also confirm this as follows.

    The next thing I would do would be to write a trivial C application that reads from stdin and writes to stdout and call that via QProcess to see what buffering you observe.

    I wrote the trivial C application you suggested, and it looks like this.

    #include <stdio.h>
    int main ( int argv, char** argc )
    {
        char input[100];
        do {
            scanf( "%s", input );
            printf( "You typed this in: %s\n", input );
    //        fflush( stdout );
        } while ( strcmp( input, "quit" ) );
        return 0;
    }
    

    The results of my experimentation with talking to that application through the QProcess API are consistent with what I read on the internet, as follows.
    1. If the flush line is commented out, I don't get any responses to the inputs sent to the program over the pipe until I send the "quit" command and wait for the process to terminate, at which point the pipe closes and all output comes through.
    2. If the flush line is uncommented, I get all responses to the inputs promptly, like the interactive nature one experiences when running it from the shell.  (This is odd…how does the shell flush the output buffer?  Does printf() somehow know about the shell?)
    3. If the behavior experienced in 1. does not change if I try to turn off buffering on the other end of the pipe.

    I had THOUGH that the code did and ensure_screen() before each new prompt for input, and that goes to fwin_ensure_screen which does a fflush(stdout).

    No, it is not called before each prompt.  I put a printf() debugging call in it, and I experience this behavior in an interactive session:

    $ ./reduce 
    Reduce (Free CSL version), 09-Mar-10 ...
    fwin_ensure_screen() called here
    fwin_ensure_screen() called here
    5+5;
    10
    2*9;
    18
    quit;
    fwin_ensure_screen() called here
    fwin_ensure_screen() called here
    

    Perhaps fixing that to behave the way you thought it should behave will fix this whole problem.  Is that easy to do?  (As mentioned in a previous message, I've poked around a good bit in the code, but it's hard for me as a new pair of eyes to figure out where such a call belongs…sorry!)

    Yes one of the pains with using a text-mode interface is that of interactive queries to you, error messages and stray "informative" output that you may or may not want….

    I don't mind writing code that examines the output and looks for error messages, if and when I can get the input/output channels working correctly.  It would indeed be more elegant to have direct calls to the reduce library, but this will probably be faster to implement, and is a good first start.

    Thank you again, Arthur, for patiently walking through these questions with me.  It's a very big help!

    Nathan

     
  • John ffitch
    John ffitch
    2010-08-18

    Surely rather thanh calling fflush one should set stdout to be line buffered, or not buffered.
    ie calling
    setvbuf(stdout,  NULL, _IOLBF, 0);
    before any use

    Of have I missed something?

     
  • Arthur Norman
    Arthur Norman
    2010-08-18

    If you want to do a setvbuf to make stdin & stdout unbuffered or line buffered in CSL I suggest you do it as about the first thing in "main". That should be as easy a place to spot as any.

    Wrt the flushing - on looking, the "embedded" option misses out lots of stuff to do with user interfaces and potentially system-specific file handling etc etc, and one of the thinge misse dout is prompts!  Hance the ensure_screen/fflush that I mentioned before does not happen. But it looks plausible to me that if you put a fllush(stdout); fflush(stderr) on around line 2248 of cslread.c that will be just before the code tries to read anything.

    That gives you two things you can experiment with there!

    Meanwhile I have been sketching code for a scheme that lets C code build parse trees of expressions and pass them to Reduce and get back results in the same style - and that is garbage collector safe. If I get a first version working in a few days I will upload that. It is after the style of the RPN idea I mentioned earlier, but has had 2 more days of though!

    Arthur

     
  • Nathan Carter
    Nathan Carter
    2010-08-18

    jpff
    Surely rather thanh calling fflush one should set stdout to be line buffered, or not buffered.
    ie calling
    setvbuf(stdout,  NULL, _IOLBF, 0);
    before any use

    arthurcnorman
    If you want to do a setvbuf to make stdin & stdout unbuffered or line buffered in CSL I suggest you do it as about the first thing in "main". That should be as easy a place to spot as any.

    This was one of my initial attempts, gleaned from the internet discussions I mentioned Googling earlier.  However, it doesn't seem to accomplish anything in this case.  When I try putting that line at the start of the simple C program I gave above, it works like a charm, obviating the need for the fflush() line.  But when I try putting it as the first line of main() for reduce (that's at line 2854 of csl.c, which I think is what Arthur was suggesting) it doesn't seem to have any effect.  I do not know why; I grepped for other setvbuf() or setbuf() calls in the reduce sources but didn't find any.

    arthurcnorman
    But it looks plausible to me that if you put a fllush(stdout); fflush(stderr) on around line 2248 of cslread.c that will be just before the code tries to read anything.

    Aha!  This worked!  I'm all set now!  Woohoo!  (insert dance of joy)  Now I can set about integrating REDUCE into Lurch-excellent!

    Thank you again for all of this help; that's exactly what I needed.  Do you want this change in the master reduce svn repo or should I just keep it as a local change in my repo?  It's a one-liner, so I suppose you don't need me specifically to commit it. :)

    arthurcnorman
    Meanwhile I have been sketching code for a scheme that lets C code build parse trees of expressions and pass them to Reduce and get back results in the same style - and that is garbage collector safe. If I get a first version working in a few days I will upload that. It is after the style of the RPN idea I mentioned earlier, but has had 2 more days of though!

    This would be less error-prone than parsing and watching for errors, so I'd be happy to have it, but it's truly up to you since you'd be doing the work.  Do you think other projects that use embedded REDUCE would want it?

    Nathan

     
  • Nathan Carter
    Nathan Carter
    2010-08-18

    PS: What I actually did was just to put in ensure_screen(); at 2248 of cslread.c, before the for(;;) loop.  I don't think you need to fflush(stderr); because I've read that it's usually unbuffered already.

     
  • Arthur Norman
    Arthur Norman
    2010-08-18

    I have just checked some stuff into the main version - see the file proc.h and relevant implementation in csl.c. I have compiled it so I believe it doe snot wreck other things, but have not yet tried it at all. And the code in "embedded" is a snapshot that does not automatically track the main version, so I will need to merge stuff into there.

    It is great that the text-mode interface works. I like the idea of giving you and future projects a choice of approaches - I can not tell which will end up nicest.

    I will try to migrate stuff into the embedded directory and set up a test harness there.

              Arthur

     
  • Arthur Norman
    Arthur Norman
    2010-08-18

    OK - I just updated the "embedded" directory so it should be in sync with the main version. The most obvious change is that the newer version has much more stuff from Reduce compiled into C - so it is bigger! But then the image file may be slightly smaller and the system will go faster. Somebody who needed to could manufacture versions with eithermore or less in teh files u*.c,

    I hope I have not messed anything up in the process, but accidents can happen so please moan if I have. And with subversion you can always check out a previous version…

                 Arthur

     
  • Arthur Norman
    Arthur Norman
    2010-08-19

    If you update from subversion again, then in the embedded directory you may be able to go
         make -f Makefile.proc
    to make an executable called testharness where all the driver code is in driver.c and all the Reduce stuff is elsewhere.
    I KNOW it still has some bugs, but the RPM stuff runs at least one example happily.

    If anybody would care to spot the error I have in there at present (beyond the fact that at preent I do not deal with stray IO properly - and I may need to alter the API to cope…) that would be jolly helpful.

    If you just look at one file please look at driver.c and concentrate on the PROC_xxx stuff rather than the old-style use of execute_lisp_function…

              Arthur

     
  • Nathan Carter
    Nathan Carter
    2010-08-19

    I'll be more than glad to look it over.  But currently the driver.c file is missing from the embedded subdirectory.

    Even so, if this keeps going well, I may join your fan club.

    Nathan

     
  • Arthur Norman
    Arthur Norman
    2010-08-20

    Apologies. I had not gone "svn add". I hope nothing else is missing. Overnight I thought what my glitch was, and now the code in driver.c is set up to construct
       (int (quotient  1 (difference (expt x 6) 1)) x)
    and evaluate it, and then display the prefix form of the result.

    What is WRONG at present is that trace output (and error messages if present) go to stdout. I will (in a while) chance that so they go via the callback procedures. That should not be hard.

    There will be other facilities and features needed to interface to Reduce. Eg things to set and clear switches and something to get a package loaded. Again those will not be hard to do and I will put in what comes to my mind! I enjoy projects like this where there is actually not a lot of code involved but where the leverage seems good.

            Arthur