Menu
▾
▴
Re: [netrexx-pipelines] Migrate OS/2 pipelines to NetRexx Pipes
|
From: René J. <rvj...@xs...> - 2022-06-16 09:37:57
|
Hi Mike,
to know what would be a good description of njp I would need the structure of the required documentation. That would maybe be a welcome addition to the document we have now.
Let me try to make a start:
NetRexx Pipelines are not interpreted; the implementation predates the NetRexx interpreter. Because the NetRexx translator, before version 2.00, only contained a compiler, the sourcefile that contains the pipe specification will be processed by a program called org.netrexx.njpipes.pipes.compiler. (Used to have another name but I do not have reason to include that info now- but that might pop up).
We call this program the 'pipes compiler’. It rewrites your pipeline specification to a program that is composed of a number of threads and a synchronisation mechanism. These threads call the precompiled stages (from the org.netrexx.njpipes.stages package) and your own precompiled stages. The ‘command’ stage can start OS native commands and put their output into the pipe.
This program source will now be passed on to the NetRexx compiler (we call the translator now a compiler, because we could also pass the source to the interpreter part of the translator, to be executed. These two, compilerr and translator, share most of the code, but are different things for different purposes). The compiler translates the generated pipeline program (NetRexx) to Java source code, and calls the Java compiler. The Java compiler is a compiler Java class - we call that, and not the platform executable, javac or javac.exe. We can also call an alternative Java compiler, the Eclipse Batch compiler, of which a modified version is delivered in the NetRexxC.jar archive (and not in the NetRexxC.jar archive, which you can use if you do not need the alternative compiler. We do this because a lot of companies provide systems to their workers that have a Java runtime and not a javac compiler.)
The NetRexx translator was always made to support the human factors of programming and it was not designed to be a speed devil. Somewhere in the 2010’s - I think it was 2014 - the translator was modified (by Kermit Kiser) to keep all source and compiled Java classes in memory. Because of 64 bit Java and platform technology memory shortage ceased to be a problem. So now the NetRexx translator did not use files for its intermediate products anymore, the process became faster by avoiding disk I/O. Later developments as SSD disk diminished that speed gap, although the overall speed increased.
At this point it became possible to offer a VM-like experience with typing the pipeline at a Ready prompt and having it executed immediately. This M1 mac I am typing this email on has a turnaround of 0.102 s for a short pipe in nrws. The NetRexx workspace came about when I tried to make a small REPL for this purpose, but while researching I found Martin Lafaix’ workspace from 20 years ago. This is very usable for development of pipelines where you up-arrow a lot and change the stage parameters (why count columns for a SPEC stage when you can just binary search until you find the right number.)
Still, the original pipes compiler yields the best performance (around 0.05 s for execution of the compiled classfile) - because three compiles (and the writing of the .class file) are avoided..
To compile an .njp file to a class
The compiler is invoked with the command:
pipc <arguments>
Where <arguments> are:
<compopt>|<fileid> {<compopt>|<fileid>} - compile the files listed
( <name> { <compopt>} ) <pipe> - compile the <pipe> name
Where <compopt> is:
-gen - keep the .nrx source of pipes following
-nogen - delete the .nrx source. This is the default
-<option> - where <option> is a valid NetRexx option
And <fileid> is:
<name>.nrx - any NetRexx file
<name>.njp - any pipe or NetRexx or Java souce using pipes
<name>.grp - a file containing a list of <compopt>s and/or
<fileid>s. lines can be commented using '--'
The <options> are:
runaway <milliseconds> - Interval to use for deadlock checking. This
or only detects true pipe deadlocks, not stages
stall <milliseconds> that are looping or hung. (Should this be
changed?)
package <name> - insert a NetRexx package statement
import <name> - insert a NetRexx import statement
config <name> - override the default config file name (see below)
sep <char> - override the default stage separator '!'
end <char> - override the default end of pipe separator '?'
cont <char> - override the default line continuation character ','
debug <flag> - debugging options
When the pipe is contained in a file the options must be set there.
When a pipe is read from a file, text on the same line following a '-- ',
will be treated as a comment.
Errors issued by the compiler.
Stages can issue additional messages.
Error - name of pipe is missing
Error - runaway time must be numeric
Error - stall time must be numeric
Error - debug level must be numeric, found 'lvl'
Error - cache must be followed by a valid symbol, found 'work'
Error - append or prefix stages cannot be labeled 'stg.word(1)'
Error - missing stage/pipe after 'stg.word(1)'
Error - missing range and/or stage/pipe after 'stg.word(1)'
Error - specs has only one output stream, not requires two
Error - pipe definition 'stg' must be terminated by 'pend'
Error - pipe definition 'stg' must define a pipe
Error - Label 'label' already used
Error - Label 'label' must not be numeric
Error - pipe 'stg' cannot have a label
Error - connectors must be named
Error - Label 'label' not defined
Error - Use a nop stage between 'label' definition a second use
Error - expected a colon after 'stg.word(1)'
Error - pipe as stage definition 'stg' is missing a period
Error - 'parms' incorrect. Use: {({class} {size})} {A|D} {target}
Error - need to use a nop stage between 's[c[i,j-1]]' 'ssep' 's[c[i,j]]'
Error - problem reading group rc='r'
Error - 'w' is unreconized
Warning - could not delete 'fileid'
Error - 'key' is only valid in a class
Warning - Possible netrexx exit in 'arg()' at 'l'
Error - pipe name and parms must be on same line as 'key'
Error - Pipe name missing for 'key'
Error - Body of 'wp' is empty
Error - *: connectors not implemented. Use *in: or *out:
Error - Connector 'w1' should start with *in or *out
Error - Missing colon at 'w1'
Error - Connect 'w1' cannot contain a period
Error - cannot connect 'in' to an input stream with 'key'
Error - Pipe fragment 'sub' needs atleast one 'sep'
Error - cannot connect 'out' to an output stream with 'key'
Error - A object name cannot contain spaces, found: 'a'
Debugging pipes
To find out what is happening in pipeline you have three tools. First,
you can set a debug flag when you compile the pipe. The bits you set
in the flag control what it does:
1 - Show all pipes starting
2 - Show all pipes ending
4 - Show all stages starting
8 - Show all stages stopping
16 - Show all Commit requests
32 - Show all Commit completions
64 - Show StageErrors raised via stage's Error(int,String) method. The
stage class uses Error for all its StageError signals.
128 - Show the arguement that each stage is recieving. Handy since
shells have a habbit of doing unexpected thing to arguements.
(try: java findtext exit *.nrx vs java findtext "exit *.nrx")
- If there is a need I can easily add more events - just ask.
To create a flag to see all stages starting and stopping you would
add 8+4 and use:
pipc (apipe debug 12) ...
The second option is to use the invoke the dump() method in a stage.
This dumps the status of the pipe using the same format you see when
a pipe deadlocks <file:///Users/rvjansen/Library/Mobile%20Documents/com~apple~CloudDocs/Documents/Programming_languages/Pipes/njpipes.webarchive#dead>. Using dump() does not normally cause a pipe to
terminate. Once in a while dump() will generate an exception. This
happens since dump() does not use protect or syncronize so it does not
stall.
When all else fails, you can recompile all the stages used by the pipe
with trace results enabled.
There is a category of problems that is caused by the different ways of processing the pipe - most of them are very new, like using address in NetRexx. Some of them are older, because to enable calling from a commandline, a synthetic name will be generated. This is caused by the fact that you can call an unnamed pipe from the commandline in VM - if that were disallowed, we would not have this problem now. For the moment, I can only say that all .njp examples in the examples/pipes directory work as specified.
Stages have their own error messages - I must compile a pipe to list those all to the documentation.
Please let me know what more you need for documentation.
best regards,
René.
> On 16 Jun 2022, at 08:18, hp...@we... wrote:
>
> Hi René!
>
> Am 15.06.2022 um 08:42 schrieb René Jansen:
>> The way to make it faster would be to precompile the pipe to a java .class file using pipe and an .njp file, and instead of addressing pipe, to address java yourpipe.class. Of course, when called from a NetRexx program, you would have the java context already.
>
> I tried it, but found no documentation or at least a description
> about coding njp. NetRexx 4.03 Pipelines Guide and Reference shows
> some examples, but no detailed explanation. I prefer know-why, no
> programming by trial and error.
>
> At the moment I'm stuck with:
>
>> "Error - pipe as stage definition VILMA_1.CLASS is missing a period"
>
> Sorry, if I am not able to solve this with the manual and have to
> ask for help.
>
>> But to show the difference:
>> [...]
>> So compiling it speeds it up around 100 times, I expect the same thing to happen with your pipeline.
>
> I hope so, otherwise it's not worth the drudgery.
>
>> [...]
>>
>> There is one thing I am fixing at the moment, and that is an issue that Leslie Turriff reported first: with a portrait pipeline, the pipe separators must be at the end of the line for the pipe compiler (for some reason), and he (and I) like them at the start of the line.
>
> Same with me, up to now. But maybe I get used to something new ;)
>
> Best,
> M.
>
>
> _______________________________________________
> netrexx-pipelines mailing list
> net...@li...
> https://lists.sourceforge.net/lists/listinfo/netrexx-pipelines
|
