I have two related questions, or maybe one of them will turn out to be a feature request.
I'm using makepp to make huge files. In order to save disk space, instead of creating temporary files, I'd like to create temporary pipes. This means that to create file1 I need to create pipe1, pipe2 and pipe3. When file1 is finished building, I'd like to remove the pipes. That is the only mechanism I know for makepp to know if behind a pipe there's a command waiting to spew out data. I.e. it creates the pipe, runs the command to put data into it, and then after everything is done reoves the pipe. Or maybe there's a better way?
The second question is for a similar reason, disk space. Say file1 creates temporary file (i.e. in this case not pipes) temp1, and from temp1 I create result1, result2 and result3.
Now, what I'd like to have is that after result1 and 2 and 3 are created from temp1, temp1 is removed.
But, I'd also like to be able to say that say result3 also depends on temp1, but since I haven't yet specified how to make result3 the file temp1 shouldn't yet be removed. I.e. in this case making result1 and result2 is not enough to remove temp1.
The reason is that I'm debugging making result1 and result2, but haven't yet worked on 3. Once everything works, I'd like the temporary files to be automatically removed, but not before.
Are these possible?
Thanks!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I suppose you mean named pipes (fifos)? At least on Linux their time stamp changes when they are used, so with a rule option :build_check target_newer you should be fine. Not sure why you'd delete the fifos…
As for your second, there is $(temporary file) which declares file as disappearing. But this is only for within a rule, if the lexer or command parser detects that the rule creates some file (implicit target), but not that it will delete it before finishing. Otoh, inter-rule dependencies must remain, at least if you want to run mpp again on the same build tree. This is necessary for guaranteed correct builds, because otherwise mpp can't check that some target is still up to date, leading to rebuilds.
regards - Daniel
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
And I wanted to delete the pipes so that I always know that if there is a pipe, then there must be a command waiting to push data in it. (Otherwise I might wait forever for a command that never starts)
Oh, but I think I understand what you say. If the pipe is older than the files that create it, then the command will be run…. hmmm the problem is that a pipe can be read just once, so once I need it again, I need to 'make it old' so it is run again.
Anyway, this whole scheme didn't work in the end, because the pipes are non-blocking, so command1 finds nothing and exits. For now I use an external bash script with
command1 <(command2) <(command3) >result
Though because of this I forgo some of the coolness of makepp.
And if someone is already listening… (I hope..)
I had problems with giving the command "join" a separator 'tab' inside a makefile for makepp.
It is hard enough from bash. I use
join -t $'\t' ....
But from inside my makefile I just couldn't make it work.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
ends up depending on itself. It gives an error: "warning: `pipe1' depends on itself; circular dependency removed."
I tried all sorts of tricks to make this disappear, but couldn't. Actually, pipes are strange.
Usually
command >file
Creates "file", so doesn't automatically depend on it. But
command >pipe
Puts things in "pipe", so it needs "pipe" to be there, i.e. it depends on "pipe".
A second problem I'm having is about what happens when a second rule depends on the pipe:
target2: pipe1com1
command4 pipe1>target2
I depend again on pipe1com1. This is supposed to run command1 into the pipe1. But for the current makepp run pipe1com1, though phony, was already created for taget1, so it doesn't need to be recreated. However because it is a pipe, once taget1 is created, the contents of the pipe is already gone, so that target needs to be rebuilt. Again, I tried various tricks, but because of my relative newbieness, couldn't find how to accomplish this.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Wow, you made me find a bug in mpp, where it is too hard on shell syntax it doesn't grok. Can you try this patch:
--- Mpp/Lexer.pm.~1.47.~ 2011-08-06 13:48:32.000000000 +0200+++ Mpp/Lexer.pm 2011-12-07 23:07:22.834383437 +0100@@ -193,7 +193,9 @@
$command =~ s/\d$//; # strip off "2" in "2> file"
next if $expr =~ /^\&/; # Ignore ">&2" etc.
$expr =~ s/^\s*//;
- my $file = unquote +(split_on_whitespace $expr)[0] or+ my( $file ) = split_on_whitespace $expr;+ next unless defined $file; # something weird like bash <(cmd) syntax?+ $file = unquote $file or
die "Undefined redirection in rule " . $rule->source . "\n";
next if $file eq '/dev/null' or $file =~ m|[^-+=@%^\w:,./]|;
if( $is_in ) {
This will allow you to have bash -c 'command1 <(command2) <(command3) >result' as command action.
The tab problem is the same as above, setting SHELL is very limited and outside Windows doesn't work (not that you said you tried it). And here you have the extra problem that a single $ is makepp's. If you want to pass it to a command, you must double it, so this echoes a tab: bash -c "echo a$$'\t'b" >out
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Wow, you crossed my lines there. Actually I didn't reply about your explicit pipes. They certainly are blocking but have the funny thing that the reader must be there before the writer(s). Backgrounding actions is not a good idea, because mpp can't catch their final return code, and might end before some of its actions end.
$(phony pipe1com1): pipe1
command1 >pipe1 &
depends on itself, because mpp recognizes the >.
Hiding real things behind phony is wrong. Certainly rules get run only once. If they do what they declare, there's no reason otherwise. Anyway I showed in my previous post how to get your original intention working (albeit with an ugly bash -c), so all these problems should be irrelevant now.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Makepp uses command parsers to glean extra information. This involves things like compiler's -Iinc, or -llib options. Alas it is unthinkable to have parsers for the 1000s of commands out there, so mpp has no clue that the argument to zcat is a filename.
Otoh, >target is recognized, because one step earlier the lexer understands the Shell syntax. And it knows that someshell -c '…' contains a nested command. For the thing I patched yesterday, it would require an Mpp::Lexer::Bash to understand all the extra goodies. For now it ignores the one you brought up (done right it would recognize that <(cmd) contains a command and recursively try to parse that).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Well, if you go to all these lengths, you might as well just put the inputs after the first colon…
As for parsing the built in commands, you have a point there. The sad part is that they have a rather rich option syntax. So getting everything right would mean doing a good part of what the command will really do later. Since a long time I haven't come to a final conclusion about whether it's worth it.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
I have two related questions, or maybe one of them will turn out to be a feature request.
I'm using makepp to make huge files. In order to save disk space, instead of creating temporary files, I'd like to create temporary pipes. This means that to create file1 I need to create pipe1, pipe2 and pipe3. When file1 is finished building, I'd like to remove the pipes. That is the only mechanism I know for makepp to know if behind a pipe there's a command waiting to spew out data. I.e. it creates the pipe, runs the command to put data into it, and then after everything is done reoves the pipe. Or maybe there's a better way?
The second question is for a similar reason, disk space. Say file1 creates temporary file (i.e. in this case not pipes) temp1, and from temp1 I create result1, result2 and result3.
Now, what I'd like to have is that after result1 and 2 and 3 are created from temp1, temp1 is removed.
But, I'd also like to be able to say that say result3 also depends on temp1, but since I haven't yet specified how to make result3 the file temp1 shouldn't yet be removed. I.e. in this case making result1 and result2 is not enough to remove temp1.
The reason is that I'm debugging making result1 and result2, but haven't yet worked on 3. Once everything works, I'd like the temporary files to be automatically removed, but not before.
Are these possible?
Thanks!
Hi gently,
I suppose you mean named pipes (fifos)? At least on Linux their time stamp changes when they are used, so with a rule option :build_check target_newer you should be fine. Not sure why you'd delete the fifos…
As for your second, there is $(temporary file) which declares file as disappearing. But this is only for within a rule, if the lexer or command parser detects that the rule creates some file (implicit target), but not that it will delete it before finishing. Otoh, inter-rule dependencies must remain, at least if you want to run mpp again on the same build tree. This is necessary for guaranteed correct builds, because otherwise mpp can't check that some target is still up to date, leading to rebuilds.
regards - Daniel
Thank you for the reply!
I am just trying to work with pipes in a way that looks nice and works.
What I'm trying to recreate is bash's feature:
By which bash easily allows one to replace files as input for commands with a pipe from a second command.
I thought I'd do
And I wanted to delete the pipes so that I always know that if there is a pipe, then there must be a command waiting to push data in it. (Otherwise I might wait forever for a command that never starts)
Oh, but I think I understand what you say. If the pipe is older than the files that create it, then the command will be run…. hmmm the problem is that a pipe can be read just once, so once I need it again, I need to 'make it old' so it is run again.
Anyway, this whole scheme didn't work in the end, because the pipes are non-blocking, so command1 finds nothing and exits. For now I use an external bash script with
Though because of this I forgo some of the coolness of makepp.
And if someone is already listening… (I hope..)
I had problems with giving the command "join" a separator 'tab' inside a makefile for makepp.
It is hard enough from bash. I use
But from inside my makefile I just couldn't make it work.
OK, I partially solved my problem.
Actually, I think what I described in my previous post was some error I had. mkfifo pipes work.
So I can do:
One problem that I have is that
ends up depending on itself. It gives an error: "warning: `pipe1' depends on itself; circular dependency removed."
I tried all sorts of tricks to make this disappear, but couldn't. Actually, pipes are strange.
Usually
Creates "file", so doesn't automatically depend on it. But
Puts things in "pipe", so it needs "pipe" to be there, i.e. it depends on "pipe".
A second problem I'm having is about what happens when a second rule depends on the pipe:
I depend again on pipe1com1. This is supposed to run command1 into the pipe1. But for the current makepp run pipe1com1, though phony, was already created for taget1, so it doesn't need to be recreated. However because it is a pipe, once taget1 is created, the contents of the pipe is already gone, so that target needs to be rebuilt. Again, I tried various tricks, but because of my relative newbieness, couldn't find how to accomplish this.
Wow, you made me find a bug in mpp, where it is too hard on shell syntax it doesn't grok. Can you try this patch:
This will allow you to have bash -c 'command1 <(command2) <(command3) >result' as command action.
The tab problem is the same as above, setting SHELL is very limited and outside Windows doesn't work (not that you said you tried it). And here you have the extra problem that a single $ is makepp's. If you want to pass it to a command, you must double it, so this echoes a tab: bash -c "echo a$$'\t'b" >out
Wow, you crossed my lines there. Actually I didn't reply about your explicit pipes. They certainly are blocking but have the funny thing that the reader must be there before the writer(s). Backgrounding actions is not a good idea, because mpp can't catch their final return code, and might end before some of its actions end.
depends on itself, because mpp recognizes the >.
Hiding real things behind phony is wrong. Certainly rules get run only once. If they do what they declare, there's no reason otherwise. Anyway I showed in my previous post how to get your original intention working (albeit with an ugly bash -c), so all these problems should be irrelevant now.
Cool! Thanks! This works.
And it even parses through the bash command!! That is amazing! Now my makefile is basically
So nice!
Hmmm…. not entirely related, but…
In my reply above, files used within COM1 were nicely recognized as dependencies. How come the same doesn't happen for the command "zcat"? I.e.
How come makepp doesn't recognize that target depends on file.gz?
Makepp uses command parsers to glean extra information. This involves things like compiler's -Iinc, or -llib options. Alas it is unthinkable to have parsers for the 1000s of commands out there, so mpp has no clue that the argument to zcat is a filename.
Otoh, >target is recognized, because one step earlier the lexer understands the Shell syntax. And it knows that someshell -c '…' contains a nested command. For the thing I patched yesterday, it would require an Mpp::Lexer::Bash to understand all the extra goodies. For now it ignores the one you brought up (done right it would recognize that <(cmd) contains a command and recursively try to parse that).
Oh, I see. But I can use
This works very nicely.
BTW: it is a bit strange that
works (i.e. recognizes that target depends on file), while
doesn't….
Sorry, I meant:
(Sorry for my many replies….)
Maybe this is somewhat backwards, but I guess you can tell the dependency of any command on file arguments by using
Is there an easier way to do this? Other than, adding it to the dependeny line manually…
Bah! Again a mistake… I meant
Well, if you go to all these lengths, you might as well just put the inputs after the first colon…
As for parsing the built in commands, you have a point there. The sad part is that they have a rather rich option syntax. So getting everything right would mean doing a good part of what the command will really do later. Since a long time I haven't come to a final conclusion about whether it's worth it.