I've run into a couple of problems using Kelp (0.5 + latest checkout). I'd really just like to know if:
a) They're not fully implemented yet/they're broken and it's known about.
b) I'm misusing them
c) There's something wrong in my compilation, but it's not effecting anyone else.
1. kelp _module @id
kelp _module @id #index
Don't appear return anything? (kelp _module and kelp @id work fine)
2. Long comments in kelp @id
Cuts short output if blade exceeds handlable length. If it has to be trimmed or fragmented, so be it, but it needs to return an EOF so that that stream is recognised as finished.
eg. 0.5 release version of kelp_file_format.kelp @format
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
On item 1, I mentioned to Filippo Sunday that the vim plugin did not produce results for that combination and that I was not too worried because the command line invocation failed too. He mentioned that it was something he was working on.
I was poking at the problem last night when I veered off into the structure padding weeds. I should have something figured out tonight.
Dave
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
On item two, what length of text is causing problems?
I ran tests with lots of short lines, several very long lines, one very long line (17K). And several very, very long lines totaling 300K. They all seemed to print completely.
This was using 0.5 compiled with mingw32.
Thanks,
Dave
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Looking at the print_blade() code in kelp.c, it is writing to stdout. It could be that VC++ is treating the stream differently (as text instead of binary?).
I think a fflush(stdout) at the end of the routine should probably correct it. Try this:
Still doesn't work on 0.5, but adding fflush(stdout) to the latest checkout version (for loop has already been changed to something different) did fix it in the prompt.
Still makes the plug-in hang though - investigating.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've been working exclusively with source checked out of CVS, not the released source.
I need to be clearer about what code I'm compiling when we're discussing defects. Otherwise, we could end up talking past each other.
Dave
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2007-09-05
Hi Ian,
RE:
a)
That's correct, I have not yet fully implemented all the options.
And up until (and including) 0.5, the text printing was not 100% correct. I fixed text printing since KELP_0_6.
b)
You are not necessarily misusing them, although checking in your code now may help us identify the problem too.
I am not flushing stdout, which is something I will do.
I could also send an EOF at the end of the stream (although that should be detected by the process ending).
I am going to write a test client that will test this outside of any environment (i.e. not a plugin) and see what I get.
For now, you will have to bear with me until I can fix it properly. Won't be long, I promise.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Could it be that because you are reading from stderr first, you are effectively waiting until there is an EOF from stderr (which happens when the process terminates) and therefore, when moving on to the stdout, it has no content?
Try not reading from stderr and see what happens.
That is, after you add the fflush(stdout) to my code.
fflush is needed because you are redirecting stdout from a terminal to a pipe, and that usually means stdout becomes "block buffered" rather than "line buffered": on the console, every line gets printed on screen whenever a new-line character is encountered.
The bottom line is that kelp.exe was not designed to be used in plugins really.
I am going to implement an interactive version, ikelp.exe, which will be used in plugins.
But nonetheless, kelp.exe should behave properly.
It is just that a non-interactive batch program, a filter as it is known in unix, should not write errors to stdout, but only to stderr. This is to prevent error output to be passed down the filter chain.
An interactive program instead is supposed to be understood in a more sophisticated way by the controller process, and all its output, including errors, go to stdout. Only log messages go to stderr.
I will get this sorted ASAP but you will have to bear with me while I get through some heavy sessions at work before the weekend.
Cheers,
Filippo
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Apparently so. Swapping them does indeed resolve the problem.
I had a reason for using them that way round, can't quite recall..
Anyway, as you've obviously noticed, my code's been on CVS the last couple of days. It works, just doing some minor fiddling. Feedback always appreciated.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2007-09-05
Excellent.
Could you try now removing the fflush(stdout) from kelp.exe and check that it was only the order in which you read from stdout/stderr that was highlighting the issue?
I still think there is an issue which I will address as soon as I have some time.
Cheers,
Filippo :-)
P.S.
Thank you very much for the plugin!
I will post news on the site about it!
BTW, did you check the statistics? we are now in the top 50 projects by activity in the latest 7 days.
Well done guys! :-)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've just re-downloaded a clean 0.5 source release to test this.
Due to the change in the plugin (processing the input stream before the error stream) the plugin no long hangs, BUT the text is still clipped.
This isn't a plugin issue - I get the same results when calling Kelp from cmd.exe
The problem still occurs in the 0.5 version, even after adding fflush(stdout). I don't seem to be able to compile it with Daves other change now for some reason.
The problem doesn't occur in the latest checkout version, even after removing fflush(stdout).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2007-09-05
Hi Ian,
I think it is best to use KELP_0_6 from now on.
I will release KELP_0_6 tonight once I get back from work and I have put my son Daniel to bed.
There was a bug in KELP_REL_0_5 related to text file output. If you know the bug is there, you know what is happening.
But i must admit that without knowing the bug was there, the output looks crippled.
The bug was submitted by me on the bug tracker: [ 1785553 ] Incorrect blade information printed
against 0.5
It has been fixed from 0.6 onwards.
It is all to do with the way "text files" are handled in Windows.
In Unix, a file is a file, there don't exist any difference, programmatically, between a binary and a text file.
In Windows instead (thank you BillG!) text files are files that have the combination \r\n at the end of a line on disk, but when read in a program, only present the fmailiar \n at the end of a line (the \r gets swallowed by the system routines).
This affects some system routines (fseek, ftell, fread, fwrite) because teh number of bytes your read from a file does not coincide with the number of bytes in the actual file.
Since most of my text file reading days were spent in the unix world, I had to find a way to deal with it without having "conditionals" around the code, and make it as portable as possible.
I found a solution, and I applied it to kelp in KELP_0_6.
I think I will release 0.6 tonight so that any new downloads will get the fixed code! :-)
Thanks again,
Filippo
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've been watching the stats periodically (obsessively) over the past 3 days. I need a hobby....
Thanks for the encouragement,
Dave
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2007-09-05
I added the fflush on stdout in CVS.
Although I didn't think that being necessary: stdout is flushed as part of the C exit routines.
I definitely need to write a simple "plugin emulator", that grabs the outputs of kelp.exe and writes them to disk, so I can assess what is happening to the streams.
Stay tuned! :-)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2007-09-05
Ian,
I think this method in ProcessBuilder will help solve the issue of how to read two streams concurrently, without involving synchronisation:
public ProcessBuilder redirectErrorStream(boolean redirectErrorStream);
This will effectively merge the two streams. All you have to do is parse the first line of the reply, or effectively, you could just print whatever is output by kelp.
If you can then test and check your code in, I can then help you tag your plugin ready for the next release (0.8), together with the vim plugin.
Cheers,
Filippo
P.S.
Please check the new topic on this forum about the change in kelp's project aim.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes, I've already implemented redirectErrorStream, I just haven't committed yet, as I was in the process of adding something else, which is almost finished.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've run into a couple of problems using Kelp (0.5 + latest checkout). I'd really just like to know if:
a) They're not fully implemented yet/they're broken and it's known about.
b) I'm misusing them
c) There's something wrong in my compilation, but it's not effecting anyone else.
1. kelp _module @id
kelp _module @id #index
Don't appear return anything? (kelp _module and kelp @id work fine)
2. Long comments in kelp @id
Cuts short output if blade exceeds handlable length. If it has to be trimmed or fragmented, so be it, but it needs to return an EOF so that that stream is recognised as finished.
eg. 0.5 release version of kelp_file_format.kelp @format
On item 1, I mentioned to Filippo Sunday that the vim plugin did not produce results for that combination and that I was not too worried because the command line invocation failed too. He mentioned that it was something he was working on.
I was poking at the problem last night when I veered off into the structure padding weeds. I should have something figured out tonight.
Dave
Again on item 1. The answer is a) Not implemented yet. The find_blade() routine in kelp.c looks like this:
int find_blade(void)
{
if (!arg_id[0] && !arg_module[0] && !arg_index) return list_modules();
if (!arg_id[0] && arg_module[0] && !arg_index) return list_ids_in_module();
if (arg_id[0] && !arg_module[0] && !arg_index) return print_id_in_forest(1);
return 0;
}
It ignores the condition where both the module and the id are non-blank.
Dave
On item two, what length of text is causing problems?
I ran tests with lots of short lines, several very long lines, one very long line (17K). And several very, very long lines totaling 300K. They all seemed to print completely.
This was using 0.5 compiled with mingw32.
Thanks,
Dave
If you've got the 0.5 .kelp files to, kelp @format should cut short (at least mine does).
I'm compiled via VC++ 2005 command line.
No, I get the full text.
Looking at the print_blade() code in kelp.c, it is writing to stdout. It could be that VC++ is treating the stream differently (as text instead of binary?).
I think a fflush(stdout) at the end of the routine should probably correct it. Try this:
int print_blade(FILE *kelp, char *buf, struct kelp_blade *b)
{
...
for (bladesz = b->size; bladesz > 0; bladesz -= count)
{
...
}
fflush(stdout);
return 0;
}
Dave
Still doesn't work on 0.5, but adding fflush(stdout) to the latest checkout version (for loop has already been changed to something different) did fix it in the prompt.
Still makes the plug-in hang though - investigating.
I've been working exclusively with source checked out of CVS, not the released source.
I need to be clearer about what code I'm compiling when we're discussing defects. Otherwise, we could end up talking past each other.
Dave
Hi Ian,
RE:
a)
That's correct, I have not yet fully implemented all the options.
And up until (and including) 0.5, the text printing was not 100% correct. I fixed text printing since KELP_0_6.
b)
You are not necessarily misusing them, although checking in your code now may help us identify the problem too.
I am not flushing stdout, which is something I will do.
I could also send an EOF at the end of the stream (although that should be detected by the process ending).
I am going to write a test client that will test this outside of any environment (i.e. not a plugin) and see what I get.
For now, you will have to bear with me until I can fix it properly. Won't be long, I promise.
I'm in no hurry. I can easily get on without it by simply not running over any long blades, but I thought I should highlight the odd behaviour.
Ian,
I also noticed, in your NativeInterface.java, you have:
57 BufferedReader bufErr = new BufferedReader(new InputStreamReader(proc.getErrorStream()));
58 BufferedReader bufStd = new BufferedReader(new InputStreamReader(proc.getInputStream()));
59
60 String line = null;
61
62
63 //Errors
64 while ((line = bufErr.readLine()) != null) {
65 output.replaceTextRange(0, 0,"> ERROR (Kelp): " + line + "\n");
66 output.setStyleRange(new StyleRange(0, 14, null, null, SWT.BOLD));
67 }
68
69 //Output
70 String temp = output.getText();
71 output.setText("");
72 while ((line = bufStd.readLine()) != null) {
73 output.append(line + "\n");
74 System.out.println(line);
75 }
76 output.append(temp);
77
78 exitVal = proc.waitFor();
79 //TODO Timeout required on proc.waitFor to allow for failure of application to send EOF.
80
81 } catch (Throwable t) {
82 output.replaceTextRange(0, 0,"> ERROR (Internal): " + t.getMessage() + "\n");
83 }
84
Could it be that because you are reading from stderr first, you are effectively waiting until there is an EOF from stderr (which happens when the process terminates) and therefore, when moving on to the stdout, it has no content?
Try not reading from stderr and see what happens.
That is, after you add the fflush(stdout) to my code.
fflush is needed because you are redirecting stdout from a terminal to a pipe, and that usually means stdout becomes "block buffered" rather than "line buffered": on the console, every line gets printed on screen whenever a new-line character is encountered.
The bottom line is that kelp.exe was not designed to be used in plugins really.
I am going to implement an interactive version, ikelp.exe, which will be used in plugins.
But nonetheless, kelp.exe should behave properly.
It is just that a non-interactive batch program, a filter as it is known in unix, should not write errors to stdout, but only to stderr. This is to prevent error output to be passed down the filter chain.
An interactive program instead is supposed to be understood in a more sophisticated way by the controller process, and all its output, including errors, go to stdout. Only log messages go to stderr.
I will get this sorted ASAP but you will have to bear with me while I get through some heavy sessions at work before the weekend.
Cheers,
Filippo
Apparently so. Swapping them does indeed resolve the problem.
I had a reason for using them that way round, can't quite recall..
Anyway, as you've obviously noticed, my code's been on CVS the last couple of days. It works, just doing some minor fiddling. Feedback always appreciated.
Excellent.
Could you try now removing the fflush(stdout) from kelp.exe and check that it was only the order in which you read from stdout/stderr that was highlighting the issue?
I still think there is an issue which I will address as soon as I have some time.
Cheers,
Filippo :-)
P.S.
Thank you very much for the plugin!
I will post news on the site about it!
BTW, did you check the statistics? we are now in the top 50 projects by activity in the latest 7 days.
Well done guys! :-)
I've just re-downloaded a clean 0.5 source release to test this.
Due to the change in the plugin (processing the input stream before the error stream) the plugin no long hangs, BUT the text is still clipped.
This isn't a plugin issue - I get the same results when calling Kelp from cmd.exe
The problem still occurs in the 0.5 version, even after adding fflush(stdout). I don't seem to be able to compile it with Daves other change now for some reason.
The problem doesn't occur in the latest checkout version, even after removing fflush(stdout).
Hi Ian,
I think it is best to use KELP_0_6 from now on.
I will release KELP_0_6 tonight once I get back from work and I have put my son Daniel to bed.
There was a bug in KELP_REL_0_5 related to text file output. If you know the bug is there, you know what is happening.
But i must admit that without knowing the bug was there, the output looks crippled.
The bug was submitted by me on the bug tracker: [ 1785553 ] Incorrect blade information printed
against 0.5
It has been fixed from 0.6 onwards.
It is all to do with the way "text files" are handled in Windows.
In Unix, a file is a file, there don't exist any difference, programmatically, between a binary and a text file.
In Windows instead (thank you BillG!) text files are files that have the combination \r\n at the end of a line on disk, but when read in a program, only present the fmailiar \n at the end of a line (the \r gets swallowed by the system routines).
This affects some system routines (fseek, ftell, fread, fwrite) because teh number of bytes your read from a file does not coincide with the number of bytes in the actual file.
Since most of my text file reading days were spent in the unix world, I had to find a way to deal with it without having "conditionals" around the code, and make it as portable as possible.
I found a solution, and I applied it to kelp in KELP_0_6.
I think I will release 0.6 tonight so that any new downloads will get the fixed code! :-)
Thanks again,
Filippo
I've been watching the stats periodically (obsessively) over the past 3 days. I need a hobby....
Thanks for the encouragement,
Dave
I added the fflush on stdout in CVS.
Although I didn't think that being necessary: stdout is flushed as part of the C exit routines.
I definitely need to write a simple "plugin emulator", that grabs the outputs of kelp.exe and writes them to disk, so I can assess what is happening to the streams.
Stay tuned! :-)
Ian,
I think this method in ProcessBuilder will help solve the issue of how to read two streams concurrently, without involving synchronisation:
public ProcessBuilder redirectErrorStream(boolean redirectErrorStream);
This will effectively merge the two streams. All you have to do is parse the first line of the reply, or effectively, you could just print whatever is output by kelp.
If you can then test and check your code in, I can then help you tag your plugin ready for the next release (0.8), together with the vim plugin.
Cheers,
Filippo
P.S.
Please check the new topic on this forum about the change in kelp's project aim.
Yes, I've already implemented redirectErrorStream, I just haven't committed yet, as I was in the process of adding something else, which is almost finished.