htdig-dev

[htdig-dev] truncated output from htsearch...a strace

[Aaron B. <ab...@mi...>]

I posted this as a question to the users list but have more detailed 
info that suggests this may be a bug with the system libraries...

These tests were ran using v3.1.6.

If i run htsearch from the command line and allow the output to go to 
the screen the results appear complete and proper.  If i run htsearch 
from the command line and redirect the output to a file via ">/tmp/pg" 
or from a browser via apache then the results will be truncated.  I have 
ran strace against htsearch with output to the screen and to a file and 
there are clearly some missing write()'s.

On the strace with output redirected to a file the write()'s are 
clearly much less even though the read()'s from the templates are the same.

Can someone with a little more intimate knowledge of htsearch help out 
and see what the problem might be?

Thanks,
-ab

Re: [htdig-dev] truncated output from htsearch...a strace

[Aaron B. <ab...@mi...>]

The previous post had the same file attached twice.  Here is the correct 
  strace.file.

Maybe a problem with not flushing stdout?  Is this necessary?

Thanks,
-ab

[htdig-dev] Re: truncated output from htsearch...a strace

[Lachlan A. <lh...@us...>]

Sorry Aaron, I don't know what could be causing the problem.  However,=20
the fact that there are fewer  write()s  when the ouput is a file is=20
to be expected.  Output is "block-buffered" if the output is not a=20
terminal, which means it saves up 4096 bytes before writing.  If the=20
output is a tty, the data is "line-buffered", which means it is=20
written after every '\n'.

There *shouldn't* be a need to flush stdout unless the program exits=20
abnormally (e.g. segfaults).  Is the resulting file a multiple of=20
4096 bytes long?  What happens if you simply pipe the output through =20
more/less?  Is it still truncated?

Good luck!
Lachlan

On Saturday 05 April 2003 07:21, Aaron Bush wrote:

> If i run htsearch from the command line and allow the output to go
> to the screen the results appear complete and proper.  If i run
> htsearch from the command line and redirect the output to a file
> via ">/tmp/pg" or from a browser via apache then the results will
> be truncated.  I have ran strace against htsearch with output to
> the screen and to a file and there are clearly some missing
> write()'s.

Re: [htdig-dev] Re: truncated output from htsearch...a strace

[Aaron B. <ab...@mi...>]

> 
> Sorry Aaron, I don't know what could be causing the problem.  However, 
> the fact that there are fewer  write()s  when the ouput is a file is 
> to be expected.  Output is "block-buffered" if the output is not a 
> terminal, which means it saves up 4096 bytes before writing.  If the 
> output is a tty, the data is "line-buffered", which means it is 
> written after every '\n'.

This explains all the small writes to the terminal and the 4K to the file.

> 
> There *shouldn't* be a need to flush stdout unless the program exits 
> abnormally (e.g. segfaults).  Is the resulting file a multiple of 
> 4096 bytes long?  What happens if you simply pipe the output through  
> more/less?  Is it still truncated?

The entire stdout to a file is 12315 bytes.  If you remove all the header:
Content-type: text/html^M
^M

the file becomes 12288 (4096 * 3).  The same problem is seen when i pipe the output to less.

Thanks,
-ab

Re: [htdig-dev] Re: truncated output from htsearch...a strace

[Aaron B. <ab...@mi...>]

Lachlan Andrew wrote:
> There *shouldn't* be a need to flush stdout unless the program exits 
> abnormally (e.g. segfaults).  Is the resulting file a multiple of 
> 4096 bytes long?  What happens if you simply pipe the output through  
> more/less?  Is it still truncated?

Here are two more strace's which are set to watch read()'s on FD=5 and 
write()'s on FD=1.  I have found that when the output is sent to the 
screen (line buffered) the final write() before the open of the footer 
is different than the final write() when the output is to a file or program.


output to screen (line buffered) strace.screen line # 1114:
----------------------------------
write(1, "</dd></dl>\n", 11)            = 11
| 00000  3c 2f 64 64 3e 3c 2f 64  6c 3e 0a                 </dd></d l>.|

open("/web/pspec/htdocs/Templates/footer.html", O_RDONLY) = 5


output to file/program (4K block buffered) strace.file line # 865:
------------------------------------------
  | 00fe0  20 32 36 20 4e 65 77 65  73 74 3a 20 30 34 2d 32   26 Newe 
st: 04-2 |
  | 00ff0  31 2d 39 38 20 4f 6c 64  65 73 74 3a 20 30 35 2d  1-98 Old 
est: 05- |

open("/web/pspec/htdocs/Templates/footer.html", O_RDONLY) = 5

----
The line buffered write() data is the final result of the search (#9 of 
9).  This is correct since the output of results is complete and the 
next item to read/write is the footer.

The block buffered write() data is not correct since the output of 
results is not complete (this output is actually part of result # 7 of 
9).  Here the program does not complete the output of results before it 
goes on to read() in the footer template.  However the write() after the 
read of the footer template picks up where it left off in result # 7 of 
9.  Is this correct behavior?

Does this help pinpoint where the bug may be?  I'm not the best when it 
comes to C++  so any help tracking this down would be appreciated.

Thanks,
-ab