From: SourceForge.net <no...@so...> - 2012-08-16 20:27:15
|
Bugs item #3464439, was opened at 2011-12-23 08:15 Message generated for change (Comment added) made by szotsaki You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=111118&aid=3464439&group_id=11118 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Pending Resolution: None Priority: 5 Private: No Submitted By: Kevin (photo2110) Assigned to: Fabian Keil (fabiankeil) Summary: Page truncation in Opera Initial Comment: When working with Opera 11.60 and Privoxy, pages, images, etc will randomly get truncated. It just started with the Opera upgrade, so it might be a bug with Opera, but when I stopped using Privoxy, I couldn't get any pages to fail. Note: 'stopped using' means configuring Opera to not go through Privoxy, simply unchecking 'Enable' on the Privoxy menu doesn't do it. I attempted to capture data with Wireshark, but the pages I caught were gzipped and I can't be certain that they were good. I did notice that they were http 1.1, if that makes a difference. Only one other person has mentioned on the Opera news group that they are having an issue like this. They too are running Privoxy. NYTimes.com seems especially bad to me, but it happens on all sites including SouceForge. Privoxy 3.0.18. Opera 11.60 Windows 7 and Windows XP (I have two machines, both have the trouble) No other proxy software. Windows Firewall and MSE on both machines. SF user: photo2110 ---------------------------------------------------------------------- Comment By: szotsaki (szotsaki) Date: 2012-08-16 13:27 Message: I made a mistake in the previous capture with Wireshark (it was bound to port 80 only). Here is the updated files on all interfaces and on all ports: http://home.sch.bme.hu/~szotsaki/privoxy4.tar.bz2 ---------------------------------------------------------------------- Comment By: szotsaki (szotsaki) Date: 2012-08-16 13:16 Message: I'm here again, sorry for the delay. On Linux I could successfully reproduce this error and Wireshark had an option to capture on all interfaces. I turned on all of the debug options in privoxy along with prevent-compression. The affected page was: index.hu. I started Opera with a whole new "personal directory", so basically it was a pure default installation without extensions or anything unrelated. You can download the compressed log files here: http://home.sch.bme.hu/~szotsaki/privoxy3.tar.bz2 ---------------------------------------------------------------------- Comment By: Kevin (photos2110) Date: 2012-07-24 12:12 Message: IIRC, Wireshark can't capture localhost on Win 7. You'd have to install privoxy on a different machine to capture the browser<->Privoxy traffic. ---------------------------------------------------------------------- Comment By: Fabian Keil (fabiankeil) Date: 2012-07-24 11:48 Message: Thanks for the additional information. The pcap file still doesn't contain the traffic between Opera and Privoxy. Is this intentional or is the tool you are using for the capture incapable of capturing localhost traffic? Anyway, I realized that I interpreted the the previous Privoxy log file incorrectly. The client socket isn't actually kept open. I'll add another log message to make this more obvious in the future. Without seeing the traffic between Privoxy and Opera I can't be sure, but my impression is that Opera is doing the page truncation itself even though it gets the entire page from Privoxy. Privoxy could be more aggressive about client socket reuse and I'll think about that some more, but that's not a good reason for Opera to partially discard already received data (assuming Opera is indeed doing that). ---------------------------------------------------------------------- Comment By: szotsaki (szotsaki) Date: 2012-07-23 11:36 Message: This time I could reproduce the problem with this very site. I added the levels 16 and 32768 to the debug options and copied the "{+prevent-compression} /" to the end of the Default action file. And now I had quit Opera before I closed Wireshark. You can find the uploaded files here: http://home.sch.bme.hu/~szotsaki/privoxy2.tar.bz2 ---------------------------------------------------------------------- Comment By: Fabian Keil (fabiankeil) Date: 2012-07-23 07:24 Message: Thanks a lot for the log files, but unfortunately the redacted pcap files do not show enough information. PrivoxyNYTimes.pcapng shows that Privoxy got a complete response from the server and privoxy-log-NYTimes.log indicates that Privoxy parsed and decoded it as expected before forwarding it to the client. The Privoxy log also indicates that the connection between Privoxy and Opera is then kept open without Privoxy detecting an additional request. This could be the result of a bug in Privoxy and would explain why disabling pipelining in Opera works around the problem, but without seeing the traffic between Privoxy and Opera it's impossible to tell if an additional request has actually been sent. The +prevent-compression action doesn't actually seem to be used. Please additionally enable "debug 16" and "debug 32768" and reproduce the log and pcap file, this time capturing the complete traffic between both Privoxy and the server and Privoxy and Opera. Please quit Opera before stopping the capture to make sure all connections between Privoxy and Opera get closed. If you can reproduce the problem with: {+prevent-compression} / it would make analyzing the logs easier. ---------------------------------------------------------------------- Comment By: szotsaki (szotsaki) Date: 2012-07-23 03:04 Message: Success :) I could reproduce this issue on nytimes.com while both the logging and the Wireshark was active. I couldn't figure it out how can I add attachments here (…), so I uploaded the compress files here: http://home.sch.bme.hu/~szotsaki/privoxy.tar.bz2 The content: - NYTimes.png: the truncated nytimes.com page. After the debugging I turned off the pipelining in Opera and it loaded normally again - PrivoxyNYTimes.pcapng: Wireshark pcap file with only one request to nytimes.com - privoxy-log-NYTimes.log: Privoxy log file with the recommended debug values (1,2, 4, 8, 128, 256, 512, 1024, 4096, 8192, +prevent-compression) which was made parallel with the above pcap file - PrivoxyOperaDesktopTeam.pcapng: one request to my.opera.com/desktopteam, ended up a "Network error" in Opera I'll be offline for two weeks, but I hope I could help you a bit regarding this issue. ---------------------------------------------------------------------- Comment By: Fabian Keil (fabiankeil) Date: 2012-07-21 06:50 Message: A log file would be appreciated, especially if it's created with +prevent-compression and comes together with a pcap file showing the actual packets. ---------------------------------------------------------------------- Comment By: szotsaki (szotsaki) Date: 2012-07-20 04:05 Message: I reported this bug to Opera bug tracker, but they forwarded me here. Turning off opera:config#pipelining (https://secure.wikimedia.org/wikipedia/en/wiki/HTTP_pipelining) really solves the issues with Privoxy. I can generate a log file, but if you start using Opera and set up Privoxy there, in an hour you will encounter this bug. Affected Hunarian sites: * origo.hu * index.hu * hup.hu Opera version: from 11.50 to 12.50 (now is alpha) all affected Operating system: Windows XP, 7 and Linux also affected. On Win7, with Opera 12.50 you can reproduce this bug _very_ often (I mean on almost every page request) with slashdot.org. ---------------------------------------------------------------------- Comment By: Fabian Keil (fabiankeil) Date: 2012-03-10 02:56 Message: The fact that disabling pipelining or putting Polipo between Opera and Privoxy helps could hint that Opera doesn't like the way Privoxy deals with pipelined requests, but it's unclear if that's Opera's or Privoxy's fault. The previously provided log indeed shows that Opera seems to be trying to pipeline requests at first (for example in thread 0000025c), but when Privoxy is done dealing with the first request it doesn't seem to get the next one possibly because Opera grew impatient. I can't reproduce this here (without Opera of course) so without a packet trace I will not be able to analyze this any further. ---------------------------------------------------------------------- Comment By: Arcady Shpak () Date: 2012-03-09 19:32 Message: opera:config -> Performance -> Enable Pipelining -> off ---------------------------------------------------------------------- Comment By: esmandil (esmandil1) Date: 2012-02-25 01:58 Message: I have the same problem. One thing I have noticed is that when I install polipo proxy ( http://www.pps.jussieu.fr/~jch/software/polipo/ ) and put it between Opera and Privoxy (i.e. pointing Opera to polipo, and setting up polipo to forward to Privoxy), the problem seems to disappear. If I put polipo between Privoxy and Internet, however, web pages in Opera are still broken. I don't know if this helps to diagnose the problem, but it's a workaround at worst. ---------------------------------------------------------------------- Comment By: qqqqqw (qqqqqw) Date: 2012-02-05 00:58 Message: Yes, is from mashable.com when i have some time i post a new log ---------------------------------------------------------------------- Comment By: Fabian Keil (fabiankeil) Date: 2012-01-23 10:33 Message: I assume in the logfile from sendspace.com/ the truncated output is the response for mashable.com? I tried reproducing it by sending the same headers Opera used, but the response appears to be complete. Unfortunately Privoxy's log buffer is smaller than the receive buffer so full receive buffers aren't logged completely in which case the received data can't be reconstructed. Can you still reproduce the problem if you enable +prevent-compression? A log file without compressed content would probably be easier to analyze. ---------------------------------------------------------------------- Comment By: qqqqqw (qqqqqw) Date: 2012-01-17 11:31 Message: I never use turbo mode ---------------------------------------------------------------------- Comment By: Lee (ler762) Date: 2012-01-15 12:59 Message: Try disabling turbo mode in Opera. I installed Opera and, with extremely limited testing, it worked fine until I enabled "turbo mode". At which point it broke. Changing the privoxy config to keep-alive-timeout 0 helped some, but did NOT fix all of the problems. I had wireshark capturing traffic during my testing. I thought I'd figured out the problem but now I'm not so sure :( I need to look at the packet traces again. Lee ---------------------------------------------------------------------- Comment By: qqqqqw (qqqqqw) Date: 2012-01-14 06:15 Message: I cannot see "add a file" so i upload another log file (with same debugs enabled) here: http://www.sendspace.com/file/n4v8k2 Another 2 thing: 1. I have never page truncation on https 2. If i have page truncation on a site and the i use a cgi proxy to go to the same page i have no page truncation ---------------------------------------------------------------------- Comment By: Fabian Keil (fabiankeil) Date: 2012-01-13 14:23 Message: Downloading the file from zshare.net doesn't work for me. The download server doesn't even bother to return HTTP headers. In theory you should be able to upload a file through the "Add a file" section on this page. ---------------------------------------------------------------------- Comment By: qqqqqw (qqqqqw) Date: 2012-01-13 03:30 Message: Page truncation on mashable.com with Opera 11.60 + Privoxy 3.0.19 and socks4a + Tor Log file with this debug enabled: debug 1 debug 2 debug 4 debug 8 debug 64 debug 128 debug 256 debug 512 debug 1024 debug 4096 debug 8192 debug 32768 Log file is here: http://www.zshare.net/download/986014474c9073b3/ How can i attach log file in this discussion? ---------------------------------------------------------------------- Comment By: qqqqqw (qqqqqw) Date: 2012-01-01 10:09 Message: Yes, i use Opera 11.60 and the problem is always on same site (i dont know why) After the upgrade to 3.0.19 and a quick surf seem that the problem is socks5 + Tor (so Privoxy 3.0.18 for bug in socks4a) and NOT socks4a + Tor (so Privoxy 3.0.19) But i need more test (surf to more internet pages and try to use socks5 on Privoxy 3.0.19) If i have other page truncation i post a log ---------------------------------------------------------------------- Comment By: Fabian Keil (fabiankeil) Date: 2011-12-31 05:02 Message: qqqqqw, are you using Opera as well? Is the problem easy to reproduce? Can you provide a logfile with the usual debug directives and additionally "debug 16" and "debug 32768"? It would be especially useful to additionally get a pcap file that shows the packages Privoxy was receiving and sending. I think it's unlikely that upgrading to 3.0.19 makes a difference. You could try the recently uploaded privoxy-3.0.19-ipv4only.zip as it uses different code paths, but it probably will not make a difference either. ---------------------------------------------------------------------- Comment By: Kevin (photo2110) Date: 2011-12-29 11:58 Message: Sorry I've not replied sooner. Xmas break took priority. I put in a section to downgrade the NYTimes, and couldn't get any more issues with that site. So, I put in a downgrade for all of .com, and haven't seen anymore short pages on a .com site. I'll have to try the new .19 version. ---------------------------------------------------------------------- Comment By: qqqqqw (qqqqqw) Date: 2011-12-29 11:53 Message: I have the same problem. Initially i think that this problem is related with socks4a/5 bug, but after read this i'm not so sure. I have page truncation with Privoxy 3.0.18, not with 3.0.17. Now i try with 3.0.19 ---------------------------------------------------------------------- Comment By: Fabian Keil (fabiankeil) Date: 2011-12-23 11:21 Message: Thanks. I can't see any obvious problems in the log. Please additionally add "debug 16" so the data written to the network is included as well. Does it make a difference if you add the following to one of your action files?: {+downgrade-http-version} # Sticky Actions = +downgrade-http-version # URL = http://www.nytimes.com/pages/travel/index.html .nytimes.com/ ---------------------------------------------------------------------- Comment By: Kevin (photo2110) Date: 2011-12-23 10:06 Message: New log file added with debug 32768 turned on. Truncated page is http://www.nytimes.com/pages/travel/index.html which dies at: <h6 class="kicker">Practical Traveler</h6> <div class="thumbnail"> <a href="http://travel.nytimes.com/2011/12/25/travel/apps-to-keep-children-happy.html?ref=travel"><img src="http://graphics8.nytimes.com/images/2011/12/25/travel/25PRAC/25PRAC-thumbStandard-v2.jpg" width="75" height="75" alt="" border="0" /></a> </div> <h3><a href="http://trav ---------------------------------------------------------------------- Comment By: Fabian Keil (fabiankeil) Date: 2011-12-23 09:41 Message: Please additionally add "debug 32768" so we can see the data read from the network. ---------------------------------------------------------------------- Comment By: Kevin (photo2110) Date: 2011-12-23 09:23 Message: I've attached a log file for a truncated page. http://www.nytimes.com/interactive/2011/12/22/magazine/the-lives-they-lived.html?hp#view=dennis_ritchie Opera's source view goes along until: "100000001233615":{count:93, commentsEnabled:false}, "100000001233630":{count:57, commentsEnabled:false}, "100000001233632":{count:318, commentsEnabled:false}, "100000001233655":{count:441, commentsEnabled:fa and we see it just stops in the middle of the word false. I'm apparently not understanding the log file, it looks like it grabbed the page twice, with the second grab bad. ---------------------------------------------------------------------- Comment By: Fabian Keil (fabiankeil) Date: 2011-12-23 08:36 Message: Thanks for the report. Please reproduce the problem after enabling logging as described at http://www.privoxy.org/user-manual/contact.html#CONTACT-BUGS and provide a log file. If you can make the pcap file created with Wireshark available, that may be useful, too. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=111118&aid=3464439&group_id=11118 |