Re: [Simpleweb-Support] Binary data errors on ResponseStream?
Brought to you by:
niallg
From: Kai S. <sch...@gm...> - 2007-07-30 19:42:28
|
Hi, This bug probably originates from a JVM encoding property issue, and a MySQL Java driver blob conversion issue. What I didn't mention in the below mail is that I was running Simple on my desktop WinXP machine and I always run Tomcat on my Linux (debian) box. I was using the same MySQL server for each. BLOB values from Mysql are converted to Strings, so character encoding matter. The default encoding for my debian server is ISO-8859-1, while my desktop makes the JVM use windows-1252. I haven't fully tested this yet, cause I'm still struggling with JVM configurations, but I'm quite sure that's the issue. Damn windows... Either way, I've run a test to transfer all bytes from -127 to 128 with a protocol handler, and it worked fine. This bug is definitely NOT Simple related. I originally was worried because the bytes that were getting corrupted all had their high bit set, and numerical cast conversion in Java can be a little weird, since there's no unsigned numerical types in Java. After digging into it more, only 5 of the 127 negative bytes were being converted into hexadecimal 3f, so it's obvious that wasn't the problem. Thanks for your reply Niall, I'll post one more reply with the full fix once I've tested it. I'm looking forward to 4.0 as well =D Regards, -Kai On 7/29/07, Niall Gallagher < gal...@ya...> wrote: > > Hi Kai, > > The issue you describe is quite strange. It could very well be a problem I > have not encountered before, perhaps its related to the output buffer in the > response stream. I've used Simple in quite a few projects and never seen any > data corruption issues. To verify this it may be worth while tweaking the > ResponseStream object to collect the data written to it and dump it out on a > close, just to see if its being used to write an initially valid file. Also, > setting the response buffer to zero bytes and trying would clarify if the > response buffer is at fault. > > Nectar sounds like a very interesting project. Its always interesting > deploying large web based distributed systems, in fact it was the original > reason I wrote the framework. I had intended it to cope with very large > loads, at the time there was no open source alternative. One thing that may > help you out in this is using asynchronous capabilities of Simple. Something > I have not documented very much is that the request and response are > completely independent of the servicing thread, unlike Servlets where the > Servlet.service method must perform the entire transaction. This can be > useful in a distributed system as it allows you to perform many back end > operations in parallel without serializing throughput from the server. In > this sense the ProtocolHandler.handle is more like an event handler, or a > handover, than a conventional method invocation. You may have read something > similar in the Async Servlet JSR, or Jetty continuations, both of which I > feel are greatly hindered by legacy compatibility. > > Also, I have nearly finished the implementation of Simple 4.0. Its almost > a complete rewrite, and its about 30% faster, and much more responsive. I > think it will be a little while yet before I release it though, could be a > few months yet. > > Niall > > > ----- Original Message ---- > From: Kai Schutte < sch...@gm...> > To: sim...@li... > Sent: Sunday, July 29, 2007 4:28:00 AM > Subject: [Simpleweb-Support] Binary data errors on ResponseStream? > > Hello Niall and all the others, > > I've come to my first real problem using Simple. I'd like to ask for a > little help about this issue, then give you a little summary of what I'm > doing with Simple: first of all to let you know how much I appreciate > Simple, and then maybe see if you have any advice. > > My specific problem is in the context of writing a binary file (a JPG > image) to the client. I'm setting Content-type, Content-length, adding no > cache headers. The data of the image comes from a MySQL table. You may argue > that storing static images in a database is a silly idea, and I'd agree, > this is just test case. My system runs with Tomcat, and just a few days ago > I started to get it to run with Simple as well. Under Tomcat, the image > appears fine, but with Simple, the image is corrupted. > > I've attached the two images from Tomcat and Simple, and a binary diff > generated by bsdiff. The two files are the same length (so it's not a > character escaping issue, and the start and end bytes are the same, so > neither file has been truncated) and they are very similar (the diff is very > small). I'm thinking it might be an encoding setting problem, or maybe I'm > using the wrong OutputStream for binary data? I've also attached the > implementation of my ProtocolHandler , and the action implementation that > retrieves the image from the database and prints it to the OutputStream. The > request / response objects are tossed around a few classes in between, but > are never really interacted with. The nectar.http.Response and > nectar.http.Request classes are just wrappers to interleave Tomcat's > javax.servlet.http.HttpServlet(Request|Response) and Simple's equivalents. > MySQL is in UTF8_bin collation, but this shouldn't affect BLOB fields, > right? > > ________________ > > My project started in 2003 as an expansion on Apache Struts (a servlet > framework for Tomcat), integrating a modular database adapter (not > specifically relying on SQL), a very strict Model-View-Action design for > each webpage's functionality. Action can access 'common' functionality > through Service classes, unique in the server instance. I worked 4 months > straight on the project non-stop after coming home to my parents from a > failed .com experiment (don't we all have similar stories?). I returned to > university, and only sporadically worked on the project since. > > The core design of my system was very good, but Struts very quickly became > more of a restriction than a supporting platform. I don't know if you've > ever stripped out a core framework out of 50,000+ lines of code, but, it's > painful! Right about the time Java 1.5.0 came out, my project was totally > free of Struts. I then went to work on implementing generics throughout. The > last public release of the project was in September 2004 ( > http://sourceforge.net/projects/nectar/ ); it was called Nectar Web > Services and it's a very outdated version. I prefer to call it Nectar Web > Platform today. It's very unlikely there will be a new release in the year > to come. > > My overall goal with Nectar is for it to become a very modular system for > a cluster of servers... Load balancers, worker servers, database nodes, > monitoring servers, a main controller, the works. The application domain > would be mainly anything a web browser can display, plus eventually > streaming content (audio / video), content syndication (SOAP style), while > keeping the architecture open enough to deal with pretty much anything that > comes along. > > My current main goal in the project are to get rid of more dependencies. > The Servlet / JSP framework is nice and all, but it often feels like a bad > implementation of PHP. MySQL is solid, but for the specific data structures > I'm using, it's just overkill, leads to extra latency, and will be plain > counterproductive in a large cluster environment where a fully distributed > database starts to show it's potential. Tomcat also provides tons of > features and gimmicks, none of which I'm using. Stack traces of exceptions > thrown by my code usually show a list of 15 functions in Tomcat. > > My cluster design will hopefully be using one big Layer 4 switch at the > very front, handing out (round robin style) requests to Simple on several > load balancers. The load balancers would be the only servers running Simple, > just to decode the HTTP request and pass the information on to > Object*putStreams connected to the worker servers, which in turn will do all > the internal stuff to render the response, and pass it back to the load > balancers in another IOStream. I'm not sure about the details, I might use > IPv6 connections internally to allow for multiple channels per connection > between load balancer and worker node, and then figure out some way to > manage multiple requests asynchronously on the load balancers... > > Like many Open Source projects, I've got no funding, no specific > requirements or strict goal. It's just my hobby which I'm fascinated by, and > I've rarely written any code without a beer to keep my company! Of course > I'll never have the funds to fully test a cluster setup, but maybe I can get > something going with enough Virtual PC instances :) > > I like being in control of things: SQL on the one hand already pretty much > determines how you have to think about queries. Tomcat on the other hand > provides dozens of strict and complex tools, which grasp deep into your own > code. With Simple I basically just have one function to implement. My code > initializes Simple, while Tomcat initializes my code. This alone is a huge > paradigm shift that makes me feel a lot more comfortable. > > I took one long evening to read through some Simple's code, before I chose > to integrate it. I found it quite fascinating how... well... simple... it > was. It shows you've thought well about what your were going to write before > you wrote it. Goodness, I wish I had your documentation discipline too! > > ________________ > > Thanks very much for your work, Niall and thanks to all those who helped > you. I hope I can be of some use to your project once I'm more familiar with > it. > > Kind Regards, > > -Kai > > > > ------------------------------ > Luggage? GPS? Comic books? > Check out fitting gifts for grads > <http://us.rd.yahoo.com/evt=48249/*http://search.yahoo.com/search?fr=oni_on_mail&p=graduation+gifts&cs=bz>at Yahoo! Search. > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. > Still grepping through log files to find problems? Stop. > Now Search log events and configuration files using AJAX and a browser. > Download your FREE copy of Splunk now >> http://get.splunk.com/ > _______________________________________________ > Simpleweb-Support mailing list > Sim...@li... > https://lists.sourceforge.net/lists/listinfo/simpleweb-support > > |