Invalid response coming from server

Help
Levia
2009-01-07
2013-04-25
  • Levia
    Levia
    2009-01-07

    I'm using the Simple + Redstone XML-RPC libraries to provide a server. Now, everything went fine, until I started returning some big sets of data...The responses are incorrect XML and seem to be mixed up. Here's a small part of the incorrect format found in the returned XML response:

    <value><string>bkabka</string>ing>Co-Worker
                                            </string>

    and then it just continues like nothing really went wrong. Any ideas?

    Thanks,
    Lars

     
    • Greger Ohlson
      Greger Ohlson
      2009-01-07

      Hi,

      Hmm... this is new to me. Before I start troubleshooting this it would be great to know some background information:

      * The XML dump with the problem you included in your message, how was that captured? With tcpdump and Wireshark, for instance, this may occur at packet boundaries occasionally, although the actual traffic end-to-end is OK. Just to rule out any errors related to capturing the XML traffic.

      * A brief explanation of the mix of data you are sending (lots of structs? deep nesting? how large are the arguments/payload, roughly?)

      * What XML-RPC client are you using and does it report that the error is related to parsing of payload?

      Sorry if these questions sound dumb, but it will help during troubleshooting. I will try to reproduce it with Simple HTTP and with Jetty or Tomcat to see if it is really related to the library or the Simple HTTP server.

      Thanks,
      Greger.

       
      • Hi,

        I am aware this issue may have been resolve. However if you are going to use the Simple HTTP server, you would need to upgrade to the most recent release 4.1.12. An earlier bug in Simple is known to cause the issues mentioned here.

        Niall

         
        • Greger Ohlson
          Greger Ohlson
          2009-06-09

          Thanks Nial, that's really good to know! I will test with the latest Simple release and update our release package accordingly.

          Best Regards,
          Greger.

           
    • Levia
      Levia
      2009-01-07

      1) I did it in two ways, the first way is the actual client I wrote. It uses the PHP-XMLRPC library, and it has the ability to print out the complete response before it is interpreted by the library. That response already contains the errors. So that response looked something like (if I recall correctly)

      HTTP 1/1.1 OK
      Connection close

      <the xml data>

      2) Yes, lots of structs, in fact, everything is a struct - all the data returned. (In Java I simply returned an ArrayList) and yes, deep nesting. For example, a struct contains another struct. (It doesn't go any deeper than that, I believe).

      3) PHP-XMLRPC. I also wrote a test client using Redstone XML-RPC and the same problem seems to occur - exceptions (in that Java client):

      redstone.xmlrpc.XmlRpcException: The response could not be parsed.
          at redstone.xmlrpc.XmlRpcClient.handleResponse(Unknown Source)
          at redstone.xmlrpc.XmlRpcClient.endCall(Unknown Source)
          at redstone.xmlrpc.XmlRpcClient.invoke(Unknown Source)
          at nl.csh4u.urenregistratie.client.Main.<init>(Main.java:43)
          at nl.csh4u.urenregistratie.client.Main.main(Main.java:26)
      Caused by: redstone.xmlrpc.XmlRpcException: A problem occured during parsing
          at redstone.xmlrpc.XmlRpcParser.parse(Unknown Source)
          ... 5 more
      Caused by: java.io.IOException: missing CR
          at sun.net.www.http.ChunkedInputStream.processRaw(ChunkedInputStream.java:378)
          at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:545)
          at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:582)
          at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:669)
          at java.io.FilterInputStream.read(FilterInputStream.java:116)
          at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2391)
          at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
          at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
          at com.sun.org.apache.xerces.internal.impl.XMLEntityManager$RewindableInputStream.read(XMLEntityManager.java:2939)
          at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
          at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
          at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
          at java.io.InputStreamReader.read(InputStreamReader.java:167)
          at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1742)
          at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.arrangeCapacity(XMLEntityScanner.java:1619)
          at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipString(XMLEntityScanner.java:1657)
          at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1740)
          at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2930)
          at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
          at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140)
          at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
          at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:807)
          at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
          at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:107)
          at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
          ... 6 more

      Process finished with exit code 0

      In the PHP version, it just shows: XML parse error on line 1

      Thanks,
      Lars

       
      • Greger Ohlson
        Greger Ohlson
        2009-01-07

        Thanks a lot! It sure looks like a problem at the server end given your explanation. I was considering a possible memory problem with your client but since you get the same result with a Java implementation it is not very likely.

        I'll see if I can get the Simple HTTP server-version to reproduce it on my end, and I'll also try the servlet version (XmlRpcServlet) which can be published in a servlet container (we've used mainly Jetty).

        It's getting a bit late here so possibly I won't have any findings until tomorrow.

        Regards,
        Greger.

         
        • Greger Ohlson
          Greger Ohlson
          2009-01-07

          Hi,

          I think I may have reproduced it. I created a simple test service that generated a three level hierarchy of XML-RPC structs with any given number of mid-level and third-level structs (SF is gonna squash this code):

          package test;

          import java.util.HashMap;
          import java.util.Map;

          public class Service
          {
              // Returns a 3 level hierarchy of XML-RPC structs.
              public Map getLargeNestedStruct(int midLevelCount, int leafCount)
              {
                  // The leaf instance is reused and referenced many times from each mid level
                  // instance. On the wire, this is expanded to a large payload.
                  HashMap leaf = new HashMap();
                  leaf.put("abc", "string");
                  leaf.put("def", 42);
                  leaf.put("ghi", 42.0);
                 
                  // Each mid-level instance has midLevelCount number of leafs. Although the same
                  // leaf instance is reused in all mid-level objects, when serialized they will
                  // become multiple instances on th wire.
                  HashMap midLevel = new HashMap();
                  midLevel.put("abc", "abcdefghijklmnopqrstuvxyz1234567890");
                  midLevel.put("def", 42);
                  midLevel.put("ghi", 42.0);
                  for (int i = 0; i < leafCount; ++i)
                      midLevel.put("" + i, leaf);

                  // Top level container, containing midLevelCount mid-level instances, each containing
                  // leafCount leaf instances, a total of midLevelCount * leafCount + 1 XML-RPC structs.
                  HashMap topLevel = new HashMap();
                  for (int i = 0; i < midLevelCount; ++i)
                      topLevel.put("" + i, midLevel);
                 
                  return topLevel;
              }
          }

          Anyhoo... when invoking this service with arguments (100,100) we will get a top level struct with 100 members, keyed by "1".."100". Each of the members are structs as well with four members (string, int, double, and array). The array element contains yet another 100 structs containing a string, int, and double member.

          So the response contains 10.000 XML-RPC structs in total, over three levels with 3-4 member fields. This can be considered a really large XML-RPC response (considering XML-RPC is not designed for large datasets).

          The good news is I managed to get the exact same stack trace as you did. A value of (50,50) in the test service succeeds (rendering 2.500 structs in the response).

          I'll check if the problem occurs when the library is running in a servlet container using the XmlRpcServlet.

           
    • Levia
      Levia
      2009-01-08

      Hey,

      Thanks for giving this a try, and I'm glad you got the same stacktrace, however, I don't think I have that many structs. Let's see. About 27 structs, and all of em have one struct in there. That'd be 54 structs. I'll give you more information on the invocation that fails, like how many structs there actually were.

      Thanks!
      Lars

       
    • Levia
      Levia
      2009-01-09

      Hello,

      After poking through the code a bit, it almost seems like it's streaming the responses to the client? I mean, I can see the writer.write() calls everywhere, and it is flushed in the execute command..it's not like a packet is constructed - it seems to be streamed?

      Also, is there some way to print the response of the server before it is sent? Since it seems to be streamed, I would guess not?

      Thanks,
      Lars

       
    • Levia
      Levia
      2009-01-09

      Okay, kinda cheated here and there - in Server.java, I changed
      Writer responseWriter = new PrintStream(response.getOutputStream()); to
      Writer responseWriter = new StringWriter();

      I then did System.out.println(responseWriter.toString()); after the serialization was done. The XML is perfectly valid! Then I thought I'd send that string to the client, by simply doing:

      OutputStream stream = response.getOutputStream();
      stream.write(responseWriter.toString().getBytes());
      stream.flush();

      But it arrived invalid again. This proves that the problem is most probably not with Redstone XML RPC, since it sends out the XML data correctly. I think it's because of Simple. I'm doing some more research on this.

      Thanks,
      Lars

       
      • Greger Ohlson
        Greger Ohlson
        2009-01-09

        Hi,

        Sorry for the late reply, I've been in meetings all day. I too was starting to suspect Simple since the XML-RPC library (XmlRpcDispatcher to be specific) only writes directly to the response stream supplied by Simple, and flushes it appropriately. There appears to be some problems with HTTP chunking.

        If you project is critical I would suggest using the XmlRpcServlet and a servlet container like Jetty while this is being resolved. There is information on how to do this on the homepage but I can also send you a complete project structure if you're in a hurry, based on Jetty.

        Regards,
        Greger.

         
    • Levia
      Levia
      2009-01-09

      Hello,

      Sending the project structure would be nice. I'm on a deadline, by the end of this month basically so this could save me some time. Also, I heard that using a servlet for this kind of stuff is better anyway - thanks!

      Lars

       
      • Greger Ohlson
        Greger Ohlson
        2009-01-09

        Hi,

        I sent you a Jetty starter kit, running Jetty embedded (no WEB-INF or web.xml required and all that). Not sure if it arrived. Otherwise you can get it at http://hwww.redstone.se/jetty-starter.tgz

        Regards,
        Greger.

         
    • Levia
      Levia
      2009-01-09

      I already sent you an email to let you know, but let me post here for other people:

      This solution works for me. Thanks alot.

      By the way, have you thought about replacing the Simple + XML RPC package with the one you just linked?

      Lars