#1384 HTTP service fails when downloading large FLV files

Unix::Linux
open
Sharon Lucas
5
2014-09-23
2010-12-14
sandrews
No

This happens when large files are downloaded from inside a perl script using the API provided.

submitting http request : REQUEST METHOD GET URL http://hike.bytemobile.com/tmo/flv/64kbs/evangel_op.flv SESSION 6 FOLLOWREDIRECT
Error on http request get url http://hike/tmo/flv/64kbs/evangel_op.flv
Expected RC: 0, Result: html page and status code
Received RC: 4004, Result: Error getting target url
java.net.SocketException: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)

at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at com.ibm.staf.service.http.WebSession.requestMethod(WebSession.java:1671)
at com.ibm.staf.service.http.HTTP.handleSessionMethod(HTTP.java:1637)
at com.ibm.staf.service.http.HTTP.handleMethod(HTTP.java:1324)
at com.ibm.staf.service.http.HTTP.acceptRequest(HTTP.java:222)
at com.ibm.staf.service.STAFServiceHelper.callService(STAFServiceHelper.java:349)

If I execute the same command from Shell prompt, I don't get this error

staf 10.21.29.100 http request method get url http://hike/tmo/flv/64kbs/evangel_op.flv session 6

This download is expected last a few seconds to a full minute. Is there a wait parameter that can be added to the HTTP Request method?

Discussion

  • Sharon Lucas
    Sharon Lucas
    2010-12-15

    What version of the HTTP service are you using? Make sure you're using the latest, V3.0.2.

    STAF httpServiceMachine HTTP VERSION

    You said this downloads a large FLV file. What is the size of this file?

    What is the content of the downloaded FLV file? If it isn't Html, then you should try setting the parse content to Disabled on the session before submitting the GET request so that it doesn't try to parse the result content. For example:

    STAF httpServiceMachine HTTP SET PARSECONTENT DISABLED SESSION <SessionID>

    Are you saying the result returned by your HTTP REQUEST METHOD GET request is very large? If so, you should try adding the FILE option to specify a file to contain the result. This because otherwise the result is returned as a string and if it's very large, it could exceed the available memory on your machine.

    Note that it doesn't matter if the HTTP service request is submitted via a command shell prompt or via a perl script. The HTTP service responds in the same manner. However, perhaps you have less available memory when the HTTP service request is run via a perl script (hence why I suggested adding the FILE option to store the result in a file instead of in a string in memory).

     
  • sandrews
    sandrews
    2010-12-15

    I'm using HTTP 302.

    The file is not VERY large - it is about 3.5MB in size.

    I tried adding the FILE parameter to the HTTP request - it didn't make any difference.

    I also set the PARSING DISABLED. That didn't help either.

    [root@vm-linux-client services]# staf local http query session 7
    Response
    --------
    {
    Url : http://hike/tmo/flv/64kbs/evangel_op.flv
    Title : <None>
    Status Code : -1
    Status Message : <None>
    Cookie Policy : default
    Follows Redirects: Disabled
    Parse Content : Disabled
    Owner : {
    Instance UUID: 613DD34C433900000A00048073746572
    Machine : tcp://10.21.29.199@6500
    Handle Name : Lang/Perl/Test/Basic
    Handle : 372
    }
    }

    submitting http request : SET PARSECONTENT DISABLED SESSION 7
    submitting http request : REQUEST METHOD GET URL http://hike/tmo/flv/64kbs/evangel_op.flv FILE /tmp/out.flv SESSION 7
    Error on http request get url http://hike/tmo/flv/64kbs/evangel_op.flv
    Expected RC: 0, Result: html page and status code
    Received RC: 4004, Result: Error getting target url
    java.net.SocketException: Connection reset
    java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(Unknown Source)
    at java.io.BufferedInputStream.fill(Unknown Source)
    at java.io.BufferedInputStream.read(Unknown Source)
    at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)

     
  • Sharon Lucas
    Sharon Lucas
    2010-12-15

    • assigned_to: nobody --> slucas
     
  • sandrews
    sandrews
    2010-12-15

    Thanks Sharon for the feedback and your help.

    The product I'm testing is an HTTP proxy, so it doesn't make sense for me to do an FS copy or FTP.

    I'm able to run the same STAF command from the command line and it goes through successfully, every single time.

    I get the connection error when I call the perl submit2 API, it happens every single time.

     
  • Sharon Lucas
    Sharon Lucas
    2010-12-15

    It doesn't make any sense to me (assuming you are submitting the exact same HTTP GET request) that it works when submitted via a command shell prompt but fails when submitted via a perl script. The HTTP service (and the underlying HttpClient) have no idea how the request was initiated and thus, nothing is done differently within the HTTP service. So, I don't see how this is a problem in the HTTP service. Unfortunately, I don't know what else to suggest.