curl-loader-devel

TCP & curl-loader question

[Gary F. <ga...@in...>]

Hello Robert,

I've run into a problem that seems unrelated to the curl-loader  
extensions I'm building. This occurs whenever I have two or more  
virtual clients running simultaneously. Here's what I see when I look  
at the TCP traffic with Wireshark:

Roughly half the clients proceed normally. The server sends the final  
response of the conversation, the client ACKs this, and then the  
client initiates a FIN-ACK and the server replies with FIN-ACK. That's  
it, and everyone's happy.

But with the rest of the clients, something happens in the middle of  
the conversation, before the client is ready to finish. The server  
replies to a client request, the client ACKs this reply, and then the  
server unexpectedly initiates a FIN-ACK. The client responds with two  
RST packets, but the server doesn't react, and the client times out.

Unfortunately I'm no TCP expert. Is this a server malfunction? Or  
perhaps this is normal server behavior, and the curl clients should  
handle the unexpected FIN-ACK differently? I don't know if you can  
shed any light on this, but I'd be grateful for any ideas.

I've included a tcpdump file (which you can view with Wireshark) and  
the .log file. (It's interesting that in the case of the  
malfunctioning clients, the final server reply is never logged,  
although it appears in the tcpdump.) I can send the .conf file as well  
if you want (but that might just be a distraction). The command line  
was $curl-loader -u -v -f genusers.conf. The same thing happens when I  
add the -r option.

Thanks,
Gary Fitts

Re: TCP & curl-loader question

[Robert I. <cor...@gm...>]

On Sat, Nov 1, 2008 at 2:47 AM, Gary Fitts <ga...@in...> wrote:

> Hello Robert,
>
> I've run into a problem that seems unrelated to the curl-loader extensions
> I'm building. This occurs whenever I have two or more virtual clients
> running simultaneously. Here's what I see when I look at the TCP traffic
> with Wireshark:
>
> Roughly half the clients proceed normally. The server sends the final
> response of the conversation, the client ACKs this, and then the client
> initiates a FIN-ACK and the server replies with FIN-ACK. That's it, and
> everyone's happy.
>
> But with the rest of the clients, something happens in the middle of the
> conversation, before the client is ready to finish. The server replies to a
> client request, the client ACKs this reply, and then the server unexpectedly
> initiates a FIN-ACK. The client responds with two RST packets, but the
> server doesn't react, and the client times out.
>


Could it be, that your URL_COMPLETION_TIMEOUT value is too short
and you need to increase it?

It may happen, when POST-ing is accompanied by 100-Continue as seen in your
log file.

-- 
Truly,
Robert Iakobashvili, Ph.D.
......................................................................
www.ghotit.com
Assistive technology that understands you
......................................................................

Re: TCP & curl-loader question

[Robert I. <cor...@gm...>]

Hi Gary,

On Sat, Nov 1, 2008 at 2:47 AM, Gary Fitts <ga...@in...> wrote:

> Hello Robert,
>
> I've run into a problem that seems unrelated to the curl-loader extensions
> I'm building. This occurs whenever I have two or more virtual clients
> running simultaneously. Here's what I see when I look at the TCP traffic
> with Wireshark:
>
> Roughly half the clients proceed normally. The server sends the final
> response of the conversation, the client ACKs this, and then the client
> initiates a FIN-ACK and the server replies with FIN-ACK. That's it, and
> everyone's happy.
>
> But with the rest of the clients, something happens in the middle of the
> conversation, before the client is ready to finish. The server replies to a
> client request, the client ACKs this reply, and then the server unexpectedly
> initiates a FIN-ACK. The client responds with two RST packets, but the
> server doesn't react, and the client times out.
>
> Unfortunately I'm no TCP expert. Is this a server malfunction? Or perhaps
> this is normal server behavior, and the curl clients should handle the
> unexpected FIN-ACK differently? I don't know if you can shed any light on
> this, but I'd be grateful for any ideas.
>
> I've included a tcpdump file (which you can view with Wireshark) and the
> .log file. (It's interesting that in the case of the malfunctioning clients,
> the final server reply is never logged, although it appears in the tcpdump.)
> I can send the .conf file as well if you want (but that might just be a
> distraction). The command line was $curl-loader -u -v -f genusers.conf. The
> same thing happens when I add the -r option.
>


What are saying the server's logs. Are there any errors at server side?

-- 
Truly,
Robert Iakobashvili, Ph.D.
......................................................................
Assistive technology that understands you
......................................................................

Re: TCP & curl-loader question

[Gary F. <ga...@in...>]

Thanks, Robert.

The URL_COMPLETION_TIMEOUT is set to 30 seconds (30000 ms), and  
nothing happens for the last 25 seconds. And timing out wouldn't  
explain why the server initiates the disconnect. If the client were  
timing out too soon, the client would initiate the disconnect, I would  
think. This certainly looks like a server problem. I wonder if the  
client could somehow be taught to reestablish the connection when the  
server disconnects, rather than timing out?

FWIW here's the configuration file. You can see some of the extensions  
I've built. But I doubt they're the cause of the problem, because they  
work to generate the URLs, and the server doesn't know anything about  
them.


On Nov 1, 2008, at 7:22 AM, Robert Iakobashvili wrote:

> Could it be, that your URL_COMPLETION_TIMEOUT value is too short
> and you need to increase it?
>
> It may happen, when POST-ing is accompanied by 100-Continue as seen  
> in your log file.

Re: TCP & curl-loader question

[Gary F. <ga...@in...>]

There's no indication of anything amiss in the server's logs,  
unfortunately.

Thanks,
Gary


On Nov 1, 2008, at 7:37 AM, Robert Iakobashvili wrote:

> What are saying the server's logs. Are there any errors at server  
> side?

Re: TCP & curl-loader question

[Robert I. <cor...@gm...>]

On Sat, Nov 1, 2008 at 4:47 PM, Gary Fitts <ga...@in...> wrote:

> There's no indication of anything amiss in the server's logs,
> unfortunately.
>
When both server and clients are happy,
this means, that tests passed sucessfully!

Make them happy! :-)

-- 
Truly,
Robert Iakobashvili, Ph.D.
......................................................................
Assistive technology that understands you
......................................................................

Re: TCP & curl-loader question

[Robert I. <cor...@gm...>]

Hi Gary,

On Sat, Nov 1, 2008 at 4:46 PM, Gary Fitts <ga...@in...> wrote:

> Thanks, Robert.
>
> The URL_COMPLETION_TIMEOUT is set to 30 seconds (30000 ms),


Y, it is OK

>
> FWIW here's the configuration file. You can see some of the extensions I've
> built. But I doubt they're the cause of the problem, because they work to
> generate the URLs, and the server doesn't know anything about them.


URL_TEMPLATE - it is the patch, that you have probably developed.

What happens, if instead you are using the existing URL facility?
Try it just to debug yourself.

Sincerely,
Robert

Re: TCP & curl-loader question

[Gary F. <ga...@in...>]

I did use the URL facility, and the same thing happens. I don't have  
that conf and log file around anymore. I can reconstruct them if you'd  
like to see them.


On Nov 1, 2008, at 8:40 AM, Robert Iakobashvili wrote:

> What happens, if instead you are using the existing URL facility?
> Try it just to debug yourself.

Re: TCP & curl-loader question

[Robert I. <cor...@gm...>]

hi Gary

On Sat, Nov 1, 2008 at 6:57 PM, Gary Fitts <ga...@in...> wrote:

> I did use the URL facility, and the same thing happens. I don't have
> that conf and log file around anymore. I can reconstruct them if you'd
> like to see them.
>

I have filtered the problematic TCP session from your capture by the filter
below in wireshark:

(ip.addr eq 65.105.205.215 and ip.addr eq 65.105.205.203) and (tcp.port eq
38507 and tcp.port eq 9000)


Client initiates closure: 38507 > cslistiner FIN-ACK Seq=5013
Server responds by ACK to that packet - ACK Ack=5014 (5014=5013+1)

Now server is expected to send FIN-ACK, but what it does is as he has a
SO_LINGER
option at his socket:
a) send the last data packet
b) afterwards sends FIN-ACK

So far so good, but client does not like (and this is also legal), that
server sends him
data packets after the FIN-ACK acknowledged by ACK and sends RST.

I would say, that nothing serious besides that the TCP-stacks of the server
and client
are tuned a bit different.

Is it a server with a Microsoft operating system?
-- 
Truly,
Robert Iakobashvili, Ph.D.
......................................................................
Assistive technology that understands you
......................................................................

Re: TCP & curl-loader question

[Gary F. <ga...@in...>]

Hello Robert,

Thanks for all your help so far. Now I've run into a possible bug, and  
I want to see what you think. In loader.c, setup_curl_handle_init(),  
there is this code:

   if (url->upload_file)
     {
       if (! url->upload_file_ptr)
         {
           if (! (url->upload_file_ptr = fopen (url->upload_file,  
"rb")))
             {
               fprintf (stderr,
                        "%s - error: failed to open() %s with errno %d. 
\n",
                        __func__, url->upload_file, errno);
               return -1;
             }
         }

       /* Enable uploading */
       curl_easy_setopt(handle, CURLOPT_UPLOAD, 1);
...


Now if this is a cycling URL, we're going to want to upload the file  
many times. But after the first upload, the file pointer will be set  
to the end of the file, and subsequent reads will return eof. So I  
changed the code to this:

   if (url->upload_file)
     {
       if (! url->upload_file_ptr)
         {
           if (! (url->upload_file_ptr = fopen (url->upload_file,  
"rb")))
             {
               fprintf (stderr,
                        "%s - error: failed to open() %s with errno %d. 
\n",
                        __func__, url->upload_file, errno);
               return -1;
             }
         }
       else
         {
         rewind(url->upload_file_ptr); /* Added by GF October 2008 */
         }

       /* Enable uploading */
       curl_easy_setopt(handle, CURLOPT_UPLOAD, 1);

This seems to fix the problem when a single client is cycling the url.  
But now I'm worried about multiple clients. As it is, all the clients  
share this url context, and so they all share this single  
upload_file_ptr. What if client A begins the upload, and while the  
upload is proceeding in stages, client B comes along and tries to use  
the same pointer? It seems that this could certainly happen when we're  
running clients in multiple threads, and it looks as though it could  
happen even with multiple clients in a single thread.

I'm thinking that each upload url has to keep a file offset for each  
client, and arrange to use that offset for each read. What do you think?

Thanks,
Gary

Re: TCP & curl-loader question

[Gary F. <ga...@in...>]

Sorry, I meant to say "they DON'T use up extra file descriptors".

> These streams are actually just file offsets, so they use up extra  
> file descriptors.

Re: TCP & curl-loader question

[Gary F. <ga...@in...>]

Hello Robert,

I think I've found a solution to this. I've patched curl-loader to  
maintain a separate "url upload stream" for each client, so that  
multiple clients cannot interfere with each other. (These streams are  
actually just file offsets, so they use up extra file descriptors.)  
This seems to have fixed the timeouts I was seeing before. I believe  
they were caused when the server was told to expect an upload of N  
bytes, but then received < N because the upload_file_ptr had been  
consumed by another client.

I'll send you the files, if you'd like, as soon as I get them properly  
commented.

Gary


> Thanks for all your help so far. Now I've run into a possible bug,  
> and I want to see what you think. In loader.c,  
> setup_curl_handle_init(), there is this code:
>
>  if (url->upload_file)
>    {
>      if (! url->upload_file_ptr)
>        {
>          if (! (url->upload_file_ptr = fopen (url->upload_file,  
> "rb")))
>            {
>              fprintf (stderr,
>                       "%s - error: failed to open() %s with errno %d. 
> \n",
>                       __func__, url->upload_file, errno);
>              return -1;
>            }
>        }
>
>      /* Enable uploading */
>      curl_easy_setopt(handle, CURLOPT_UPLOAD, 1);
> ...
>
>
> Now if this is a cycling URL, we're going to want to upload the file  
> many times. But after the first upload, the file pointer will be set  
> to the end of the file, and subsequent reads will return eof. So I  
> changed the code to this:
>
>  if (url->upload_file)
>    {
>      if (! url->upload_file_ptr)
>        {
>          if (! (url->upload_file_ptr = fopen (url->upload_file,  
> "rb")))
>            {
>              fprintf (stderr,
>                       "%s - error: failed to open() %s with errno %d. 
> \n",
>                       __func__, url->upload_file, errno);
>              return -1;
>            }
>        }
>      else
>        {
>        rewind(url->upload_file_ptr); /* Added by GF October 2008 */
>        }
>
>      /* Enable uploading */
>      curl_easy_setopt(handle, CURLOPT_UPLOAD, 1);
>
> This seems to fix the problem when a single client is cycling the  
> url. But now I'm worried about multiple clients. As it is, all the  
> clients share this url context, and so they all share this single  
> upload_file_ptr. What if client A begins the upload, and while the  
> upload is proceeding in stages, client B comes along and tries to  
> use the same pointer? It seems that this could certainly happen when  
> we're running clients in multiple threads, and it looks as though it  
> could happen even with multiple clients in a single thread.
>
> I'm thinking that each upload url has to keep a file offset for each  
> client, and arrange to use that offset for each read. What do you  
> think?
>
> Thanks,
> Gary

Re: TCP & curl-loader question

[Robert I. <cor...@gm...>]

Hi Gary,

On Mon, Nov 3, 2008 at 8:01 AM, Gary Fitts <ga...@in...> wrote:

> Hello Robert,
>
> I think I've found a solution to this. I've patched curl-loader to
> maintain a separate "url upload stream" for each client, so that
> multiple clients cannot interfere with each other. (These streams are
> actually just file offsets, so they use up extra file descriptors.)
> This seems to have fixed the timeouts I was seeing before. I believe
> they were caused when the server was told to expect an upload of N
> bytes, but then received < N because the upload_file_ptr had been
> consumed by another client.
>
> I'll send you the files, if you'd like, as soon as I get them properly
> commented.
>
> Gary
>
>
>
>

Sounds great!

If you could also update the man page regarding the new tag,
it would bwe very much appreciated.

Truly,
Robert