Re: TCP & curl-loader question

[Gary F. <ga...@in...>]

Hello Robert,

I think I've found a solution to this. I've patched curl-loader to  
maintain a separate "url upload stream" for each client, so that  
multiple clients cannot interfere with each other. (These streams are  
actually just file offsets, so they use up extra file descriptors.)  
This seems to have fixed the timeouts I was seeing before. I believe  
they were caused when the server was told to expect an upload of N  
bytes, but then received < N because the upload_file_ptr had been  
consumed by another client.

I'll send you the files, if you'd like, as soon as I get them properly  
commented.

Gary


> Thanks for all your help so far. Now I've run into a possible bug,  
> and I want to see what you think. In loader.c,  
> setup_curl_handle_init(), there is this code:
>
>  if (url->upload_file)
>    {
>      if (! url->upload_file_ptr)
>        {
>          if (! (url->upload_file_ptr = fopen (url->upload_file,  
> "rb")))
>            {
>              fprintf (stderr,
>                       "%s - error: failed to open() %s with errno %d. 
> \n",
>                       __func__, url->upload_file, errno);
>              return -1;
>            }
>        }
>
>      /* Enable uploading */
>      curl_easy_setopt(handle, CURLOPT_UPLOAD, 1);
> ...
>
>
> Now if this is a cycling URL, we're going to want to upload the file  
> many times. But after the first upload, the file pointer will be set  
> to the end of the file, and subsequent reads will return eof. So I  
> changed the code to this:
>
>  if (url->upload_file)
>    {
>      if (! url->upload_file_ptr)
>        {
>          if (! (url->upload_file_ptr = fopen (url->upload_file,  
> "rb")))
>            {
>              fprintf (stderr,
>                       "%s - error: failed to open() %s with errno %d. 
> \n",
>                       __func__, url->upload_file, errno);
>              return -1;
>            }
>        }
>      else
>        {
>        rewind(url->upload_file_ptr); /* Added by GF October 2008 */
>        }
>
>      /* Enable uploading */
>      curl_easy_setopt(handle, CURLOPT_UPLOAD, 1);
>
> This seems to fix the problem when a single client is cycling the  
> url. But now I'm worried about multiple clients. As it is, all the  
> clients share this url context, and so they all share this single  
> upload_file_ptr. What if client A begins the upload, and while the  
> upload is proceeding in stages, client B comes along and tries to  
> use the same pointer? It seems that this could certainly happen when  
> we're running clients in multiple threads, and it looks as though it  
> could happen even with multiple clients in a single thread.
>
> I'm thinking that each upload url has to keep a file offset for each  
> client, and arrange to use that offset for each read. What do you  
> think?
>
> Thanks,
> Gary