curl-loader-devel

Re: Is it possible to use data from the response header to a POST in a subsequent GET?

[Gary F. <ga...@in...>]

Hello list,

There seems to have been no followup to this post from last
November. I have a similar need, I know C and I'm willing to
write a patch. Has anyone tried something similar?
Any guidance?

Thanks,
Gary Fitts


> From: Robert Iakobashvili <coroberti@gm...> -	2007-11-26 20:56
> Hi Ken,
>
> On Nov 26, 2007 10:46 PM, Ken Mamitsuka <kenm@sh...> wrote:
>
> > Hi, we have a web app that first takes a POST to a generic URL and  
> returns
> > an id in the POST response header. I'd like to use curl-loader to  
> grab that
> > unique id and the subsequently PUT to that id. >From a straight curl
> > perspective, it'd look like:
>
> If you know C and willing to write a patch, I can guide you about  
> what is
> necessary to do for that.

Re: Is it possible to use data from the response header to a POST in a subsequent GET?

[Robert I. <cor...@gm...>]

Hi Gary,

On Thu, Oct 2, 2008 at 9:16 PM, Gary Fitts <ga...@in...> wrote:
> Hello list,
> There seems to have been no followup to this post from last
> November. I have a similar need, I know C and I'm willing to
> write a patch. Has anyone tried something similar?

Where exactly is the text you need to grab? I an header or
in body. Please, a bit more details.

> Any guidance?

I will guide you.

> Thanks,
> Gary Fitts

Truly,
Robert Iakobashvili, Ph.D.
......................................................................
Assistive technology that understands you
......................................................................

Re: Is it possible to use data from the response header to a POST in a subsequent GET?

[Gary F. <ga...@in...>]

> Where exactly is the text you need to grab? I an header or
> in body. Please, a bit more details.

I'd like to do two things. Both involve constructing client-specific  
URLS.
If the second is too hard, I could probably make do with the first.

1. Client-specific urls built from tokens in a file. For example, here's
some possible syntax for the configuration file:

URL=http://a.xyz.com/users/%s/profiles/%s/subscriptions
URL_TOKEN_FILE=foo

In this case, each client would take two tokens from foo and use them  
to build the
client-specific URL.

Or, it might be simpler to build a file of complete client-specific  
URLs in advance,
and take whole URLs from the file. Maybe that could look like this in  
the config file:

URL=file("foo")


2. Build client-specific URLs from tokens returned in the body of  
previous responses.
For instance, here's a sample client-server exchange (with most  
headers omitted):

POST /login HTTP/1.1

login=gu...@in...&password=guest

HTTP/1.1 200 OK

<?xml version="1.0" encoding="UTF-8"?>
<user user_id="38" profile_id="3"/>

PUT /users/38/profiles/3/subscriptions HTTP/1.1

stuff ...

In this case I'd like to be able to grab user_id and profile_id from  
the returned
XML, and use these tokens to build the subsequent URL.
Maybe the config file could look something like this:

URL=http://a.xyz.com/login
RESPONSE_TOKEN=user_id
RESPONSE_TOKEN= profile_id

URL=http://a.xyz.com/users/<user_id>/profiles/<profile_id>/subscriptions

I realize that it would be hard to make this completely general, but  
I'm sure
I could write response-parsing code specific to my needs.


Any guidance would be appreciated!
Gary



On Oct 2, 2008, at 11:28 AM, Robert Iakobashvili wrote:

> Hi Gary,
>
> On Thu, Oct 2, 2008 at 9:16 PM, Gary Fitts <ga...@in...>  
> wrote:
>> Hello list,
>> There seems to have been no followup to this post from last
>> November. I have a similar need, I know C and I'm willing to
>> write a patch. Has anyone tried something similar?
>
> Where exactly is the text you need to grab? I an header or
> in body. Please, a bit more details.
>
>> Any guidance?
>
> I will guide you.
>
>> Thanks,
>> Gary Fitts

Re: Is it possible to use data from the response header to a POST in a subsequent GET?

[Robert I. <cor...@gm...>]

Hi Gary,

On Thu, Oct 2, 2008 at 11:28 PM, Gary Fitts <ga...@in...> wrote:

>
> 1. Client-specific urls built from tokens in a file. For example, here's
> some possible syntax for the configuration file:
>
> URL=http://a.xyz.com/users/%s/profiles/%s/subscriptions
> URL_TOKEN_FILE=foo
>
> In this case, each client would take two tokens from foo and use them to
> build the
> client-specific URL.
>
> Or, it might be simpler to build a file of complete client-specific URLs in
> advance,
> and take whole URLs from the file. Maybe that could look like this in the
> config file:
>
> URL=file("foo")
>

Sounds great! Several people have requested something like this, and you
have formulated
the requirements.

Let's call it

URL_SET_TEMPLATE, which will have any number of %s symbols in any places of
its url template,
which will  be read from a file
URL_SET_TOKEN_FILE

It presumes, that all other tags for the set URL will be the same.

Files parse_conf.c contains
tag_parser_pair_tp_map
where the two new tag can be easily added with their handling functions.

Now the matters to care about.
URLS_NUM from general section should contain number of total URLs, which can
come
from individual URL tags as well as from URL_SET_TEMPLATE.

parser of URL_SET_TEMPLATE does not allocate a URL by itself, but instead
should
read the template and sets a flag to wait for the next URL_SET_TOKEN_FILE.

parser of URL_SET_TOKEN_FILE will create N URL objects, where N is the
number of
rows in the file. It may keep in the batch object the two numbers: where the
SET starts,
and the number of the URLs in the set (N) . Each next tag of the URL section
(e.g.
REQUEST_TYPE, TIMER_AFTER_URL_COMPLETION, etc) should be applied to all the
URL objects, which to be signaled to the parsers of the tags by the non-zero
N, or by a flag,
as you wish.

The next URL or  URL_SET_TEMPLATE tags should care about advancing a field
in batch
object

Therefore, in url_parser (and in url_set_template_parser) instead the
current

bctx->url_index++;

should appear something like this:

if (!N)
{
   bctx->url_index++;
}
else
{
   url_index += N;
   N = 0;
}


>
> 2. Build client-specific URLs from tokens returned in the body of previous
> responses.
> For instance, here's a sample client-server exchange (with most headers
> omitted):
>
> POST /login HTTP/1.1
>
> login=gu...@in...&password=guest
>
> HTTP/1.1 200 OK
>
> <?xml version="1.0" encoding="UTF-8"?>
> <user user_id="38" profile_id="3"/>
>
> PUT /users/38/profiles/3/subscriptions HTTP/1.1
>
> stuff ...
>
>
> In this case I'd like to be able to grab user_id and profile_id from the
> returned
> XML, and use these tokens to build the subsequent URL.
> Maybe the config file could look something like this:
>
> URL=http://a.xyz.com/login
> RESPONSE_TOKEN=user_id
> RESPONSE_TOKEN= profile_id
>
> URL=http://a.xyz.com/users/<user_id>/profiles/<profile_id>/subscriptions
>
> I realize that it would be hard to make this completely general, but I'm
> sure
> I could write response-parsing code specific to my needs.
> Any guidance would be appreciated!
> Gary
>

It could be made in generic fashion.

Function client_tracing_function () filters all bytes from headers as well
as from bodies.
Incoming body bytes are passing switch
CURLINFO_DATA_IN

cctx - (client context) object can have a set of filters and parse the
passing body bytes
to extract them. Take care however about partial token, e.g. by keeping
several last
bytes from the previous packet, or etc method.

A suggestion is for you to look at the existing mechanism delivered
by URL_USE_CURRENT.

More questions/ suggestions are welcomed!
-- 
Truly,
Robert Iakobashvili, Ph.D.
......................................................................
www.ghotit.com
Assistive technology that understands you
......................................................................

Re: Is it possible to use data from the response header to a POST in a subsequent GET?

[Gary F. <ga...@in...>]

Hello Robert,

I'm finishing my URL_SET_TEMPLATE code. The last step is to grab the  
appropriate URL from the url-set just before the client needs it. I  
suspect that the place to do this is in load_fsm.c : setup_url(). Do  
you agree that this is the right place?

Thanks,
Gary

Re: Is it possible to use data from the response header to a POST in a subsequent GET?

[Gary F. <ga...@in...>]

Hello Robert,

Yes, I saw your previous message, and I posted a followup question. My  
reply is on the archive website

http://sourceforge.net/mailarchive/forum.php?thread_name=A01460E9-2C40-4BA3-A536-C67563D53C64%40inTouchGroup.com&forum_name=curl-loader-devel

(Meanwhile, an older query that I posted on September 29 has finally  
appeared. This is what you you're responding to here. I had given up  
and reposted my question, and that's the conversation we've been  
having.)

I'll repeat my latest question:

I'm finishing my URL_SET_TEMPLATE code. The last step is to grab the  
appropriate URL from the url-set just before the client needs it. I  
suspect that the place to do this is in load_fsm.c : setup_url(). Does  
this look right to you?

Thanks,
Gary


> Re: Help creating a patch for variable URLs
> From: Robert Iakobashvili <coroberti@gm...> -	2008-10-05 04:52
>
> Attachments: Message as HTML
> Hi Gary,
>
>
> Have you seen my previous message?

Re: Is it possible to use data from the response header to a POST in a subsequent GET?

[Robert I. <cor...@gm...>]

On Thu, Oct 9, 2008 at 11:18 PM, Gary Fitts <ga...@in...> wrote:

> Hello Robert,
>
>
> I'm finishing my URL_SET_TEMPLATE code. The last step is to grab the
> appropriate URL from the url-set just before the client needs it. I suspect
> that the place to do this is in load_fsm.c : setup_url(). Does this look
> right to you?
>


Sorry, no.

setup_url just switches URLs upon loading.

All configuration job is to be done in parse_conf.c
Please, try to look thru the previous instructions and
if any questions, just ask them.

A point. that was missed is the URLs name to appear
as a URL-name for each URL of the set.
When dealing  with a SET, you may take a URL_NAME
specified name just as a base and add e.g. some number,
like
"MyURLSet-0"
"My URLSet-1",

....


> Thanks,
> Gary
>

Best wishes and appreciating the job, that you are doing.


Truly,
Robert Iakobashvili, Ph.D.
......................................................................
www.ghotit.com
Assistive technology that understands you
......................................................................

Re: Is it possible to use data from the response header to a POST in a subsequent GET?

[Gary F. <ga...@in...>]

Hello Robert,

I have my changes working now, although I want to test them a bit more  
before letting them out. I'm having one problem, though. In the  
application I'm testing, the very first URL is a POST request using a  
FORM_STRING.

This seems to cause a failure in loader.c, setup_curl_handle_appl().  
The code is copied below.

When this code is called, cctx->post_data has been allocated but not  
initialized, so it's an empty string. This causes the condition "else  
if (cctx->post_data && cctx->post_data[0])" to fail, and we drop  
through to the "post_data is NULL" error.

I've fixed this by commenting out the second condition:
	else if (cctx->post_data /* && cctx->post_data[0] */)

So far this seems to work for me, but I wonder if I'm missing  
something that might cause an error somewhere else.


if (url->req_type == HTTP_REQ_TYPE_POST)
   {
     /*
        Make POST, using post buffer, if requested.
     */
       if (url->upload_file && url->upload_file_ptr && (!cctx- 
 >post_data || !cctx->post_data[0]))
       {
           curl_easy_setopt(handle, CURLOPT_POST, 1);
       }
       else if (cctx->post_data && cctx->post_data[0])
       {
         /*
            Sets POST as the HTTP request method using either:
            - POST-fields;
            - multipart form-data as in RFC 1867;
         */
         if (init_client_url_post_data (cctx, url) == -1)
           {
             fprintf (stderr,
                      "%s - error: init_client_url_post_data() failed. 
\n",
                      __func__);
             return -1;
           }
       }
       else
       {
         fprintf (stderr, "%s - error: post_data is NULL.\n", __func__);
         return -1;
       }
   }


Thanks,
Gary

Re: Is it possible to use data from the response header to a POST in a subsequent GET?

[Robert I. <cor...@gm...>]

Hi Gary

On Mon, Oct 27, 2008 at 8:13 PM, Gary Fitts <ga...@in...> wrote:

> Hello Robert,
>
> I have my changes working now, although I want to test them a bit more
> before letting them out. I'm having one problem, though. In the
> application I'm testing, the very first URL is a POST request using a
> FORM_STRING.


Great!

>
>
> This seems to cause a failure in loader.c, setup_curl_handle_appl().
> The code is copied below.
>
> When this code is called, cctx->post_data has been allocated but not
> initialized, so it's an empty string. This causes the condition "else
> if (cctx->post_data && cctx->post_data[0])" to fail, and we drop
> through to the "post_data is NULL" error.
>
> I've fixed this by commenting out the second condition:
>        else if (cctx->post_data /* && cctx->post_data[0] */)
>
> So far this seems to work for me, but I wonder if I'm missing
> something that might cause an error somewhere else.
>

Actually, when cctx->post_data is allocated,
it should be set zero to the first char, like
cctx->post_data[0] = '\0';

You may wish add this fix to your patches.
Thanks.

Robert

Re: Is it possible to use data from the response header to a POST in a subsequent GET?

[Gary F. <ga...@in...>]

Thanks for replying, Robert.

> Actually, when cctx->post_data is allocated,
> it should be set zero to the first char, like
> cctx->post_data[0] = '\0';


Yes, cctx->post_data[0] is set to zero, and that's the problem. This  
causes the condition "else if (cctx->post_data && cctx->post_data[0])"  
to fail, and to drop through to the "post_data is NULL" error, which  
aborts the batch run.

When I comment out the second part: "else if (cctx->post_data /* &&  
cctx->post_data[0] */)", we call init_client_url_post_data, and this  
seems to work.


if (url->req_type == HTTP_REQ_TYPE_POST)
  {
    /*
       Make POST, using post buffer, if requested.
    */
      if (url->upload_file && url->upload_file_ptr && (!cctx- 
 >post_data || !cctx->post_data[0]))
      {
          curl_easy_setopt(handle, CURLOPT_POST, 1);
      }
      else if (cctx->post_data && cctx->post_data[0])
      {
        /*
           Sets POST as the HTTP request method using either:
           - POST-fields;
           - multipart form-data as in RFC 1867;
        */
        if (init_client_url_post_data (cctx, url) == -1)
          {
            fprintf (stderr,
                     "%s - error: init_client_url_post_data() failed. 
\n",
                     __func__);
            return -1;
          }
      }
      else
      {
        fprintf (stderr, "%s - error: post_data is NULL.\n", __func__);
        return -1;
      }
  }


Thanks,
Gary

Re: Is it possible to use data from the response header to a POST in a subsequent GET?

[Robert I. <cor...@gm...>]

On Mon, Oct 27, 2008 at 8:38 PM, Gary Fitts <ga...@in...> wrote:

> Thanks for replying, Robert.
>
> > Actually, when cctx->post_data is allocated,
> > it should be set zero to the first char, like
> > cctx->post_data[0] = '\0';
>
>
> Yes, cctx->post_data[0] is set to zero, and that's the problem. This
> causes the condition "else if (cctx->post_data && cctx->post_data[0])"
> to fail, and to drop through to the "post_data is NULL" error, which
> aborts the batch run.
>
> When I comment out the second part: "else if (cctx->post_data /* &&
> cctx->post_data[0] */)", we call init_client_url_post_data, and this
> seems to work.
>


We had a one more user suggesting such fix.
Y may do it, and I will try to remember, why it was actually added.
Thanks.

-- 
Truly,
Robert Iakobashvili, Ph.D.
......................................................................
www.ghotit.com
Assistive technology that understands you
......................................................................

Re: Is it possible to use data from the response header to a POST in a subsequent GET?

[Robert I. <cor...@gm...>]

Hi Gary,

On Mon, Oct 27, 2008 at 8:38 PM, Gary Fitts <ga...@in...> wrote:

> Thanks for replying, Robert.
>
> > Actually, when cctx->post_data is allocated,
> > it should be set zero to the first char, like
> > cctx->post_data[0] = '\0';
>
>
> Yes, cctx->post_data[0] is set to zero, and that's the problem. This
> causes the condition "else if (cctx->post_data && cctx->post_data[0])"
> to fail, and to drop through to the "post_data is NULL" error, which
> aborts the batch run.
>
> When I comment out the second part: "else if (cctx->post_data /* &&
> cctx->post_data[0] */)", we call init_client_url_post_data, and this
> seems to work.
>


Committed your fix.
You and Alex, which reported this issue previously, have been added
to our THANKS list.

Thank you very much!

-- 
Truly,
Robert Iakobashvili, Ph.D.
......................................................................
Assistive technology that understands you
......................................................................