Again thanks for the very solid software you built. I was able to get my ETL tasks up and running within a week, and had no major issues since 6 month. Great!
I do not know how to classify this request: bug report, feature request or just proof of my own stupidity. Maybe you can help me?
What I want to do:
I have a simple postal-adresses Table in my RDBMS and I want to add latitude/longitude data from the google geocoding api. My table (adresses) has five columns: id, name, address, lat, lon. (address already beeing an aggregate of country, zip code and street). The Google geocoding syntax is https://maps.googleapis.com/maps/api/geocode/outputFormat?parameters where outputFormat = xml and parameters are address=xxx&key=yyy (key beeing a valid google api key). The google api only accepts GET method, so I cannot put the request to the body part, but instead I need to use http parameters.
What I already tried:
I created a model and imported all data from the RDBMS into a model-based message. I have created a ressource and did set up the webrequest component for a static sample (adding key:yyy and address:some+street+smalltown+germany to the "http parameters" section. Both works fine, the result is a xml fort this particular address that can be converted to a new model based message.
What I cannot do:
No matter what I tried, I was not able to use the data stored in model based-messages to influence the http parameters of the webrequest component. This component only accepts text based messages, so I tried to serialize, I also tried to manually feed Text constants to the webrequest. I dont completely understand the concept of "parameters" in mETL, but I tried to use different syntax taken out of the sample flows, all without success.
My questions:
Is it possible to use the webrequest component for once-per-row GET api calls?
How do I have to prepare my inbound message?
How can I have a reference to the original id field in my outbound message? (google won't forward this information in its response).
Please add some documentation about flow parameters (usage and referencing).
EDIT: I am still on v1.5.5
Thank you and best regards,
Alexander, Vienna, Austria
Last edit: A Hromadka 2017-07-11
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Right now the Web component only takes text messages. The thought was that you would always format your body in one of the formatters (i.e. XML formatter) and then send the resulting text to the web services component. So, you would have an RDBMS reader to an XML formatter to the web service caller. Unfortunately, right now the 1.5.5 version does not support paratemer replacement in the http parameters (i just added parameter replacement in the http headers for 2.0.7). I'll add an enhancement request to do the same for http parameters.
Now - for trying to do that all in 1.5.5... Can't immediately think of a way to do it in that version. You could write a custom component - see this link to understand how to write your own components (this doc is based on metl 2.x though). I'll let you know when I complete the enhancement above.
Thanks,
Greg W.
Last edit: Greg Wilmer 2017-07-12
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Dear mETL Team!
Again thanks for the very solid software you built. I was able to get my ETL tasks up and running within a week, and had no major issues since 6 month. Great!
I do not know how to classify this request: bug report, feature request or just proof of my own stupidity. Maybe you can help me?
What I want to do:
I have a simple postal-adresses Table in my RDBMS and I want to add latitude/longitude data from the google geocoding api. My table (adresses) has five columns: id, name, address, lat, lon. (address already beeing an aggregate of country, zip code and street). The Google geocoding syntax is https://maps.googleapis.com/maps/api/geocode/outputFormat?parameters where outputFormat = xml and parameters are address=xxx&key=yyy (key beeing a valid google api key). The google api only accepts GET method, so I cannot put the request to the body part, but instead I need to use http parameters.
What I already tried:
I created a model and imported all data from the RDBMS into a model-based message. I have created a ressource and did set up the webrequest component for a static sample (adding key:yyy and address:some+street+smalltown+germany to the "http parameters" section. Both works fine, the result is a xml fort this particular address that can be converted to a new model based message.
What I cannot do:
No matter what I tried, I was not able to use the data stored in model based-messages to influence the http parameters of the webrequest component. This component only accepts text based messages, so I tried to serialize, I also tried to manually feed Text constants to the webrequest. I dont completely understand the concept of "parameters" in mETL, but I tried to use different syntax taken out of the sample flows, all without success.
My questions:
EDIT: I am still on v1.5.5
Thank you and best regards,
Alexander, Vienna, Austria
Last edit: A Hromadka 2017-07-11
Right now the Web component only takes text messages. The thought was that you would always format your body in one of the formatters (i.e. XML formatter) and then send the resulting text to the web services component. So, you would have an RDBMS reader to an XML formatter to the web service caller. Unfortunately, right now the 1.5.5 version does not support paratemer replacement in the http parameters (i just added parameter replacement in the http headers for 2.0.7). I'll add an enhancement request to do the same for http parameters.
Now - for trying to do that all in 1.5.5... Can't immediately think of a way to do it in that version. You could write a custom component - see this link to understand how to write your own components (this doc is based on metl 2.x though). I'll let you know when I complete the enhancement above.
Thanks,
Greg W.
Last edit: Greg Wilmer 2017-07-12