I make an RPC call from a client to a server. The server queries the database, and returns a large number of objects. I return over 33,000 objects. The query takes less than 2 minutes to return the objects, but the XML RPC conversion takes about 22 minutes !
Has anybody tried something like this ?
Is it normal to take that long ?
Any suggestions on how to improve this conversion ?
thanks,
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I once needed to communicate ~50,000 objects of a quite a large class I wrote.
What I did, was have the XmlRpcSeverMethod return std::string, which is the xml manifastation of the list of object.
On the client side, I received the std::string and fromXml()-ed it back to objects.
Could that work for you?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for your reply.
The objects I return contain multiple attributes (for example an object could have 5 attributes: one bool, two strings, two integers and one array). Of course, they have variable number of attributes.
Therefore I would have to transform all this information into strings, which adds overhead....
How fast was the XML conversion in your case ?
Could you transfer the 50,000 objects in, lets say, 60 seconds or less ?
I am thinking myself, to stream all these objects using the std::ostrstream , which means they get buffered.
After that, to read from the buffer into the XmlRpcValue::BinaryData and transport it this way.
I am not sure how fast it is though ....
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I used apache's xerces-c to parse XML documents and elements, so the conversion itself took very little time, and is a matter of micro seconds per object. The conversion back to objects using xerces-c took about the same.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
IMHO, one must notice that during conversion from XML to objects multiple requests of memory allocations are performed. So without proper memory control mechanism it is impossible to significally reduce needed time.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
I make an RPC call from a client to a server. The server queries the database, and returns a large number of objects. I return over 33,000 objects. The query takes less than 2 minutes to return the objects, but the XML RPC conversion takes about 22 minutes !
Has anybody tried something like this ?
Is it normal to take that long ?
Any suggestions on how to improve this conversion ?
thanks,
I once needed to communicate ~50,000 objects of a quite a large class I wrote.
What I did, was have the XmlRpcSeverMethod return std::string, which is the xml manifastation of the list of object.
On the client side, I received the std::string and fromXml()-ed it back to objects.
Could that work for you?
Danny,
Thanks for your reply.
The objects I return contain multiple attributes (for example an object could have 5 attributes: one bool, two strings, two integers and one array). Of course, they have variable number of attributes.
Therefore I would have to transform all this information into strings, which adds overhead....
How fast was the XML conversion in your case ?
Could you transfer the 50,000 objects in, lets say, 60 seconds or less ?
I am thinking myself, to stream all these objects using the std::ostrstream , which means they get buffered.
After that, to read from the buffer into the XmlRpcValue::BinaryData and transport it this way.
I am not sure how fast it is though ....
I used apache's xerces-c to parse XML documents and elements, so the conversion itself took very little time, and is a matter of micro seconds per object. The conversion back to objects using xerces-c took about the same.
IMHO, one must notice that during conversion from XML to objects multiple requests of memory allocations are performed. So without proper memory control mechanism it is impossible to significally reduce needed time.