From: Marty K. <mrk...@co...> - 2011-09-02 16:36:47
|
On 09/02/2011 11:36 AM, jam...@di... wrote: > Hi Greg > > We're talking about the same thing. You can't have hetrogenous arrays of structures that change from message to message without sending the schema with for each message, however that's certainly possible with PVData, and I'm using it now. About the user interface, the user gets an interface like the DOM. There is a method PVStructure::getSubField(int) which will return a numbered field of your structure. If you had a native hetrogenous array of structures the interface would be called something like PVHetrogenousStructureArray::getElement(int) but the wire format already supports what you want. You could add a new type code for PVHetrogenousStructureArray and save a little space by not serializing dummy names for the indices but I wouldn't bother. > > Now I can't quite tell from the ChannelRPC message documentation if the schema is sent with each reply message, that's the important bit. I thought we decided that we would do that for precisely this use case. > Yes the introspection as well as the data is sent with each reply. Marty > James > > > -----Original Message----- > From: White, Greg [mailto:gr...@sl...] > Sent: 02 September 2011 16:09 > To: Rowland, James (DLSLtd,RAL,DIA); mrk...@co...; epi...@li... > Subject: RE: pvDataCPP implementations > > Hi James, > > Sure, of course, one can define a structure to contain structures, but then the end user's > interface is completely different. Secondly, as you say, it's fixed length and in fact fixed > structure. From what you've written, I don't think we're talking about the same class > of use cases. I'm thinking about the common class we've found, of which returning data > from *any* SQL database query is a good example. Until runtime, you don't know the > number of rows, nor the number or type of columns that must be returned. Server side code has to > dynamically construct the single return structure, composed of either, an array of > structures (little used we found), or an array of differently typed arrays. In a full > implementation, the leaf node structures, may themselves be structures (that is, one of > the db table columns is of structure type). My point is, what's returned was dynamically > constructed, and heterogenous. > > This is the primary use case I'm aiming at. This pattern is probably the most used service > API data exchange pattern in our services layer at LCLS. It has to be just right in pvData. > > Cheers > Greg > ________________________________________ > From: jam...@di... [jam...@di...] > Sent: Friday, September 02, 2011 7:40 AM > To: White, Greg; mrk...@co...; epi...@li... > Subject: RE: pvDataCPP implementations > > Regarding the hetrogenous array of structures, we can already talked about doing this with PVData like so: > > structure HetrogenousSortOfArray > { > Structure Item1 > { > Whatever you like > } > > Structure Item2 > { > Completely different structure > } > } > > The disadvantage is that this is a fixed-length hetrogenous array. But if you always send the schema (is that a good name for the PVData type information?) then that doesn't matter. > > Regarding not using ICE or EXI, I was able to roll my own PVData packer and unpacker from the nice spec in a day to explore the issues surrounding zero-copy for this data acquisition. The simplicity of implementation and ability to experiment is a big win for me. I agree for services it doesn't matter, I stuck with XMLRPC for our Archiver. > > James > > -----Original Message----- > From: White, Greg [mailto:gr...@sl...] > Sent: 02 September 2011 15:23 > To: Marty Kraimer; epi...@li... > Subject: RE: pvDataCPP implementations > > Hi Marty, > > Responding to your information about heterogeneous arrays of structures, rather than James' specifc point. > > AIDA supports both homogenous array (what you've done, with one introspection interface), and heterogeneous array as well. > > Both are built on top of CORBA Any. If I understand what you're calling the introspection interface, neither of these use one. > I'm thinking what you're calling the introspection interface is what in CORBA is the shared IDL binary output > product for a given user defined structure. And you are putting that binary on the wire so the other side can > decode what you give it. > > In AIDA, I rather let CORBA send the metadata for decoding with the data every time (so it's no less > efficient than what you're doing, except I guess you may be caching introspection interfaces you've seen > before - in which case, I wonder how you are keying them? This a problem AIDA had in the early days, until > I just disabled caching. I've never turned it back on). > > In practice 95% of the structures exchanged in AIDA are dynamically encoded and deserialized at runtime from > deserializaiton metadata that came with the data, using one of these AIDA structure interfaces (heterogenoeous or > homogeneous array of structures, implemented over CORBA Any). It's slower than we would get with statically defined > structures, but really, not so slow that it makes a significant difference for services. > Bear in mind, services tend to be called very little (100s of times per day, not 1000s of times per second). > AIDA roundtrip for a dynamically defined type, for a simple example, is< 10ms, probably only 3 or 4 ms after VM warmup [1]. > > The way we do tables, is precisely, to return a heterogenous array of strutures, where each > element of the structure is a structure that defines a table column. > > The remaining 5% of exchanged structures are done the statically, out of band, introspection > interface way. That is, by use of IDL. > > What this is making me think, is, firstly, I'm a little worried that we're giving up heterogenous array of structures > a little too easily. > > Secondly, I really hope defining our own data exchage protocol is the right thing to do. I'm not ready to say that > I'm wondering if we should be looking at wrapping ICE or EXI over HTTP. I know that's a huge step. But I'd like to be > clearer on why we are not. > > Cheers > Greg > > [1] http://www.slac.stanford.edu/grp/cd/soft/aida/perf/Report.html > > ________________________________________ > From: Marty Kraimer [mrk...@co...] > Sent: Friday, September 02, 2011 4:38 AM > To: epi...@li... > Subject: Re: pvDataCPP implementations > > On 09/02/2011 03:29 AM, jam...@di... wrote: >> Hi Matej >> >> For a test I have written a minimal C++ pvData implementation using a callback API for the unpacking but it's easy to change. > Are You using eclipse as Your development environment? > Also how does this relate to pvManager? >> What are you going to do? > Sorry I do not understand what You are asking. >> I have a question about the pvAccess protocol spec document, is the bitset example wrong? Looking at the code I don't think the bitmap is propagated to the members of structure arrays, structure array elements can be null but their fields can't be nulled individually. Is that correct? > Each element of a structureArray is a top level structure AND every > element has the same introspection interface. > Also the current implementation does allow null elements. I am not sure > if this should be allowed. > > This is a "design feature". > > Lets use the following as an example. > > > structure top > ... > structure[] someStructure //NOTE EVERY ELEMENT MUST HAVE THE SAME > introspection interface > > When a client calls createGet, createPut, etc the server can sent the > client an immutable introspection interface that defines top. > The client side of pvAccess can then create a PVStructure that holds > the data that is passed between client and server. > If the introspection interface is mutable then it cause major > complications AND overhead in passing data. > > Note that the above even allows recursive structure arrays: > > structure top > ... > structure[] someStructure //NOTE EVERY ELEMENT MUST HAVE THE SAME > introspection interface > ... > structure[] someOtherStructure > > > The original javaIOC defined things differently. It defined: > > enum Type { > pvBoolean, > pvByte, pvShort, pvInt, pvLong, > pvFloat,pvDouble, > pvString, > pvStructure, > } > > and then allowed an array element to be any of the above types. > > For an array of structures each element could have a different introspection interface. > This caused major problems even before Matej implemented pvAccess. > With pvAccess the complication was just too great. > > I actually completely removed arrays of structures. > There were objections raised by folks at BNL. > I then realized that by creating a type structureArray where every element had the same introspection interface then the top level introspection interface is immutable. > The major problems went away. > > This provided a lot of what is desired in arrays of structures without the severe problems. > Granted it is not as general as what originally was supported but there was a good reason. > > Marty > > >> James >> >> -----Original Message----- >> From: Matej Sekoranja [mailto:mat...@co...] >> Sent: 01 September 2011 15:25 >> To: epi...@li... >> Subject: pvDataCPP implementations >> >> Hi, >> >> we have C++ and Java implementation. Are there any more? >> I am considering a minor change in array serialization. >> >> Matej >> >> ------------------------------------------------------------------------------ >> Special Offer -- Download ArcSight Logger for FREE! >> Finally, a world-class log management solution at an even better >> price-free! And you'll get a free "Love Thy Logs" t-shirt when you >> download Logger. Secure your free ArcSight Logger TODAY! >> http://p.sf.net/sfu/arcsisghtdev2dev >> > > ------------------------------------------------------------------------------ > Special Offer -- Download ArcSight Logger for FREE! > Finally, a world-class log management solution at an even better > price-free! And you'll get a free "Love Thy Logs" t-shirt when you > download Logger. Secure your free ArcSight Logger TODAY! > http://p.sf.net/sfu/arcsisghtdev2dev > > ------------------------------------------------------------------------------ > Special Offer -- Download ArcSight Logger for FREE! > Finally, a world-class log management solution at an even better > price-free! And you'll get a free "Love Thy Logs" t-shirt when you > download Logger. Secure your free ArcSight Logger TODAY! > http://p.sf.net/sfu/arcsisghtdev2dev > > -- > This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. > Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom > > > > > |