RE: proposal for implementing unsigned

[<dav...@di...>]

What I said I'd do at last week's meeting was investigate whether it was possible to implement the interface we agreed at Brookhaven and so that's what I've done.

Although it's hardly the end of the world if we can't implement that interface, we did spend some time getting an agreement on it and I think we had strong arguments for the decisions we made, so if at all possible I think we should stick with it.

Marty had reservations about the possibility of implementing the interface, about the impact on the serialization and the consistency of the interfaces. I think we quickly established last week that there was no impact on serialization and I think it was also pretty clear that there weren't any implementation difficulties since the interfaces are essentially the same, but with two functions renamed (getWider->get and get->getRaw). In any case having looked at this it's clear to me that there are no issues. The only problem I wasn't able to see the answer to in the meeting was the issue of the consistency of interfaces. Having looked at this, as I think was clear in my proposal, there are no issues with the consistency of interfaces either, so we can implement the interface we agreed if we still want this.

Sorry if I've confused things by suggesting possible enhancements to the interface. I do think there is need for this, but perhaps this is a another issue and should be discussed separately. The functions I suggested were optional except for additions to the unsigned types to provide equivalent functions for the narrower type (getRaw), so don't affect the decision.

We had sound reasons for making the wider type the return type of the default get function.

If the narrower, signed type is used a user there is an issue when the user calls the function when the true value is greater than the max signed int and value raw value is negative. The user has to understand to test for negative numbers or else the function has to throw an exception half the time and the user has to trap and handle this. My understanding was that we weren't going to have overflows generating exceptions, but from Marty's email he seem to be advocating the exception.

If we make the wider type the return type a user familiar with calling get on a signed integer PV can call get on am unsigned PV, safely, without having to do any of this.

If a user wants the signed value he can get it via the getRaw function. The name should act as a warning to the user that the value is negative, or at least that the behaviour might be different from the get() function that they may be used to. Documentation makes the behaviour clear. The function raises no exception. If a user knows about the function they should know how to handle negative values.

Marty's interface also has no method for putting a wider type primitive into an unsigned, although it could be added (e.g. putWider()).

Just to address some of Marty's comments:

>"What is proposed [by DH] has two main features:

>a) It does not state it below but it has been said before that in Java the implementation of the standard get and/or put will throw an exception if a data value is negative, i. e. it traps integer overflow."

I've addressed this above. 'It does not state it below' because it was never my intention.

>"In C++ templates are used to reduce the code size.
>What makes this possible is that the method signatures are the same, except for types, between the different data types."

This is unaffected by what I've been discussing. The interfaces I have suggested are for Java which needs to use signed types giving rise to the wider/narrower issue. The C++ ones won't require the new raw functions because C++ has unsigned integer types so will use these, in the same way that, I presume, you weren't intending to add getWider() to the C++. Any extra functions to improve the interface would be added in both cases, but if consistently named could be implemented in terms of templates in the C++ case. Generic programming/templates can be used with either interface. The Java interfaces are consistent with each other. The unsigned have some extra functions, just as your interface did, and these are added in a consistent way.

>"get and de-serialize should have the same semantics.
>A Java client that does a get from a local PVUxxx.get might get an exception but if the value is obtained via pvAccess no exception occurs. Different semantics make no sense.
>A similar argument can be made for put and serialize."

Not throwing exceptions for PVUxxx.get.


> "What is proposed has two main features:"

You missed the main feature which is the user by default gets the wider type which is won't be negative and the user doesn't have to deal with exceptions.

>"The additional methods are only providing what the conversion facility already does."

I did suggest that we add some methods to make the classes more usable. These are entirely optional to implement the unsigneds. In general these classes are not easy to use to do common tasks. These were just suggestions. Looking at even simple examples like the HelloServiceFactory in  helloWorld it seems we will want to use these PVs to write services and the current interface would make it difficult to manipulate array objects. I said I'd think more about element access and on refection what I think we probably need is element access. Currently getting a element is hard. You either get a handle to the raw data, which is lousy software engineering, or you use the conversion facility to get an array and get the element. If you know you have an array of a particular type using the introspection API to find out what it is is frankly a bit daft. One suggestion for the accessor methods would be  like a Java ArrayList, e.g.

public interface PVIntArray extends PVScalarArray
{
     int put(int offset,int length, int[] from, int fromOffset);

     int get(int index);
     void put(int index, int value);

     // (Advanced) Get handle to raw data for efficient use
     int getArrayData(int offset, int length, IntArrayData data);

     void shareData(int[] from);
}

I don't think this one is a particularly excessive interface.

For the unsigneds in Java I would add the raw equivalents.

public interface PVUIntArray extends PVScalarArray
{
     int put(int offset,int length, long[] from, int fromOffset);
     int putRaw(int offset,int length, long[] from, int fromOffset);

     int get(int index);
     void put(int index, int value);

     long getRaw(int index);
     void putRaw(int index, long value);

     // (Advanced) Get handle to raw data for efficient use
     int getArrayData(int offset, int length, UIntArrayData data);

     void shareData(int[] from);
}


> "Adding all these methods:

>a) Makes life harder for applications the need to implement special versions of the data interfaces."

Adding additional functions adds a little bit of work for specializing the interface, but not that much since they could be implemented in terms of each other and it will likely save the user effort when the functions are called because the user doesn't have to all the manipulations of the data or conversions each time. The functions were just suggestions and were optional for implementing unsigned. I think the interface I've suggested above isn't particularly onerous.

> "b) Different types will likely require different sets of methods."

I'm suggesting a consistent set of interfaces for arrays. They'll differ by the types of the underlying variables (PVIntArrays will deal with ints etc.) that all.


> "c) Makes documentation more verbose."

Not much. And frankly I'd rather have a well-documented useful method than have  ever user of the class have to come up with same boiler-plate code over and over again.

> "d) Makes it harder or impossible to use templates in C++"

Adding member functions with consistent names and signatures can be achieved by templating. Adding functions doesn't decrease the possiblilty of template use by a user (in fact it increases it if the new functions are consistently named).

>"But if additional useful methods are added don't they also make sense for C++? "

The extra methods for unsigned are Java only. Other methods are C++ and Java. But if the interface is deficient in the Java case and the C++ is the same why wouldn't we fix both.

>" The fact that only a minimal set of methods need to be implemented simplifies development of alternative implementations."
However a 'minimal set of methods' adds complexity for the user who has to implement the same code over and over again.



Dave Hickin
Software Systems Engineer, Controls Group
Extn. 8032

From: Marty Kraimer [mailto:mrk...@co...]
Sent: 14 February 2012 12:19
To: epi...@li...
Subject: Re: proposal for implementing unsigned

Brief summary of scalar and scalar array data types.
What exists now are 8 Scalar types: pvBoolean,...,pvString.
Thus 8 scalar data types and 8 scalar array data types.
Thus 16 data types that need to be implemented.

The code that implements each of the 8 scalar data types implements: get, put, serialize, and de-serialize.
The get method signatures, except for the types, are identical. Same for put.
The code that implements each of the 8 scalar array data types implements: setCapacity, get, put, serialize, de-serialize, and shareData.
Again the methods for each type have the some signatures.
The methods are the minimum required to implement the desired functionality.

Currently both Java and C++ implement all the data types.

In C++ templates are used to reduce the code size.
What makes this possible is that the method signatures are the same, except for types, between the different data types.

A conversion library is provided that performs many (most?, all?) reasonable conversions to/from/between the data types.

The data types are meant to be extensible, i.e. alternative implementations are expected.
Examples are support for transient recorders, frame grabbers, CDD detectors, zero copy, etc.
The fact that only a minimal set of methods need to be implemented simplifies development of alternative implementations.

Now it has been agreed that unsigned integers will be supported.\
This adds 4 new scalar types and a total of 8 new data types.
This we now have a total of 24 data types to implement.
Must be implemented in Java and C++.
Python is planned soon.
In the future who knows. C#? Mathlab? ???

The conversion library can be extended to support the new types.
In java any method that widens an unsigned integer can do the correct conversion.
Thus the conversion library is not a problem.

So what about the methods for the unsigned integers?

Lets consider four possibilities:

1) Exact same set of methods the current data types support.
The problem is Java because the language does not support unsigned as a primitive type.
The implementation for unsigned could be exactly the  same implementation as the corresponding signed type.
If user code wants different semantics the conversion facility can be used.

The discussion at the last meeting and what David says below state that this is not sufficient so something more should be done.
So next possibility

2) Exact same set of methods as the current data type support but one additional method.
The new method will be called getWider and it returns the next wider data type.
The standard methods will have the same implementation as the corresponding signed data type.

But the possibility of additional methods leads to:
Why not additional useful methods?
So consider:

3) What David proposes below.

What is proposed has two main features:

a) It does not state it below but it has been said before that in Java the implementation of the standard get and/or put will throw an exception if a data value is negative, i. e. it traps integer overflow.
b) Several addition useful methods are added.

But if additional useful methods are added don't they also make sense for C++? Thus consider:

4) Allow all the data interfaces to define a set of useful "convenience" methods.

Now for some discussion.

serialize and de-serialize should NOT trap integer overflow. Consider a C++ server sending an unsigned integer to a C++ client via a Java gateway.

get and de-serialize should have the same semantics.
A Java client that does a get from a local PVUxxx.get might get an exception but if the value is obtained via pvAccess no exception occurs. Different semantics make no sense.
A similar argument can be made for put and serialize.

The additional methods are only providing what the conversion facility already does.
Adding all these methods:
a) Makes life harder for applications the need to implement special versions of the data interfaces.
b) Different types will likely require different sets of methods.
c) Makes documentation more verbose.
d) Makes it harder or impossible to use templates in C++.

2), 3), and 4) suffer from these in increasing difficultly.
Even 2) adds complexity that does not accomplish much.
3) adds even more complexity and as stated trapping integer overflows is not a good idea.
I hope no one  seriously considers 4).

I think that 1) above is the best, i. e. let client code use the conversion facility if safe conversions are necessary.

Marty





On 02/13/2012 03:48 PM, dav...@di...<mailto:dav...@di...> wrote:



Ok, here's my proposal for implementing unsigneds. I've attached a longer set of notes with lots of details explaning why I suggest what I suggest and clarifying a few points which you can read if you're interested, but in brief:



There's absolutely no problem in implementing the unsigned interface so that we can make the wider type be used for the get operation and still keep the interfaces consistent in the case of arrays. In fact I think we can enhance the existing interfaces, making them clearer and easier to use. We don't have to re-write any existing code (unless we want to - I've suggested a possible name change for one function but it's entirely optional). We can also get rid of some code duplication and manipulation of raw data.



The only thing we can't do is provide direct access to a buffer of the wider type internal to the PV. But Marty's suggestion doesn't allow this either since we have a single buffer of the raw signed type. However I don't think we would ever want to do this. I don't think this make the interfaces inconsistent though - I've given what they should be below and I think they look pretty consistent.





It's not substantially different from Marty's version. In its simplest form it just requires switching the names of the get and getWider functions (which would have the same implementation) in the scalar case and adding one or two functions in each unsigned type so that the narrow and wider types always have analogues of each other, except for accessing the underlying array directly. We can add some convenience functions to enhance the interface if wanted.



If you're happy with the suggestion I could do any extra work needed to go from Marty's version to mine, e.g. adding the extra functions (I could even do the name swapping as well if Marty wanted) .



The interfaces are:



public interface UInt extends PVScalar

{

    public long get();

    public int getRaw();



    public void put(long value);

    public void putRaw(int value); // Alternative name: put

}



for the scalars, whilst for the arrays a minimalist version is:



public interface PVUIntArray extends PVScalarArray

{

     int put(int offset,int length, byte[] from, int fromOffset);

     int putRaw(int offset,int length, byte[] from, int fromOffset); // Alternative name: put



     // copy to (existing) destination buffer (if there's room). Returns number of bytes copied.

     int copyToBuffer(int offset, int length, long[] dest, int toOffset); // Alternative names: get/copy

     int copyToBuffer(int offset, int length, int[] dest, int toOffset);  // Alternative names :get/getRaw/copy.





     // (Advanced) Get handle to raw data for efficient use

     int getArrayData(int offset, int length, UIntArrayData data); // Alternative name: get

     // Note This rename is entirely optional, but could be done if it make interfaces easier to understand



     void shareData(int[] from);

}





(The name choices are my preferred ones, but I've put the alternatives I discuss in my notes. I'd be interested in which ones people prefer especially for the copy functions where is was torn better copyToBuffer and get/getRaw). The signed interfaces are the same except they lack the wide/narrow type choices. UIntArrayData is a new type analagous to the IntArrayData which has to have a signed data member in the Java case. The reasons for doing it this way are explained in the notes.





A fuller interface with some easy to use get functions which give a simple put/get interface as per the scalar case and some convenience functions with sensible defaults to make common tasks easier:



public interface PVUIntArray extends PVScalarArray

{

     // return new copy of data

     long[] get()                              // Optional

     int[]  getRaw()                           // Optional

     long[] get(int offset, int length)        // Optional

     int[]  getRaw(int offset,int length)      // Optional



     int put(long[] from);                     // Optional

     int putRaw(int[] from);

     int put(int offset,int length, byte[] from, int fromOffset);    // Alternative name: put

     int putRaw(int offset,int length, byte[] from, int fromOffset); // Alternative name: put



     // copy to (existing) destination buffer (if there's room). Returns number of bytes copied.

     int copyToBuffer(int offset, int length, long[] dest, int toOffset); // Alternative names: get/copy

     int copyToBuffer(int offset, int length, int[] dest, int toOffset);  // Alternative names :get/getRaw/copy.

     int copyToBuffer(long[] dest); // Optional // Alternative names: get/copy

     int copyToBuffer(int[] dest);  // Optional// Alternative names :get/getRaw/copy.



     // (Advanced) Get handle to raw data for efficient use

     int getArrayData(int offset, int length, UIntArrayData data); // Alternative name: get - rename entirely optional





     void shareData(int[] from);

}





We could also consider adding some element access functions to make the arrays easier to use and avoid direct access of the data, but this is not necessary to implement the signed/unsigned PVs, so I'll leave these for now. However these could provide get and put methods with corresponding getRaw and putRaw in the unsigned case.



Let me know what you think.



Regards,



Dave.





Dr David  Hickin

Software Systems Engineer

Diamond Light Source Ltd

Diamond House

Harwell Science&  Innovation Campus

Didcot, Oxfordshire, OX11 0DE












------------------------------------------------------------------------------

Try before you buy = See our experts in action!

The most comprehensive online learning library for Microsoft developers

is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,

Metro Style Apps, more. Free future releases when you subscribe now!

http://p.sf.net/sfu/learndevnow-dev2




-- 

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.

Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 

Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.

Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom