Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#29 VT_DECIMAL support

1.13 Accepted
closed-accepted
nobody
5
2011-10-02
2007-08-12
Matt Lilley
No

This is a simple patch to translate BigDecimal Java objects into VT_DECIMAL types. I haven't implemented the reverse direction as I didn't require it. It should be fairly obvious how to implement it looking at this patch if someone requires it.

Matt

Discussion

  • Matt Lilley
    Matt Lilley
    2007-08-12

    Patch (against cvs as of right now) to add VT_DECIMAL support

     
    Attachments
  • clay_shooter
    clay_shooter
    2007-08-27

    Logged In: YES
    user_id=1189284
    Originator: NO

    Hi,

    I can put this patch in but have a couple questions. Can you explain what is really going on and describe the behavior. It looks like this thing sets it as a double as long as the scale is <= 28. What are the other steps doing? (I'd like to add it to the javadoc) Also, can you describe what all 5 parameters are used for in the native method?

    Thanks

    /**
    * set the value of this variant and set the type
    * @param in
    */
    public void putDec(BigDecimal in){
    // verify we aren't released yet
    getvt();
    int scale = in.scale();
    if (scale > 28) {
    putDouble(in.doubleValue());
    }
    else {
    BigInteger unscaled = in.unscaledValue();
    BigInteger shifted = unscaled.shiftRight(32);
    putVariantDec(in.signum(), in.scale(), unscaled.intValue(), shifted.intValue(), shifted.shiftRight(32).intValue());
    }
    }

     
  • clay_shooter
    clay_shooter
    2007-08-27

    Logged In: YES
    user_id=1189284
    Originator: NO

    Hi,

    We really need to have the "getDec()" method to put this in the next release. Without it, toString() and toJavaObject() won't work.

     
  • Matt Lilley
    Matt Lilley
    2007-08-31

    Patch to implement VT_DECIMAL support in both directions

     
    Attachments
  • Matt Lilley
    Matt Lilley
    2007-08-31

    Logged In: YES
    user_id=1084227
    Originator: YES

    File Added: bigdecimal.patch

     
  • Matt Lilley
    Matt Lilley
    2007-08-31

    Logged In: YES
    user_id=1084227
    Originator: YES

    I thought I'd posted a reply to this already, but it seems to have gone missing.
    VT_DECIMAL has space for 96 bits of an unscaled number plus a number up to 28 to signify the scale (see http://msdn2.microsoft.com/en-us/library/ms962433.aspx\). The original patch does this in Java to avoid having to manipulate java objects in C - as you'll see from the new patch this is quite an unpleasant thing to have to do! If the scale is greater than 28, I defer the type to double - we can't do any more precision than 28 and so our best effort is the imprecise but heavily scalable double. Other than that, the remainder of the code cuts the 96 bits into 3x32 bit chunks and passes them to C for reassembly in the tagDECIMAL struct.

    As for the reverse direction, it's quite ugly. The only constructors available are the one which takes a byte[] and one which takes a String (that seems a bit inefficient). So, we construct a byte[] by mapping the various bytes of the tagDECIMAL into a temporary array, tell the JVM to load it into a local reference, and call the constructor. Then we pass this result in to the BigDecimal constructor along with the scale to get the appropriate reference to return. To save on space I've created a function which both getDec and getDecRef call. I tried to follow standards where I could, but there didn't seem to be any ordering on the functions in C or Java, so I've just thrown them in. Someone more at one with the project can easily shift them to the right places if necessary :-)

    Any thoughts/problems/criticisms, please let me know, I'm happy to discuss/help/attempt to defend myself!

     
  • clay_shooter
    clay_shooter
    2007-08-31

    Logged In: YES
    user_id=1189284
    Originator: NO

    I kind of liked the way the previous version did as much work as possible in Java. It made it easier for my Java centric brain :-)

    We should just throw something like an illegalArgumentException if the scale is greater than 28. Converting to double can be an unexpected side effect.

    I'll try and look at the patch in the next week or so.

     
  • Matt Lilley
    Matt Lilley
    2007-08-31

    Logged In: YES
    user_id=1084227
    Originator: YES

    Heh. Well the code for Java -> C is the same as before (as much as possible in Java) but we can't really do it in reverse - either we'd have to pass back the 3 ints that make up the unscaled value by reference in the arguments to getDec or else return some kind of class that contains the information (3x int, sign, scale), and if we're going to all that effort, we may as well just return the BigDecimal instead.

    I'm not sure what the right thing to do is with the large scale - as you said, it could be an unexpected side effect, but it means that there's no trivial way to send any BigDecimal to COM - supposing you have some code which allows a user to enter a number (as a string, say) and then converts it to BigDecimal and calls new Variant(myDecimal). We'd have to have code like
    Variant v;
    try {
    v = new Variant(myDecimal);
    } catch(IllegalArgumentException iae) {
    v = new Variant(myDecimal.doubleVal());
    }
    which is a bit ugly. In any case, COM code is supposed to coerce the values as necessary - if you pass in a VT_BSTR and the other side of the interface is expecting a VT_I4, then someone is supposed to attempt the coercion if possible. My point is that we should really just aim for best-effort, since if the other side was expecting a VT_R8 and we passed in a small BigDecimal (like '4' say) then the type of the argument we send will be VT_DECIMAL and the receiver can coerce that into VT_R8 if they want to. If we have a really huge decimal ('1234.......0') which has more than 28 digits, then the receiver will either coerce that to a VT_R8 as a matter of course if it can handle large numbers (since it can't expect to ever get such a large VT_DECIMAL, owing to it being impossible!), or else a VT_BSTR if it can sort-of handle them, or else the coercion will fail and the callee will raise some kind of type error. This makes the life of the caller easier (since we can ALWAYS call new Variant(myDecimal) without having to check the value) and I don't think it makes the life of the callee any more difficult.

    I developed another COM interface (for SWI-Prolog) and my general attitude of 'pass in what the caller says and let the callee sort it all out' seems to work really well. As an example of this kind of coercion, try putting a really large decimal into a cell in Excel - when you get it back you'll find it's become a VT_R8.

    As always, happy to continue the discussion if you feel especially strongly about something! :-)

     
  • clay_shooter
    clay_shooter
    2007-09-01

    Logged In: YES
    user_id=1189284
    Originator: NO

    Yes, I had thought about having getDecimal() accept an array of integers that it fills or having it return an array of integers. Then the java wrapper would do all the bit manipulation. Basically, we'd just copy the values from the Variant structure into the result array. It's different than all the other getXXX() methods but would reduce the amount of c manipulation that would have to be done.

    Are we trying to send bigdecimal to COM or are we trying to fill a Variant with type VT_DECIMAL? The goal sort of drives the implementation.

    The other problem with conversion from BigDecimal to Double is that you have a reasonable chance of losing precision. The number that ends up in the COM layer isn't guaranteed to be the same number that you passed into the Java layer. We see this all the time in numerical calculations where folks incorrectly use Double in money calculations.

     
  • Matt Lilley
    Matt Lilley
    2007-09-02

    Logged In: YES
    user_id=1084227
    Originator: YES

    Yes, if you prefer to send back an array of integers you may, but it seems confusing to me to send back an array of 5 integers, 3 of which are the unscaled value, and the other two are the sign (promoted from a BYTE) and the scale (also a BYTE). This seems somewhat arbitrary and confusing, whereas the code in C, although complicated, is at least clear. However, the choice is not mine to make :-)

    As for the coercion, yes, this sort of thing will happen. My point is that it will happen anyway, regardless of how careful we are, because the client application won't think twice before coercing the data type, potentially losing precision. The interface might even change between when the Java code was written and the client type library was supplied. In any case, I don't think many COM applications (I'm pretty sure VB for one) can even handle decimals very well - if you pass in 90000000000000000000000000000 and try and add 5 to it, I think the runtime will 'helpfully' convert it to a double for you (though I could be wrong and have no trivial way of checking). If you don't like demoting to VT_R8, how about converting it to a VT_BSTR? It's perfectly legal to pass a number in a string to a function expecting a number. That way the client application can generate the overflow (which, I still insist is the correct behaviour, rather than letting the caller judge) if it can't handle the large number (which it almost certainly won't be able to).

    Ultimately, I'm interested in what you think a programmer would do with such a large BigDecimal given that they receive an IllegalArgumentException? They have two real choices: Either demote it to a type which COM can indeed handle, or give up and tell the user that such a large number is not supported by jacob.

     
  • clay_shooter
    clay_shooter
    2007-09-03

    Logged In: YES
    user_id=1189284
    Originator: NO

    One of the questions that comes up a lot is "who is the consumer of Jacob". Are they a Java programmer who needs to talk to a windows application? Or, are they an MS Windows programmer that is using Java? The Java programmers know only what the API docs say and don't have any idea what happens if you pass in data of the wrong type. Your notes are the first time I've heard of the on-the-fly data conversion. It would seem to be hard for a non-C programmer to understand what is happening if the code has problem data is converted to other types before even getting to the COM layer. Does the conversion always work?

    The original code was written by COM programmer with all of the JNI code exposed as public methods. There were all kinds of data assumptions because there was no way to validate input. I have a Java bias so I made the JNI API and provided public methods that check the input data. This was to reduce null pointer and other exceptions raised in the COM layer because of insufficient type checking and conversion in the COM layer. I think the Java API should do what's implied by the method names and it should document the range of data that it supports. I don't have a problem with the JNI layer being slightly different from the public API. Your setDecimal() does this. The key is documentation for whoever becomes the release coordinator after I escape from this :-)

    My lack of depth in Windows API is definitely showing.

     
  • clay_shooter
    clay_shooter
    2007-09-06

    Logged In: YES
    user_id=1189284
    Originator: NO

    The second patch has been checked into CVS. The only change is that it throws IllegalArgumentException on scale > 28 which may limit it's use. VariantTest.java has been upgraded to verify the basic functionality. I think that the double conversion would cause the put/get equality tests if if the item was converted to double with scale > 28 and then retrieved back as a big decimal. This fix will make the 1.13M4 release

     
  • clay_shooter
    clay_shooter
    2007-09-06

    • milestone: --> 1.13 Accepted
    • status: open --> pending
     
  • Matt Lilley
    Matt Lilley
    2007-09-06

    Logged In: YES
    user_id=1084227
    Originator: YES

    Right. By the way, you should note that I am by no means a COM expert - I knew nothing at all about it just a few months ago. As I mentioned in a previous comment all my understanding has come through my work on a project which is, I suppose, equivalent to Jacob except for a language other than Java. My experimentation has been limited to Excel and a control I wrote myself in VB6, so it should be noted that I don't have any experience with writing the other side of the interface (ie implementing IDispatch and friends), and my assertions about coercion might be from the implementations of IDispatch I've been exposed to.

    Case in point: I discovered to my horror the other day that while VB will happily coerce the string "123" to a float 123.0 if you have some sub with a float byval. But if you have a byref argument, and pass in an empty VT_VARIANT | VT_BYREF then it won't coerce anything to that for you. This means, for example, that VBScript can't talk to any COM object that has byref arguments that aren't of type Variant since it passes all arguments as variant (and VB will happily coerce any variant byval to any other type byval).... just as I was starting to see a lot of potential in the type system I find this out... Seems Microsoft has a knack for snatching defeat from the jaws of victory!

     
  • Matt Lilley
    Matt Lilley
    2007-09-06

    • status: pending --> open
     
  • clay_shooter
    clay_shooter
    2007-09-21

    • status: open --> closed
     
  • clay_shooter
    clay_shooter
    2011-10-02

    • status: closed --> closed-accepted