Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

Precision/Scale of float

Help
Laurent
2012-11-08
2012-11-26
  • Laurent
    Laurent
    2012-11-08

    Hi,

    I have a float field defined (type is 'real' in the PostgreSQL database). As I need a scale better than the standard 2 digits on the float, I have setup in default-size.xml a stereotype FLOAT_5.

    <for-stereotype name="FLOAT_5" size="6" scale="5"/>
    

    .
    And annotated the field with this stereotype.

    @Stereotype("FLOAT_5")
        private float factor;
    

    When I modify the field in the view, no problem: I can type all the scale I want: up to 5 digits. But when I save the view, the precision is reduced to 3 digits: the float is rounded.
    If I change the value in the database to have all 5 digits, I still get only 3 digits displayed when I open the view.

    I can't quite figure where to correct this behaviour… Can anybody help?

     
  • Javier Paniza
    Javier Paniza
    2012-11-12

    Hi Laurent,

    if you add a @PreCreate @PreUpdate method in your entity and print factor, which precision does it have?
    If it has the correct precision add a @PostLoad method and print the factor property, which precision does it have?


    Help others in this forum as I help you.
    Before posting a question search this forum for the answer,
    and read the OpenXava documentation in openxava.org.

     
  • Laurent
    Laurent
    2012-11-13

    Hi Javier,

    if you add a @PreCreate @PreUpdate method in your entity and print factor, which precision does it have?

    I save the entity and enter in @PreUpdate (called by @PreCreate) : result is correct.

    EstimationsElement.PreUpdate: factor=3.4E-4
    

    If it has the correct precision add a @PostLoad method and print the factor property, which precision does it have?

    I reload the entity and enter @PostLoad: the value is correct.

    EstimationsElement.PostLoad: factor=3.4E-4
    

    Nonetheless, the displayed value is 0.

    I now save the opened entity, without entering any value in the factor field:

    EstimationsElement.PreUpdate: factor=0.0
    

    So something happens between the moment the entity is loaded and the moment it is displayed…

    Thanks for your help!

    Laurent

     
  • Hi Laurent,

    Has you added the precision and scale attributes to @Column corresponding to the field in your entity? I rather use BigDecimal as it handles scaling as it should. Something like this.

    @Stereotype("FLOAT_5")
    @Column(name = "float", precision = 6, scale = 5)
    private BigDecimal factor;
    

    Hope this helps,

    Federico

     
  • Hola a todos

    Tengo un problema similar.

    La definicion del campo es la siguiente:

    @Column(precision=5,scale=8) @NotNull
    private double valor;

    Al agregar un nuevo registro, la vista y la actualización a la BD funcionan bien (almacena los 8 decimales), sin embargo en la lista solo muestra 3 decimales y al editar el mismo registro también muestra solo 3 decimales. Si cambio el valor a 8 decimales y lo grabo, también lo almacena bien.

    El problema solo es en la lista y en la edición.

    Intenté usar BigDecimal pero en la lista y edición no ajusta a la precisión definida, muestra un numero mayor de decimales.

    Agradezco su ayuda…

     
  • Laurent
    Laurent
    2012-11-15

    Hi Federico,

    Thanks for your input.

    I rather use BigDecimal as it handles scaling as it should.

    I did not know that, so all my code uses 'floats' and the database uses 'real'.
    The database storage class of BigDecimal is integer or real ?

    Anyway, I tried your suggestion with the @Column and apart from the field taking more room on the view, there is no change in the behaviour.

    Laurent

     
  • Javier Paniza
    Javier Paniza
    2012-11-15

    Hi Laurent,

    edit the method format() of MetaProperty class, chaning this code:

                if (Float.class.isAssignableFrom(type) || float.class.isAssignableFrom(type)) {
                    return NumberFormat.getNumberInstance(locale).format(value);
                }
                if (Double.class.isAssignableFrom(type) || double.class.isAssignableFrom(type)) {
                    return NumberFormat.getNumberInstance(locale).format(value);
                }
    

    by

                if (Float.class.isAssignableFrom(type) || float.class.isAssignableFrom(type)) {
                    NumberFormat numberFormat = NumberFormat.getNumberInstance(locale);
                    numberFormat.setMaximumFractionDigits(Integer.MAX_VALUE);               
                    return numberFormat.format(value);
                }
                if (Double.class.isAssignableFrom(type) || double.class.isAssignableFrom(type)) {
                    NumberFormat numberFormat = NumberFormat.getNumberInstance(locale);
                    numberFormat.setMaximumFractionDigits(Integer.MAX_VALUE);               
                    return numberFormat.format(value);
                }
    

    Tell me if it works for you.


    Help others in this forum as I help you.
    Before posting a question search this forum for the answer,
    and read the OpenXava documentation in openxava.org.

     
  • Muchas gracias Javier, funcionó bien para mí.

    Cabe mencionar que estoy usando tipo de dato float en la base de datos.

    Saludos…

     
  • Laurent
    Laurent
    2012-11-16

    Hi Javier,

    For me also it works well. A little too well in terms of precision…
    I have limited the number of digits to 4 instead of Integer.MAX_VALUE for readbility for the moment.
    I will dig inside the code to see how this formatting could take into account the @Stereotype which would obviously be better than having a fixed number of digits in the whole application.

    Thanks for this!

    Laurent

     
  • Javier Paniza
    Javier Paniza
    2012-11-19

    Hi Laurent,

    For me also it works well. A little too well in terms of precision…

    Fantastic! Could you add it as a bug in order I would add the fix for future OX release?

    I have limited the number of digits to 4 instead of Integer.MAX_VALUE

    Maybe it would be better to get this value from the scale.

    see how this formatting could take into account the @Stereotype

    If you stereotype is attached to a editor with formatter, the formatter is used for formatting instead of the code of MetaProperty, so you already can do it.

    -------
    Help others in this forum as I help you.
    Before posting a question search this forum for the answer,
    and read the OpenXava documentation in openxava.org.