[Plplot-devel] precision problems in c_plmtex()

[Maurice L. <mj...@ga...>]

In writing the new store-strings-as-strings-in-plmeta capability, I came
across some code in plmtex() that has been giving me fits.  Not only does it
have precision problems, but it's inconsistently written, so that under some
circumstances a quantity is rounded, and under others, it's truncated.

The big picture: to really get the string store capability (in plmeta)
correct, I need to write it out at the normalized device coordinate level,
i.e. (floating point) bounds from 0 -> 1.  Only at the render level is
it converted to physical (mm) and then device (pixel) coordinates.  So
my task was to generalize the monster if-else in plmtex() as well as the
code in plptex, plxytx, plztx and insert a new layer just above plstr()
that does all its stuff in normalized device coords.

Aside: this is a good prototype for things to come, since we eventually
want to do more things at the normalized device coordinate level.

So it looks like fixing plmtex() is going to involve some off-by-one pixel
changes to the output, by what I'm seeing in the ps output.  Just a heads-up
b/c I know others (especially Alan) will be affected by this change.  I'll try
to time it with another change I have planned that may affect low order bit
precision as well.

A look at the if-else in plmtex() reveals:

    if (plP_stsearch(side, 'b')) { 
        vert = 0; 
        xdv = plsc->vpdxmi + (plsc->vpdxma - plsc->vpdxmi) * pos; 
        ymm = plP_dcmmy(plsc->vpdymi) - disp * chrht; 
        x = plP_dcpcx(xdv); 
            ^^^^^^^^^^^^^^^  rounded to PLINT
        refx = x - shift * plsc->xpmm; 
                   ^^^^^^^^^^^^^^^^^^^  precision problem! truncated, not rounded
        y = refy = plP_mmpcy(ymm); 
    } 
    else if (plP_stsearch(side, 't')) { 
        vert = 0; 
        xdv = plsc->vpdxmi + (plsc->vpdxma - plsc->vpdxmi) * pos; 
        ymm = plP_dcmmy(plsc->vpdyma) + disp * chrht; 
        x = plP_dcpcx(xdv); 
            ^^^^^^^^^^^^^^^  rounded to PLINT
        refx = x - shift * plsc->xpmm; 
                   ^^^^^^^^^^^^^^^^^^^  precision problem! truncated & added
        y = refy = plP_mmpcy(ymm); 
    } 
    else if (plP_stindex(side, "LV") != -1 || plP_stindex(side, "lv") != -1) { 
        vert = 0; 
        xmm = plP_dcmmx(plsc->vpdxmi) - disp * chrht; 
        ydv = plsc->vpdymi + (plsc->vpdyma - plsc->vpdymi) * pos; 
        x = plP_mmpcx(xmm); 
            ^^^^^^^^^^^^^^^  rounded to PLINT
        refx = x - plP_mmpcx(shift); 
                   ^^^^^^^^^^^^^^^^  rounded!  Both inconsistent with the
above code AND a precision problem!  Because you should add first, then round,
rather than the other way around.

        y = refy = plP_dcpcy(ydv); 
    } 
    else if (plP_stindex(side, "RV") != -1 || plP_stindex(side, "rv") != -1) { 
        vert = 0; 
        xmm = plP_dcmmx(plsc->vpdxma) + disp * chrht; 
        ydv = plsc->vpdymi + (plsc->vpdyma - plsc->vpdymi) * pos; 
        x = plP_mmpcx(xmm); 
            ^^^^^^^^^^^^^^   ditto
        refx = x - plP_mmpcx(shift); 
                   ^^^^^^^^^^^^^^^^^ ditto
        y = refy = plP_dcpcy(ydv); 
    } 
    else if (plP_stsearch(side, 'l')) { 
        vert = 1; 
        xmm = plP_dcmmx(plsc->vpdxmi) - disp * chrht; 
        ydv = plsc->vpdymi + (plsc->vpdyma - plsc->vpdymi) * pos; 
        x = refx = plP_mmpcx(xmm); 
        y = plP_dcpcy(ydv); 
            ^^^^^^^^^^^^^^ rounded
        refy = y - shift * plsc->ypmm; 
                   ^^^^^^^^^^^^^^^^^^^  truncated & added
    } 
    else if (plP_stsearch(side, 'r')) { 
        vert = 1; 
        xmm = plP_dcmmx(plsc->vpdxma) + disp * chrht; 
        ydv = plsc->vpdymi + (plsc->vpdyma - plsc->vpdymi) * pos; 
        x = refx = plP_mmpcx(xmm); 
        y = plP_dcpcy(ydv); 
            ^^^^^^^^^^^^^^ rounded
        refy = y - shift * plsc->ypmm; 
                   ^^^^^^^^^^^^^^^^^^^  truncated & added
    } 

The correct way should be to add the offset (in the proper coordinate system),
then round. 

-- 
Maurice LeBrun    mj...@ga...
Research Organization for Information Science and Technology of Japan (RIST)