Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#336 Refresh plot or zoom without re-reading data

closed-out-of-date
Ethan Merritt
None
2
2009-06-01
2007-05-22
Ethan Merritt
No

This patch adds a new command 'refresh' that acts similarly to 'replot' except that it does not re-read or recalculate the input data. This allows you to zoom, add labels, or otherwise modify the current plot even if the input data is transient. In particular it allows you to zoom or otherwise modify a plot whose data came from stdin, e.g. plot '-'.

Data processing is modified in these ways:

1) Plot data is not discarded immediately after a successful plot command. Instead it is kept around until it is replaced by a subsequent plot or replot command. This is a change for 2D plots. 3D plots already worked this way.

2) The new command 'refresh' is implicitly used by mouse zooming operations. Because the axis ranges may have changed since the data was read in, we must walk through the data and re-evaluate the INBOUNDS/OUTBOUNDS flag for each point before regenerating the plot.

3) I had to re-work the way zoom operations are saved/restored. It is actually a simplification of the existing code, but the behaviour of "unzoom" is now subtly changed if it is triggered from a different plot than the original zoom operation. The old behaviour was undocumented, so I am not too worried about this at the moment.

Known issues in the first version (22 May 2007):

- Quick refresh is not yet implemented for 3D plots (splot + set view map).

- Only the primary x and y coordinates of each point are rechecked against the current axis limits. Possibly this is not sufficient for some plot styles (errorbars? candlestick?) Easy to fix if needed.

- There should be some way to turn off the quick refresh if you really do want to re-read the data during a zoom operation. Maybe a new command
set zoom {replot|refresh}
or
set mouse zoom {replot|refresh}

Discussion

1 2 > >> (Page 1 of 2)
  • Petr Mikulik
    Petr Mikulik
    2007-05-22

    Logged In: YES
    user_id=31505
    Originator: NO

    For me it does not work when zooming with '-'.
    It shows its typical
    input data ('e' ends) >

    Can you please fix it?

     
  • Ethan Merritt
    Ethan Merritt
    2007-05-22

    Logged In: YES
    user_id=235620
    Originator: YES

    >Can you please fix it?

    I don't know. For me it works fine with both x11 and wxt. The changes do not touch any driver-specific code that I know of, but I have not tested other interactive terminals to confirm this. You might try putting a debug statement in replotrequest() to see if the refresh information is making it through to that point. It should report refresh_ok = 2 when you try to zoom a 2D plot.

    diff -ur gnuplot/src/command.c gnuplot-cvs/src/command.c
    --- gnuplot/src/command.c 2007-05-14 01:05:12.000000000 -0700
    +++ gnuplot-cvs/src/command.c 2007-05-22 15:41:42.000000000 -0700
    @@ -1777,6 +1813,8 @@
    void
    replotrequest()
    {
    + fprintf(stderr,"refresh_ok = %d\n",refresh_ok);
    +
    if (equals(c_token, "["))
    int_error(c_token, "cannot set range with replot");

     
  • Petr Mikulik
    Petr Mikulik
    2007-05-23

    Logged In: YES
    user_id=31505
    Originator: NO

    Fine, it works with zooming (by mouse or via zoom history hotkeys).

    The problem I had with plot '-' was with 'g', 'l' etc. hotkeys. In the current gnuplot, the variable replot_disabled is set for inline data, and this is checked in do_string_replot. Your code #if 0 replot_disabled= ... #endif. in mouse.c. This should be changed by calling the refresh instead of replot for the inline data.

    I think the
    set mouse {replot|refresh}
    command should apply to both the zoom and the default bind commands.

    For scripting purposes, it seems that GPVAL_CAN_REPLOT automatic variable would be useful. It will be filled by !replot_disabled.

    BTW, the trick with memcpy(axes) is great!

     
  • Ethan Merritt
    Ethan Merritt
    2007-05-23

    Logged In: YES
    user_id=235620
    Originator: YES

    > The problem I had with plot '-' was with 'g', 'l' etc. hotkeys.

    'l' is a problem. I forgot to point out that the quick refresh path cannot work if you change something about the processing of the input data. This includes toggling the log scaling, because this is applied at the time the data is read in rather than when it is plotted. That is a bit unfortunate, but I see no way around it without totally re-working the way log scaling is done.

    Yes, the 'g' hotkey should be fixed. I am uncertain about the 'a' hotkey; should it continue to issue a 'replot' command, or should it try to use 'refresh' if possible?

    You may be right that the replot_disabled flag is still needed in some other cases. Do you have an example command sequence that shows this?

     
  • Petr Mikulik
    Petr Mikulik
    2007-05-25

    Logged In: YES
    user_id=31505
    Originator: NO

    > Yes, the 'g' hotkey should be fixed. I am uncertain about the 'a' hotkey;

    They should work according to
    set mouse {replot|refresh}

    In-line data should always be refreshed only.

    > 'l' is a problem.

    Then the data should be de-logged. Can they have a flag whether they are logged or not, and (de)logged if this flag for a column is not consistent with its "set log"?

    > You may be right that the replot_disabled flag is still needed in some
    > other cases. Do you have an example command sequence that shows this?

    It was the "g" hotkey or "e", for example.

     
  • Ethan Merritt
    Ethan Merritt
    2007-06-07

    Logged In: YES
    user_id=235620
    Originator: YES

    Second version.

    Log scale plots now work correctly. However it remains an intrinsic limitation that if you toggle log scale on or off, than invalidates the stored data so that a full reread+replot is necessary.

    Known issues in the second version (6 June 2007):

    - Not yet implemented for 3D plots (splot + set view map)

    - During zoom, the plot boundaries do not adhere strictly to the
    new plot limits. But the actual data points do. So there is a
    gap between the graph lines and the plot border.

    - To disable the quick refresh mode, type
    REFRESH_DEBUG = 0
    File Added: persistent_plot_data_06jun2007.patch

     
  • Petr Mikulik
    Petr Mikulik
    2007-06-08

    Logged In: YES
    user_id=31505
    Originator: NO

    > Log scale plots now work correctly.

    I would prefer to ignore "set log", instead of "freezing" the plot.
    A message could be printed on the status line, for example.

     
  • Ethan Merritt
    Ethan Merritt
    2007-06-08

    Logged In: YES
    user_id=235620
    Originator: YES

    How would you know to ignore the "set log" command?
    The refresh command might not come until much later. The freeze is only triggered by the combination of "set/unset log" + "input from stdin" + "mouse action triggers replot", and is exactly what happens in the current (no quick refresh) version of gnuplot for *all* mouse-triggered replots.

     
  • Petr Mikulik
    Petr Mikulik
    2007-06-08

    Logged In: YES
    user_id=31505
    Originator: NO

    The 'l' hotkey will be disabled if there are 'stdin' data.
    Similarly, the 'g' hotkey will use 'refresh' instead of 'replot' if there are 'stdin' data.

    Is this feasible?

     
  • Ethan Merritt
    Ethan Merritt
    2007-06-08

    Logged In: YES
    user_id=235620
    Originator: YES

    It would be possible to disable certain hotkeys, yes, although right now the code is not tracking whether or not the input data came from '-'. Daniel suggested also adding a way to tag specific data files as being volatile.

    The current patch tries to use "refresh" everywhere possible, and falls back to "replot" only if it is not possible (e.g. change in log scale setting since the plot was first drawn). My thought was this would be the quickest way to find problems with the "refresh" code.

    For production use I think it would indeed be safer to keep a full "replot" as the default, and only use "refresh" in cases where the input data is volatile.

    As to the log scale problem itself, that would go away if the transformation to log scale were deferred to plot time rather than being applied at data input time. But this would mean a total re-write of the log scale code. Rather than doing that, I'd rather see someone implement a generic axis-transformation layer. As I see it, this would allow you to assign an arbitrary scaling function to each axis:
    set xaxis transform log(x)
    set xaxis transform 1/x
    set xaxis transform (x<100) ? x : x+500
    set xaxis transform -x
    If we ever got that working, the existing log scale code could be ripped out as being no longer needed. As an additional bonus it might be possible to also rip out the reverse-axis code.

     
  • Petr Mikulik
    Petr Mikulik
    2007-06-11

    Logged In: YES
    user_id=31505
    Originator: NO

    There is a global flag whether there are volatile data:

    datafile.h
    /* flag if any 'inline' data are in use, for the current plot */
    extern TBOOLEAN plotted_data_from_stdin;

    I think this can be used to flip between replot and refresh.

     
  • Dan Sebald
    Dan Sebald
    2007-06-26

    Logged In: YES
    user_id=704782
    Originator: NO

    Is this patch in the final testing stage? Has syntax been decided?

    If I understand correctly, there is an issue with having to reload data to switch back and fourth between log and linear axes. I've considered whether gnuplot could be redone to always store the data and not take the logarithm via STORE_WITH_LOG_AND_UPDATE_RANGE()? I've found this isn't as difficult as it sounds, and in fact it somewhat simplifies things (does away with a lot of "undo log"s) and addresses some FIXME's in the code. We already have a map_x() and map_y() in locations where points[i].x, etc. are used. The logarithm can be part of map_x() and map_y().

    /* Macros to map from user to terminal coordinates and back */
    #define AXIS_MAP(axis, variable) \ (int) ((axis_array[axis].term_lower) \ + ((variable) - axis_array[axis].min) \ * axis_array[axis].term_scale + 0.5)

    Since logarithm is monotonically increasing, log(min(X)) = min(log(X)) and log(max(X)) = max(log(X)) where X is the set of all x values of the data. We don't have to do the logarithm upon saving to work with the min and max values.

    I've put together a patch that I will send to each of you. I think you might like this change. Please consider integrating into the patch here. It may cause one or two unforseen little bugs, but after a release is as best as time as any to change such a thing.

    The OUT_RANGE, UNDEF_ACTION have been removed. They are used in one or two places only, otherwise they are mostly NOOP. In the one place UNDEF_ACTION is used there is no reason to toss out data, simply handle it when it comes time to plot. Recall, we were going to properly clean up "undefined", etc. behavior anyway.

    Dan

     
  • Ethan Merritt
    Ethan Merritt
    2007-06-26

    Logged In: YES
    user_id=235620
    Originator: YES

    >> Is this patch in the final testing stage?

    There are still many known limitations and glitches. But testing is of course welcome. See extensive notes at the head of the patch itself.

    The patch includes a debugging mechanism for testing, based on the user-defined variable REFRESH_MODE. Setting this to 0 essentially turns it off. Modes 1-3 exercise slightly different code paths. Mode 1 is the simplest, and the default. If you find a bug or glitch in mode 1, please try mode 2 or 3 to see if this works any better. So far I have not found any such cases, and probably these modes will be removed from the final version.

    Mode 4 corresponds to the intended final default; it only triggers refresh if the data source is volatile. But this makes it much harder to debug using standard scripts and data files.

    Dan:

    Please leave the existing log/unlog code in place. As noted in previous discussion, the way forward is to introduce a generic mechanism for describing non-linear axes. Once that general mechanism is in place, logscale axes are just a special case and the single-purpose code can be removed.
    File Added: persistent_plot_data_25jun2007.patch

     
  • Dan Sebald
    Dan Sebald
    2007-06-26

    Logged In: YES
    user_id=704782
    Originator: NO

    "As noted in previous discussion, the way forward is to introduce a generic mechanism for describing non-linear axes."

    Well, what I have may be a start. I'll put it in a new sourceforge patch. I think it is worth looking over. (Use gvim, it highlights patches really well with colors.) The STORE_WITH_LOG_AND_UPDATE_RANGE() macro has always been a bit of a problem... too much stuff in one step. The thing I don't like about it is it tosses data logarithmic and non-positive. Tossing data is never good in my mind.

    Getting rid of STORE_WITH_LOG_AND_UPDATE_RANGE() makes programming nicer in some ways. Also, I think avoiding the log/unlog is very welcome. The consequence is having to do log more than once (when checking range and when mapping the value...could be modified but not worth it now).

    If one wants to make what I have generic, just needs to be fixed in two places in axis.c.

     
  • Ethan Merritt
    Ethan Merritt
    2007-06-26

    Logged In: YES
    user_id=235620
    Originator: YES

    minor bugfixes
    File Added: persistent_plot_data_26jun2007.patch

     
  • Petr Mikulik
    Petr Mikulik
    2007-06-27

    Logged In: YES
    user_id=31505
    Originator: NO

    Ad non-linear axes: I would also prefer to have this patch independent.

    Ad the current persistent plot patch:

    (*) The 'u' for unzoom works, but not "set autoscale; refresh".
    Could you get this working, please?

    (*) In command.c, if I change the routine do_string_replot()
    to have execute

    if (volatile_data) refresh_request();
    else

    instead of the current more complicated form
    if ((refresh_mode== 1 || (refresh_mode == 4 && volatile_data))
    && refresh_requested && refresh_ok)

    then also the "g" hotkey starts to work.

    (*) It's great that the patch works also for "plot with image"! I've tested patched gnuplot on Octave 2.9.12, and it works fine for both "plot" and "image(sc)" commands. With gnuplot's "with image", the "set pm3d map" is not necessary for Octave image functions.

     
  • Ethan Merritt
    Ethan Merritt
    2007-06-27

    Logged In: YES
    user_id=235620
    Originator: YES

    Second version
    - autoscale works
    - hotkeys work, except for 'L', 'l'
    - documentation added
    - default is to use refresh only if volatile data is present
    (REFRESH_MODE = 1 will force refresh for non-volatile data)

    Need help with
    - octave testing
    - how to autoscale image data?
    - handling reversed axes
    File Added: persistent_plot_data_27jun2007.patch

     
  • Petr Mikulik
    Petr Mikulik
    2007-06-28

    Logged In: YES
    user_id=31505
    Originator: NO

    I have tested it with octave 2.9.12 and it works fine!

    Otherwise, the following problems can be seen from gnuplot:

    (*) Hitting 'l' for a log scale freezes hotkey functionality, writing always
    refresh not possible and replot is disabled
    the only way to have them working again is to do new plot from the command (same for running from Octave).

    Could you please add a test to mouse.c:builtin_toggle_log() so that do_string("(un)set log") is not called with volatile data, and the warning is printed here?

    (*) as you write, "a" (autoscale) for a plot with image does not work

    (*) There is a strange thing/bug:

    unset grid
    plot '-'
    1 1
    2 2
    e

    Now zoom several times, hit "g" for grid on, then hit "u" for unzoom => grid is lost. This does not happen with "u" in the other usage of the plot command.

     
  • Dan Sebald
    Dan Sebald
    2007-06-28

    Logged In: YES
    user_id=704782
    Originator: NO

    I haven't compiled this patch yet, but I've looked at the patch. First, I like the idea, I think this is a high priority item. But I have a bit of reservation about this approach. I think it doesn't get us in the direction we want to go, which is to eventually store the data, all of it, and then just reuse it in whatever way we see fit. Rather, the patch kind of entangles things more and is putting effort into something that is only incrementally better.

    If we have to reread the data for switching between axis scales or a few other things, it still makes the user or application writer (e.g., octave) require code to check for certain conditions. Don't need to resend data... unless swithing to log scale. The programmer doesn't want to do that. They'll still stay with sending data again and again.

    I have a patch near completion where I've stored all the data in its raw form. I'll post that soon. But I can see now there are some other circumstances where this refresh-on-non-raw-data is a problem. The "fit" code alters the data, I believe. (That smoothing code is a kind of tacked on side bit that does't completely mesh, even with the syntax. I.e., why couldn't it be "with splines" or something?) So there may be another circumstance where refresh_ok is not valid. If I understand the patch there are some cases where this doesn't yet work for splot.

    My gut feeling is that this patch is a start, but it isn't going to get to the eventual goal in its current state. So, it might not be worth trying to finalize this and fix all bugs. I think that combining this patch with the patch I have would be nice. I'll explain when it's posted.

     
  • Ethan Merritt
    Ethan Merritt
    2007-06-28

    Logged In: YES
    user_id=235620
    Originator: YES

    Dan: <<I think it doesn't get us in the direction we want to go, which is to eventually store the data, all of it, and then just reuse it in whatever way we see fit.>>

    Please give a concise example of what you mean. How would it be different from what we are already doing? What part of "all of it" are we failing to store?

    Let's spin off discussion of axis scaling to the mailing list. It's only marginally relevant to the issue of refresh vs replot.

     
1 2 > >> (Page 1 of 2)