#228 support frequency conversion (compaction) from hourly to daily

[Sven Schreiber]

The following artificial example fails with an "unsupported" error:

<hansl>

nulldata 100
setobs 24 10:4    --time-series # hourly,  starts at day 10, hour 4
series day = $obsmajor
series hour = $obsminor
series check = day==10 ? 1 : 0 # is 1 for less than 24 hours

dataset compact 7 # error, not supported
</hansl> 

It is not obvious how to convert an hourly dataset to daily, apart from writing a bunch of loops to do it "by hand". So it would be nice to have.
Note: if instead "dataset compact 1" is used, there is no error, and the retained obs numbers are 11,12,13 -- i.e., the incomplete first day 10 is apparently simply skipped. This is absolutely defensible, of course. (And furthermore the additional option "last" indeed induces gretl to grab the last value of day 10 and then the obs 10 is kept around, good.)

---

(There is a related bug in GUI usage, I think, but I'm going to create a separate ticket in the bug tracker for that.)

[Riccardo "Jack" Lucchetti]

Here's an admittedly convoluted and unintuitive variation on your script to do what you want, while the needed compaction feature is added to the program:

nulldata 100
setobs 24 10:4 --time-series # hourly,  starts at day 10, hour 4
series day = $obsmajor
series hour = $obsminor

x = normal()
M = aggregate(x, day, sum)

xport = M[,3]
cnameset(xport, "foobar")
store mydailydata.gdt --matrix=xport
open mydailydata.gdt

setobs 7 2025-07-01

[Allin Cottrell]

We agreed some time ago that it should always be possible to compact data with $pd > 1 to frequency 1 (although the interpretation of "1" as annual will apply only in certain cases). And this seems to work fine. So a simple way of achieving the above would be:

dataset compact 1
d1 = sprintf("%d", $obsmajor[1])
setobs 7 @d1 --time-series

Or, if you have an actual date for the first full day, substitute that date for @d1 above.

I guess we could implement compact 7 for hourly data in just that way internally.

[Allin Cottrell]

That's now done in git. Is there any case for supporting compact 5 and 6 for hourly data (in the same way)?

[Sven Schreiber]

Thanks, Allin. Let me point out again that the compaction from 24 to 7 was already possible in the GUI, see the (only open) bug ticket. (The minor glitch described there does not really matter for the main point.) So I wonder if you simply hooked that mechanism up for scripting as well, or whether there are now two implementations of the same thing internally.
About your question to 5 or 6: Not sure. My actual data wasn't financial, so business days or not didn't matter. When you have hourly market data, that may differ I guess.

[Allin Cottrell]

On GUI vs CLI: basically, I just removed a blockage on the CLI path to the 24-to-7 compaction code path (but also made it slightly more efficient for both cases, I think). Extending the favour to 5- or 6-day data would be trivial, if we think it makes sense.

[Sven Schreiber]

OK, I haven't used it myself, but here's an example of hourly data only on business days, it seems: https://data.nasdaq.com/databases/FXH1

[Allin Cottrell]

In git we now have compaction from hourly to 5-, 6- or 7-day daily. However, there are some related points that may be worth discussing so I'm leaving this open for now as a reminder.

[Sven Schreiber]

Turns out there was nothing left to discuss, so this is closed as well.