I have restarted my computer and started a new system in Ravel.
I include 1 dataset (CAPE).
Wiring the variable to the graph takes several minutes before it is executed. In the meantime I see the CPU usage increase to over 20%.
Interesting! It does then seem that the delimiter is a factor in how slow
the other file was. These are identical in terms of content and a world
apart in terms of performance.
Definitely to do with date processing. It is a choice between explicitly specifying the date format (%Y-%m in this case), and leaving the date format blank, since the date fields appear in descending order. Leaving the date field blank is orders of magnitude faster than specifying it.
This is handled by my code, (I had to write a special handler for this case to handle dates like 1/2/22) so it looks like I have some optimisation to do. It should run at least as fast as the boost code!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Definitely to do with date processing. It is a choice between explicitly
specifying the date format (%Y-%m in this case), and leaving the date
format blank, since the date fields appear in descending order. Leaving the
date field blank is orders of magnitude faster than specifying it.
This is handled by my code, (I had to write a special handler for this
case to handle dates like 1/2/22) so it looks like I have some optimisation
to do. It should run at least as fast as the boost code!
Status: open Milestone: Pascal Created: Sat Jan 28, 2023 08:18 PM UTC by TomScat Last Updated: Sun Jan 29, 2023 10:19 AM UTC Owner: nobody Attachments:
I have restarted my computer and started a new system in Ravel.
I include 1 dataset (CAPE).
Wiring the variable to the graph takes several minutes before it is
executed. In the meantime I see the CPU usage increase to over 20%.
Well that was easy. The Ravel codebase was using a custom strptime function to sort dates, which curiously was just delegating to the boost time_facet code. Now that civita has been moved to live underneath Ravel, I swapped the use of strptime for the equivalent Civita calls, which have had substantial optimisation performed on them already. Now this example opens "instantly", ie so fast, I couldn't measure how fast it is.
Last edit: High Performance Coder 2023-02-03
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The model.
You've definitely caught a big one here Tom: the attached movie verifies
your experience.
It is probably something to do with the time handling, and maybe the fact
that your file used semi-colons as delimiters rather than commas?
Anyway, after I post this I'll bump this bug up the priority list.
ie_data_cape_TS_3.csv
https://drive.google.com/file/d/1MQuMzUZJsUegNEmwPi3o-TdYfnLmS3gb/view?usp=drive_web
Ts_05_CAPE_v1.rvl
https://drive.google.com/file/d/1vlSoJeGa0f41N0PU8IAcUMicnFr9KsVj/view?usp=drive_web
Ts_05_CAPE_v2.rvl
https://drive.google.com/file/d/1FJ18HEFMNUT8Ifjev7fGc2DY4VJW3iG2/view?usp=drive_web
Ts_05_CAPE_v3.rvl
https://drive.google.com/file/d/1aQNrHIjtzkX8q0JjdKoos9isK4L2mvQb/view?usp=drive_web
UltraSlowRavel20230129.mp4
https://drive.google.com/file/d/1eqVqDZknv87_Vl0iJ5CtuYS8PaDebBG6/view?usp=drive_web
I have changed the csv-file: commas separate the data now. Dates are in YYYY-MM format, no longer the strange YYYY.MM.
Is this recording helpful, or do you prefer something else?
Last edit: TomScat 2023-01-29
The dataset I used.
Interesting! It does then seem that the delimiter is a factor in how slow
the other file was. These are identical in terms of content and a world
apart in terms of performance.
Definitely to do with date processing. It is a choice between explicitly specifying the date format (%Y-%m in this case), and leaving the date format blank, since the date fields appear in descending order. Leaving the date field blank is orders of magnitude faster than specifying it.
This is handled by my code, (I had to write a special handler for this case to handle dates like 1/2/22) so it looks like I have some optimisation to do. It should run at least as fast as the boost code!
This sounds like a reason to NOT get the user to specify the date format at
all, unless the format is something very non-standard.
On Thu, Feb 2, 2023 at 5:56 AM High Performance Coder hpcoder@users.sourceforge.net wrote:
Related
Ravel:
#312There's also a minor thing that the guess file format code does not update the separator input box, but I'll raise a separate ticket for that.
Well that was easy. The Ravel codebase was using a custom strptime function to sort dates, which curiously was just delegating to the boost time_facet code. Now that civita has been moved to live underneath Ravel, I swapped the use of strptime for the equivalent Civita calls, which have had substantial optimisation performed on them already. Now this example opens "instantly", ie so fast, I couldn't measure how fast it is.
Last edit: High Performance Coder 2023-02-03
Ok. That isn't in the v20 yet, correct?
No Tom, which is one reason to look forward to version 21.
Great news Russ. Speed is something we need to focus on as stability is
largely behind us. This is a great leap forward, so to speak...