It would be nice to move the initialization of the cr tables (during startup) to a child process, so openser can start faster.
If I remember correctly, a similar approach was implemented a while ago for the usrloc module.
i did today some tests with 200K routes. I could reproduce the slow initialization that you (and others) reported. On my machine (not really high end, local mysql, some other load) the module needs around 19 seconds to start.
One thing that helps in this regards are the F_MALLOC improvements i commited in revision 4813. With this changes applied the load time drops to ca. 1/4, around 4-5 seconds. So it seems that the problem is not in the module code, but in the memory manager startup behaviour.
With even bigger rule sets the improvements is probably not that big, but perhaps you can give it a try.
Henning
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm loading 1M+ cr routes and this is taking 60s in 1.3/1.4. I will try to run some tests to see the improvement in this particular case.
Using a separate child process for loading the routes seems to be the right thing to do here.
Regards,
Ovidiu Sas
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
good, please let me know the result. I'll investigate this approach in the next days, i hope this should be not that much work. Perhaps its also possible to partition the data loading (related to #1876664), but this will need probably some bigger changes.
Best regards,
Henning
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
to do the data loading in the child init turned out to be a little bit harder then expected. The problem is that during the config file fixup some data (e.g. carrier mapping) is already needed. The fixup is done before the child init, thus the module startup process needs to be changed.
We investigate at the moment some refactoring in this code, so its probably possible to combine this with some other abstractions. BTW, did you managed to test the startup with this memory management improvements?
Regards,
Henning
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Logged In: YES
user_id=337916
Originator: NO
Hi Ovidiu,
i did today some tests with 200K routes. I could reproduce the slow initialization that you (and others) reported. On my machine (not really high end, local mysql, some other load) the module needs around 19 seconds to start.
One thing that helps in this regards are the F_MALLOC improvements i commited in revision 4813. With this changes applied the load time drops to ca. 1/4, around 4-5 seconds. So it seems that the problem is not in the module code, but in the memory manager startup behaviour.
With even bigger rule sets the improvements is probably not that big, but perhaps you can give it a try.
Henning
Logged In: YES
user_id=1395524
Originator: YES
Hello Henning,
I'm loading 1M+ cr routes and this is taking 60s in 1.3/1.4. I will try to run some tests to see the improvement in this particular case.
Using a separate child process for loading the routes seems to be the right thing to do here.
Regards,
Ovidiu Sas
Logged In: YES
user_id=337916
Originator: NO
Hi Ovidiu,
good, please let me know the result. I'll investigate this approach in the next days, i hope this should be not that much work. Perhaps its also possible to partition the data loading (related to #1876664), but this will need probably some bigger changes.
Best regards,
Henning
Hello Ovidiu,
to do the data loading in the child init turned out to be a little bit harder then expected. The problem is that during the config file fixup some data (e.g. carrier mapping) is already needed. The fixup is done before the child init, thus the module startup process needs to be changed.
We investigate at the moment some refactoring in this code, so its probably possible to combine this with some other abstractions. BTW, did you managed to test the startup with this memory management improvements?
Regards,
Henning