Thread: [Cppcms-users] Big cache
Brought to you by:
artyom-beilis
Menu
▾
▴
cppcms-users
|
From: Sergey S. <ser...@gm...> - 2012-12-27 04:44:36
|
Artyom Hi. Did you know about redis db? In one high load project i use redis db for caching. I can use native cache in my application? I have this question because if i store some 500 MB or 1GB :) and when memory is full it may breaks. Or better use cppcms_scale? But how i know how match 500 mb equals cache_limit? |
|
From: Shiv S. D. <shi...@gm...> - 2012-12-27 04:47:52
|
A suggestion. If your cache is 1g then put some more ram. I do not think anything will break with such a small cache. On Dec 27, 2012 10:15 AM, "Sergey Shutov" <ser...@gm...> wrote: > Artyom Hi. Did you know about redis db? In one high load project i use > redis db > for caching. I can use native cache in my application? I have this question > because if i store some 500 MB or 1GB :) and when memory is full it may > breaks. > Or better use cppcms_scale? But how i know how match 500 mb equals > cache_limit? > > > > ------------------------------------------------------------------------------ > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, > MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current > with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft > MVPs and experts. ON SALE this month only -- learn more at: > http://p.sf.net/sfu/learnmore_122712 > _______________________________________________ > Cppcms-users mailing list > Cpp...@li... > https://lists.sourceforge.net/lists/listinfo/cppcms-users > |
|
From: Sergey S. <ser...@gm...> - 2012-12-27 04:56:16
|
Shiv Shankar Dayal <shivshankar.dayal@...> writes: > > > A suggestion. If your cache is 1g then put some more ram. I do not think anything will break with such a small cache. Yep, i know. But when you store big data in cache it maybe a problem, you know. |
|
From: Artyom B. <art...@ya...> - 2012-12-27 08:18:28
|
See, you don't cache objects of 500MB in memory... Caching system is designed for pages and other objects of reasonable size that their generation is costly, Also remember that saving objects to cache and using it from cache involves some memcpy... and than serving. if you want to serve huge files/data (and cache them) it is much better to use file system for this. Put them on disk and tell lighttpd, nginx or even apache to serve them, it would do it much better. For example lighttpd would know to transfer them using sendfile directly from disk to socket without even copying the data to user space. Artyom Beilis -------------- CppCMS - C++ Web Framework: http://cppcms.com/ CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/ >________________________________ > From: Sergey Shutov <ser...@gm...> >To: cpp...@li... >Sent: Thursday, December 27, 2012 6:44 AM >Subject: [Cppcms-users] Big cache > >Artyom Hi. Did you know about redis db? In one high load project i use redis db >for caching. I can use native cache in my application? I have this question >because if i store some 500 MB or 1GB :) and when memory is full it may breaks. >Or better use cppcms_scale? But how i know how match 500 mb equals cache_limit? > > >------------------------------------------------------------------------------ >Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, >MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current >with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft >MVPs and experts. ON SALE this month only -- learn more at: >http://p.sf.net/sfu/learnmore_122712 >_______________________________________________ >Cppcms-users mailing list >Cpp...@li... >https://lists.sourceforge.net/lists/listinfo/cppcms-users > > > |
|
From: Artyom B. <art...@ya...> - 2012-12-27 08:24:25
|
On the second thought I maybe misunderstood your question. How do you know that cache_limit is 1G? 1. First of all what is the average size of your objects in cache? Set cache_limit = 1G / avr(object_size). 2. There is a process shared cache that uses shared memory and its size is defined by memory size, it is little bit less efficient than thread shared cache because of less efficient memory allocation algorithm (in 1.0.2 but improved in trunk) but in general you should not feel it. Artyom Beilis -------------- CppCMS - C++ Web Framework: http://cppcms.com/ CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/ >________________________________ > From: Artyom Beilis <art...@ya...> >To: "cpp...@li..." <cpp...@li...> >Sent: Thursday, December 27, 2012 10:18 AM >Subject: Re: [Cppcms-users] Big cache > > >See, you don't cache objects of 500MB in memory... > >Caching system is designed for pages and other objects of reasonable size >that their generation is costly, > >Also remember that saving objects to cache and using it from cache involves some >memcpy... and than serving. > >if you want to serve huge files/data (and cache them) it is much better to use file system for this. > >Put them on disk and tell lighttpd, nginx or even apache to serve them, it would do it much better. > >For example lighttpd would know to transfer them using sendfile directly from disk to socket >without even copying the data to user space. > >Artyom Beilis >-------------- >CppCMS - C++ Web Framework: http://cppcms.com/ >CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/ > > > >>________________________________ >> From: Sergey Shutov <ser...@gm...> >>To: cpp...@li... >>Sent: Thursday, December 27, 2012 6:44 AM >>Subject: [Cppcms-users] Big cache >> >>Artyom Hi. Did you know about redis db? In one high load project i use redis db >>for caching. I can use native cache in my application? I have this question >>because if i store some 500 MB or 1GB :) and when memory is full it may breaks. >>Or better use cppcms_scale? But how i know how match 500 mb equals cache_limit? >> >> >>------------------------------------------------------------------------------ >>Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, >>MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current >>with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft >>MVPs and experts. ON SALE this month only -- learn more at: >>http://p.sf.net/sfu/learnmore_122712 >>_______________________________________________ >>Cppcms-users mailing list >>Cpp...@li... >>https://lists.sourceforge.net/lists/listinfo/cppcms-users >> >> >> >------------------------------------------------------------------------------ >Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, >MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current >with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft >MVPs and experts. ON SALE this month only -- learn more at: >http://p.sf.net/sfu/learnmore_122712 >_______________________________________________ >Cppcms-users mailing list >Cpp...@li... >https://lists.sourceforge.net/lists/listinfo/cppcms-users > > > |
|
From: Sergey S. <ser...@gm...> - 2012-12-27 12:12:32
|
Artyom Beilis <artyomtnk@...> writes: > > See, you don't cache objects of 500MB in memory...Caching system is designed for pages and other objects of reasonable sizethat their generation is costly,Also remember that saving objects to cache and using it from cache involves somememcpy... and than serving.if you want to serve huge files/data (and cache them) it is much better to use file system for this.Put them on disk and tell lighttpd, nginx or even apache to serve them, it would do it much better.For example lighttpd would know to transfer them using sendfile directly from disk to socketwithout even copying the data to user space. Thanks for answer. But i need little bit explain. For example we have social network (real big social network). We have 3,4,5,6... million users. And we have common friends. Algorithm: 1: select friend_id from friends where user_id = 1 (10000 rows) select friend_id from friends where user_id = 2 (6000 rows) (without join's) 2: std::set_intersection (return data we need caching) So i need store friends, common friends, online friends etc... |
|
From: Lee E. <lee...@gm...> - 2012-12-29 19:56:01
|
since this type of data is already stored in some form of a db - why not get most of this information prepared before hand in the db? intersection of lists of friends for people who are likely to be used can be kept as a new table, or as an additional column and you can have a ready and waiting index ready for you to iterate over. the only thing you can not prepare before hand is the online status i would not go for any already made caching solution as a cure for the problem - i would have created approached it as a data mining problem - and calculate everything possible before hand in the background On Thu, Dec 27, 2012 at 2:12 PM, Sergey Shutov <ser...@gm...>wrote: > Artyom Beilis <artyomtnk@...> writes: > > > > > > See, you don't cache objects of 500MB in memory...Caching system is > designed > for pages and other objects of reasonable sizethat their generation is > costly,Also remember that saving objects to cache and using it from cache > involves somememcpy... and than serving.if you want to serve huge > files/data > (and > cache them) it is much better to use file system for this.Put them on disk > and > tell lighttpd, nginx or even apache to serve them, it would do it much > better.For > example lighttpd would know to transfer them using sendfile directly from > disk > to > socketwithout even copying the data to user space. > > > > Thanks for answer. But i need little bit explain. For example we have > social > network (real big social network). We have 3,4,5,6... million users. > And we have common friends. > Algorithm: > > 1: > select friend_id from friends where user_id = 1 (10000 rows) > select friend_id from friends where user_id = 2 (6000 rows) > > (without join's) > > 2: > std::set_intersection (return data we need caching) > > So i need store friends, common friends, online friends etc... > > > > ------------------------------------------------------------------------------ > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, > MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current > with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft > MVPs and experts. ON SALE this month only -- learn more at: > http://p.sf.net/sfu/learnmore_122712 > _______________________________________________ > Cppcms-users mailing list > Cpp...@li... > https://lists.sourceforge.net/lists/listinfo/cppcms-users > -- -- lee Lee Elenbaas lee...@gm... |
|
From: Artyom B. <art...@ya...> - 2012-12-29 21:05:30
|
See, I don't know what exactly the data you have I can suggest: Separete your date into 3 categories: 1. In memory so the access is very fast (think how do you update/invalidate it) also think what is good to keep in memory and what is not 2. In cache what you need, remember that serialization and deserialization may cost as well. 3. Prefer to cache HTML rather than objects (as you save HTML generation) If you have some data that you can prefetch and it is not updated frequently do it. In the application I developed for my clients I use in-memory data structures/in-memory db very intensivly as it makes things faster and easier. But you should keep in mind how would you scale and when (now in near future in far future). Artyom Beilis -------------- CppCMS - C++ Web Framework: http://cppcms.com/ CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/ >________________________________ > From: Lee Elenbaas <lee...@gm...> >To: cpp...@li... >Sent: Saturday, December 29, 2012 9:55 PM >Subject: Re: [Cppcms-users] Big cache > > >since this type of data is already stored in some form of a db - why not get most of this information prepared before hand in the db? >intersection of lists of friends for people who are likely to be used can be kept as a new table, or as an additional column and you can have a ready and waiting index ready for you to iterate over. >the only thing you can not prepare before hand is the online status >i would not go for any already made caching solution as a cure for the problem - i would have created approached it as a data mining problem - and calculate everything possible before hand in the background > > > > >On Thu, Dec 27, 2012 at 2:12 PM, Sergey Shutov <ser...@gm...> wrote: > >Artyom Beilis <artyomtnk@...> writes: >> >> >>> >>> See, you don't cache objects of 500MB in memory...Caching system is designed >> >>for pages and other objects of reasonable sizethat their generation is >>costly,Also remember that saving objects to cache and using it from cache >>involves somememcpy... and than serving.if you want to serve huge files/data >>(and >>cache them) it is much better to use file system for this.Put them on disk and >> >>tell lighttpd, nginx or even apache to serve them, it would do it much >>better.For >>example lighttpd would know to transfer them using sendfile directly from disk >>to >>socketwithout even copying the data to user space. >> >> >> >>Thanks for answer. But i need little bit explain. For example we have social >>network (real big social network). We have 3,4,5,6... million users. >>And we have common friends. >>Algorithm: >> >>1: >>select friend_id from friends where user_id = 1 (10000 rows) >>select friend_id from friends where user_id = 2 (6000 rows) >> >>(without join's) >> >>2: >>std::set_intersection (return data we need caching) >> >>So i need store friends, common friends, online friends etc... >> >> >> >>------------------------------------------------------------------------------ >>Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, >>MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current >>with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft >>MVPs and experts. ON SALE this month only -- learn more at: >>http://p.sf.net/sfu/learnmore_122712 >>_______________________________________________ >>Cppcms-users mailing list >>Cpp...@li... >>https://lists.sourceforge.net/lists/listinfo/cppcms-users >> > > >-- >-- >lee >Lee Elenbaas >lee...@gm... > >------------------------------------------------------------------------------ >Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, >MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current >with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft >MVPs and experts. SALE $99.99 this month only -- learn more at: >http://p.sf.net/sfu/learnmore_122912 >_______________________________________________ >Cppcms-users mailing list >Cpp...@li... >https://lists.sourceforge.net/lists/listinfo/cppcms-users > > > |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X