Re: [Mvpmc-devel] reducing memory use in MythTV client
Status: Alpha
Brought to you by:
gettler
From: Roger H. <ra...@wi...> - 2008-03-10 00:14:05
|
ge...@ac... wrote: > On Sun, 09 Mar 2008 16:46:39 CDT, Roger Heflin wrote: >> Change for 1024 to 256: >> diff --git a/dongle/filesystem/tree/etc/rcS b/dongle/filesystem/tree/etc/rcS >> index f3ad3fc..d24788c 100755 >> --- a/dongle/filesystem/tree/etc/rcS >> +++ b/dongle/filesystem/tree/etc/rcS >> @@ -58,7 +58,7 @@ progress >> # Create the tmpfs filesystem and mount it on top of the squashfs root >> # filesystem with unionfs. >> # >> -mount -t tmpfs -o "size=1M" tmpfs /memory >> +mount -t tmpfs -o "size=256K" tmpfs /memory >> mount -t unionfs -o dirs=/memory=rw:/=ro unionfs /union >> cd /union >> mkdir live >> > > I would not have expected this to help. My understanding of tmpfs is that > it only takes up as much memory as you are actually using. > I am not sure of that on this machine, given it has very limited paging ability, and may not have the stuff to do that correctly. But you might be correct. >> Change for no longer doing pre-allocation: >> diff --git a/src/main.c b/src/main.c >> index ee9cc2c..9734c94 100644 >> --- a/src/main.c >> +++ b/src/main.c >> @@ -1164,7 +1164,7 @@ mvpmc_main(int argc, char **argv) >> * that mvpmc doesn't get sluggish after running for a while. >> */ >> { >> -#define NPAGES 1024 >> +#define NPAGES 256 >> unsigned long heap = (unsigned long)sbrk(0); >> char *pages[NPAGES], *buffer = NULL; >> int i, sz, k = 0; > > I would not expect this to help, either. The purpose of this code is > to grow the heap beyond what is needed initially to make it less likely > that mvpmc will have to contend with other parts of the system for memory > during media playback. The problem is it makes it difficult to monitor how much memory is being used for various functions, ie 4MB is always allocated so you cannot really tell until the memory passes that. I assume it stops things like too many telnet's using up ram that it might need, but with and without it I did not see a increase in issues. > >>> I was just looking at the debug output while loading MythTV show >>> listings, and I see that MythTV is apparently sending the full details >>> of every show in the database. I can't tell from the debug output >>> whether mvpmc is attempting to store all that or if it is dynamically >>> filtering it. >> From the code, it does appear to be storing it *ALL* there is a really large >> >> structure and mallocs are going on for the larger things (descriptions, title >> s >> and such), it is probably possible to remove some of the fields (I was thinki >> ng >> to put them inside a #ifdef MINMEM section so that they could be later added >> back if needed. Also it appears that it might be wise to not load some of t >> he >> information that is not needed for all of the shows (descriptions-100+ chars > > Yes, all of the data from the myth backend is cached locally. This makes > the access very fast, but it does limit the amount of recordings that you > can have. > > A MINMEM build option would make sense. You could use it to truncate text > fields, and simply eliminate other fields that mvpmc does not use. However, > since libcmyth is used on other platforms with more memory, if someone does > go down the path of implementing this, please do make it a build or runtime > option. It was thinking an #ifdef controlling the limiting of memory, that way if it is not set it all goes in. It would also control the throwing out of any unneeded mythtv fields and/or large fields that weren't needed at the time. > >> each * 500 shows if 50k). Also all of the allocations have some bookkeeping >> >> meaning that each malloc done through the internal memory management function >> >> has several bytes extra for keeping track of access, though it is unclear to >> me >> if things would get screwed up if this was removed. > > The whole point of the internal memory management is to reduce memory usage. > The allocations have reference counts so that different parts of code can > use the data without fear that it will be free'd out from under it. This > would be far less of a problem without threads. The only other option would > be to copy the data, which would take more memory. Or implement some locking around things like the proginfo structure, if you don't have the lock then you have to wait for it to touch it. I suspect that there may be more than one copy of the proginfo structure for a short period of time around a program delete/update happening, and that results in the really show and is why it appears to happen more around certain events (deletes) but also happens at other times if things line up wrong. Is this possible given the memory management structure in it? It looked like it was possible. > >>> I assume that when it comes to recording groups, mvpmc is dynamically >>> filtering and throwing away records that don't match the filter. So >>> while it may not improve memory usage, it would improve performance if >>> we could tell MythTV more specifically what groups we wanted. >> Not that I can tell, it appears to load it all. > > Correct, all the program data is cached in memory so that it does not need > to be constantly fetched. The data is only refetched when an event from > the backend tells it to. > >>> What would improve memory is if we could tell MythTV to sort the results >>> in our preferred way and return only the first screenfull of titles. If >>> we can't do that via the MythTV protocol, maybe we need to resort to >>> using the database connection (as much as I'd hate adding to that problem). >> I would guess just load the titles and subtitles and nothing else at all unti >> l >> it is needed to be displayed and then through that away immediately, given th >> e >> size of the structure and the number of things it should reduce the memory us >> ed >> quite a bit. > > I didn't think myth gave you a choice but to get it all. So keep in mind > that you might be saving memory, but with 500 recordings, you would risk > slowing down your interface quite a bit. This is true for under 100 > recordings as well. The interface sped up quite noticeably after I added > the event code and stopped fetching the recordings list every time the > menu was changed. Given that the interface *FAILS* badly now on the mvpmc, and given the number of times the refresh happens and that happens pretty fast so even with 500 does not appear to be a serious issue. I can usually watch 1 maybe 2 shows before having to reboot it. And given the number of refreshes I see just sitting there and the speed of them, I don't think that will be an issue. > > BTW, you and Tom both mentioned that you need to restart mvpmc every so > often, pressumably due to memory problems. Do you see the memory useage > of mvpmc grow over time? And does building librefmem with DEBUG defined > point you to any memory leaks? Yes, I see it growing, if it hits over 10MB and generally gets horribly slow and/or has to be green buttoned. Often I cannot even telnet into it to see how much ram it is using, if I am already in there, the ram usage does line up with the issues. I don't think at this point we even have enough ram to load the recordings with the librefmem turned on, I took a look at that, I also checked some of the code paths that are common and did not see anything that looked wrong. But just going in and doing a few delete was significant enough to cause the issue, also going into weather and then trying to view a mythtv video will typically almost always cause a fatal crash when you view the video (it is able to pull up the listing, but starting a video is fatal). > > Jon > > And I have every intention of setting up the minmem code in such a way that it can be enabled when needed, and easily disabled to be fully functional, given that otherwise I know you won't accept the patches, and I do understand why. Roger |