From: Vaibhav S. N. <Vai...@no...> - 2005-01-23 06:50:13
|
Hi, I have to build a simulator, to check the scalability limits in memory management of linux. For this, the simulator will provide a large amount of RAM (in GBs), even if the amout of physical memory is only 512MB. Can UML be used for this purpose? I had seen the "mem" option in the kernel switches option of UML. Can UML simulate such a case? If it does, how UML handles this? Specifically, can the UML kernel allocate all this amount of memory (leaving aside the page faults which will be generated in case of read/write requests to the allocated pages, for it has to use the actual RAM)? Say, for example if I allocate 4 GB, will the UML kernel allocate all the amount in the simulated RAM. (maybe it does it in the some swap, or the /tmp, as the documentation says for the "mem" usage <http://user-mode-linux.sourceforge.net/switches.html <http://user-mode-linux.sourceforge.net/switches.html> > . If I have wrong interpretation of the "mem" option in UML, please do correct me. TIA, Vaibhav |
From: Doug D. <do...@ea...> - 2005-01-23 08:27:31
|
Vaibhav Sharma, Noida wrote: > Hi, > I have to build a simulator, to check the scalability limits in memory > management of linux. For this, the simulator will provide a large > amount of RAM (in GBs), even if the amout of physical memory is only 512MB. > > Can UML be used for this purpose? I had seen the "mem" option in the > kernel switches option of UML. Can UML simulate such a case? If it does, > how UML handles this? Specifically, can the UML kernel allocate all this > amount of memory (leaving aside the page faults which will be generated > in case of read/write requests to the allocated pages, for it has to use > the actual RAM)? Say, for example if I allocate 4 GB, will the UML > kernel allocate all the amount in the simulated RAM. (maybe it does it > in the some swap, or the /tmp, as the documentation says for the "mem" > usage <_http://user-mode-linux.sourceforge.net/switches.html_> . > > If I have wrong interpretation of the "mem" option in UML, please do > correct me. > > TIA, > Vaibhav Mr. Sharma, What you are trying to do will work, but not for large amounts of memory. UML runs the client using a single user mode memory block as the entire client's core. Thus the clients core size is limited to what a single task can allocate as plain memory. With x86 architechure cpus, this is 2G or 3G depending on how you build a kernel (but usually 2G). There are no paging mechanism here so there is no way to make it "gigabytes" Others can correct me if I have the exact numbers wrong. -- -------------------------------------------------------------------- Doug Dumitru 800-470-2756 (610-237-2000) EasyCo LLC do...@ea... http://easyco.com -------------------------------------------------------------------- |
From: Rob L. <ro...@la...> - 2005-01-23 10:53:04
|
On Sunday 23 January 2005 03:28 am, Doug Dumitru wrote: > Mr. Sharma, > > What you are trying to do will work, but not for large amounts of > memory. UML runs the client using a single user mode memory block as > the entire client's core. Thus the clients core size is limited to what > a single task can allocate as plain memory. With x86 architechure cpus, > this is 2G or 3G depending on how you build a kernel (but usually 2G). > There are no paging mechanism here so there is no way to make it > "gigabytes" > > Others can correct me if I have the exact numbers wrong. UML uses mmap (generally of a /tmp file) as physical memory. This can be way bigger than physical memory on the machine, sure, and the parent kernel will transparently use the mmaped file as backing store, demand paged and all that. (Just like it would for any other application.) You could run out of virtual address space, though. Trying to map more than a few gigabytes on a 32 bit machine is unlikely to work. Might I suggest buying a cheap x86-64 system (you can get 'em at Fry's a couple hundred bucks), feed it a cheap 300 gigabyte IDE drive, and have fun. Rob |
From: Blaisorblade <bla...@ya...> - 2005-01-24 18:23:54
|
On Sunday 23 January 2005 10:51, Rob Landley wrote: > On Sunday 23 January 2005 03:28 am, Doug Dumitru wrote: > > Mr. Sharma, > > > > What you are trying to do will work, but not for large amounts of > > memory. UML runs the client using a single user mode memory block as > > the entire client's core. Thus the clients core size is limited to what > > a single task can allocate as plain memory. With x86 architechure cpus, > > this is 2G or 3G depending on how you build a kernel (but usually 2G). > > There are no paging mechanism here so there is no way to make it > > "gigabytes" > > > > Others can correct me if I have the exact numbers wrong. > > UML uses mmap (generally of a /tmp file) as physical memory. This can be > way bigger than physical memory on the machine, sure, and the parent kernel > will transparently use the mmaped file as backing store, demand paged and > all that. (Just like it would for any other application.) > > You could run out of virtual address space, though. Trying to map more > than a few gigabytes on a 32 bit machine is unlikely to work. > > Might I suggest buying a cheap x86-64 system (you can get 'em at Fry's a > couple hundred bucks), feed it a cheap 300 gigabyte IDE drive, and have > fun. This is not (yet) exactly supported... a lot of work is going on this, and somebody was even able to compile and run (successfully?) some x86_64 umls... However, consider it still as a developer-only tool for now. 2.6.11 could effectively be a good x86_64 tree, but do not yet use it for anything critical. Clearly, 32-bit UMLs run well on x86_64 boxes, but don't have the 64-bit advantages... -- Paolo Giarrusso, aka Blaisorblade Linux registered user n. 292729 http://www.user-mode-linux.org/~blaisorblade |
From: Henrik N. <um...@hn...> - 2005-01-23 18:28:24
|
On Sun, 23 Jan 2005, Doug Dumitru wrote: > What you are trying to do will work, but not for large amounts of memory. > UML runs the client using a single user mode memory block as the entire > client's core. Thus the clients core size is limited to what a single task > can allocate as plain memory. It should be possible to work around this using the kernel HIGHMEM support. Would also be a good environment to verify the HIGHMEM support in the kernel in general. In HIGHMEM mode the kernel doen't have the whole physical RAM mapped in it's address space, remapping memory as required. This doesn't increase the per-process limitations, but allows for the total sum or processes + kernel memory to be much greater. Regards Henrik |
From: Rob L. <ro...@la...> - 2005-01-23 19:26:50
|
On Sunday 23 January 2005 01:27 pm, Henrik Nordstrom wrote: > On Sun, 23 Jan 2005, Doug Dumitru wrote: > > What you are trying to do will work, but not for large amounts of memory. > > UML runs the client using a single user mode memory block as the entire > > client's core. Thus the clients core size is limited to what a single > > task can allocate as plain memory. > > It should be possible to work around this using the kernel HIGHMEM > support. Would also be a good environment to verify the HIGHMEM support in > the kernel in general. The client kernel's highmem suport is unlikely to do much, I'd think. Not unless it's unmapping and remapping multiple mmaps. (There's large file support, but trying to mmap a 5 gig chunk out of a large file can't work: what would that mean? How could you generate an offset into the last meg? What would the pointer _be_?) The parent kernel's highmem support still doesn't provide more than 4 gigabytes per application, and the UML kernel is one application. (And that's with the 4 gig kernel/4 gig user patches from... Ingo Molnar, I think.) Highmem lets the system as a whole use PAE (up to 64 gigs physical memory), but the kernel is basically doing something very like the old DOS expanded memory behind the scenes, swapping around page tables so each process sees a different set of physical memory. pages (The overhead isn't quite so bad since it's doing that anyway as part of protecting process's memory from each other...) Hence the x86-64 system. They really are getting cheap these days. And when the PS3 comes out, that's going to be a 64 bit (mutant) Power PC for somewhere around $300 fully loaded, which runs Linux internally. (Probably with binary only drivers out the wazoo, but still...) Rob |
From: Henrik N. <um...@hn...> - 2005-01-24 10:02:19
|
On Sun, 23 Jan 2005, Rob Landley wrote: > The client kernel's highmem suport is unlikely to do much, I'd think. > Not unless it's unmapping and remapping multiple mmaps. (There's large > file support, but trying to mmap a 5 gig chunk out of a large file can't > work: what would that mean? How could you generate an offset into the > last meg? What would the pointer _be_?) off_t is 64 bits in 64-bit file pointer mode, so there is no problem to mmap portions of an multi-gigabyte "ram" file within a 32-bit address space. > The parent kernel's highmem support still doesn't provide more than 4 > gigabytes per application, and the UML kernel is one application. Yes, but with HIGHMEM UML in theory can provide as as much ram to each single process running below it as it can for a single process. So yes, each process in the UML would still be limited, but the total RAM usage of UML would be practically unlimited. And in addition a very good environment for debugging HIGHMEM in general as you could set up things to only provide a very limited directly addressible address space with the rest remapped by HIGHMEM. With HIGHMEM the host process size limitations has an direct effect on the maximum process size possible within the UML, but not on the RAM size of UML. In addition running an UML with HIGHMEM support does not depend on the host HIGHMEM support, but performance will obviously suffer if UML needs to swap.. > (And that's with the 4 gig kernel/4 gig user patches from... Ingo > Molnar, I think.) Highmem lets the system as a whole use PAE (up to 64 > gigs physical memory), but the kernel is basically doing something very > like the old DOS expanded memory behind the scenes, swapping around page > tables so each process sees a different set of physical memory. pages > (The overhead isn't quite so bad since it's doing that anyway as part of > protecting process's memory from each other...) Exctaly, and UML can do something very similar with the help of mmap(). Regards Henrik |
From: Blaisorblade <bla...@ya...> - 2005-01-24 18:35:39
|
On Monday 24 January 2005 11:02, Henrik Nordstrom wrote: > On Sun, 23 Jan 2005, Rob Landley wrote: > > The client kernel's highmem suport is unlikely to do much, I'd think. > > Not unless it's unmapping and remapping multiple mmaps. It does this, indeed and sadly (that's why it's so slow). > > (There's large > > file support, but trying to mmap a 5 gig chunk out of a large file can't > > work: what would that mean? How could you generate an offset into the > > last meg? What would the pointer _be_?) > off_t is 64 bits in 64-bit file pointer mode, so there is no problem to > mmap portions of an multi-gigabyte "ram" file within a 32-bit address > space. Offset can sure be 64-bit wide, but he refers to mmaping a 5G-wide chunk of memory, and to a "pointer offset"... you cannot mmap more than 4G in memory even with 4G/4G. Yes, you can do it like he says (i.e. mmapping back and forth). > > The parent kernel's highmem support still doesn't provide more than 4 > > gigabytes per application, and the UML kernel is one application. > Yes, but with HIGHMEM UML in theory can provide as as much ram to each > single process running below it as it can for a single process. So yes, > each process in the UML would still be limited, but the total RAM usage of > UML would be practically unlimited. And in addition a very good > environment for debugging HIGHMEM in general as you could set up things to > only provide a very limited directly addressible address space with the > rest remapped by HIGHMEM. This point holds actually. I think however that Henrik will agree that this is not yet implemented, and (maybe) nor in sight... about this, I would add somebody suggested using /proc/mm to this aim: since a single address space is not enough to map all the 64 Giga on a PAE system, he suggested using multiple address spaces and switching between them to use memory as cache for MySQL (IIRC)... this was also posted on this list. Also, since x86_64 systems are so cheap (I'm writing from one of them), x86_64 UML is going to be a reality soon, and we're in short of manpower, the logical conclusion is that this is unlikely to be implemented... unless it's very simple; do you think that making sure the offset inside /tmp/vm_file-*** is 64-bit wide would be enough? > With HIGHMEM the host process size limitations has an direct effect on the > maximum process size possible within the UML, but not on the RAM size of > UML. In addition running an UML with HIGHMEM support does not depend on > the host HIGHMEM support, but performance will obviously suffer if UML > needs to swap.. > > (And that's with the 4 gig kernel/4 gig user patches from... Ingo > > Molnar, I think.) I confirm > > Highmem lets the system as a whole use PAE (up to 64 > > gigs physical memory), but the kernel is basically doing something very > > like the old DOS expanded memory behind the scenes, swapping around page > > tables so each process sees a different set of physical memory. pages > > (The overhead isn't quite so bad since it's doing that anyway as part of > > protecting process's memory from each other...) > Exctaly, and UML can do something very similar with the help of mmap(). Which is too slow, though... -- Paolo Giarrusso, aka Blaisorblade Linux registered user n. 292729 http://www.user-mode-linux.org/~blaisorblade |
From: Jeff D. <jd...@ad...> - 2005-01-23 22:48:43
|
Vai...@no... said: > I have to build a simulator, to check the scalability limits in memory > management of linux. For this, the simulator will provide a large > amount of RAM (in GBs), even if the amout of physical memory is only > 512MB. > > Can UML be used for this purpose? I had seen the "mem" option in the > kernel switches option of UML. Can UML simulate such a case? Yes, it should be fine for what you want. > If it > does, how UML handles this? Specifically, can the UML kernel allocate > all this amount of memory It'll allocate a temp file of the appropriate size. If it's larger than what it can use for physical memory, the rest will be allocated as highmem. If you leave 2-level pagetables on, then you're limited to 4G. If you turn on 3-level pagetables, then you get up to 128G. do...@ea... said: > UML runs the client using a single user mode memory block as the > entire client's core. Thus the clients core size is limited to what > a single task can allocate as plain memory. No, it only maps the physical memory as a block. The highmem part is mapped on demand, as with i386. ro...@la... said: > You could run out of virtual address space, though. Trying to map > more than a few gigabytes on a 32 bit machine is unlikely to work. With highmem, it's not all mapped at once. ro...@la... said: > Not unless it's unmapping and remapping multiple mmaps. It is. > (There's > large file support, but trying to mmap a 5 gig chunk out of a large > file can't work: It does it a page at a time. > what would that mean? How could you generate an > offset into the last meg? The LFS interfaces take 64-bit offsets. > What would the pointer _be_?) What pointer? > The parent kernel's highmem support still doesn't provide more than 4 > gigabytes per application, and the UML kernel is one application. The host kernel's highmem support is totally irrelevant. There doesn't even need to be highmem support on the host. Jeff |
From: Rob L. <ro...@la...> - 2005-01-24 06:12:11
|
On Sunday 23 January 2005 08:11 pm, Jeff Dike wrote: > It'll allocate a temp file of the appropriate size. If it's larger than > what it can use for physical memory, the rest will be allocated as highmem. > If you leave 2-level pagetables on, then you're limited to 4G. If you turn > on 3-level pagetables, then you get up to 128G. Interesting. I wonder how that works? (PAE on x86 only lets you have 64G.) > ro...@la... said: > > Not unless it's unmapping and remapping multiple mmaps. > > It is. But an individual process running under UML can still only have 4 gigabytes. > > (There's > > large file support, but trying to mmap a 5 gig chunk out of a large > > file can't work: > > It does it a page at a time. > > > what would that mean? How could you generate an > > offset into the last meg? > > The LFS interfaces take 64-bit offsets. Yeah. (Ouch. But yeah, I can see that working...) > > What would the pointer _be_?) > > What pointer? The virtual or physical address used to access memory. (I'm guessing userspace programs running under UML are limited to 4 gigs, and UML is using page indexes and is thus limited to 4 billion pages, not 4 billion bytes.) > > The parent kernel's highmem support still doesn't provide more than 4 > > gigabytes per application, and the UML kernel is one application. > > The host kernel's highmem support is totally irrelevant. There doesn't > even need to be highmem support on the host. Just large file support, and the ability to mmap up to 4 gigs of memory at a time (with a starting offset potentially above 4 gigabytes), and unmap it and map a different 4 gigs when you switch to the next process... Makes my brain hurt just thinking about it, but that could be caffeine withdrawl... > Jeff Rob |
From: Henrik N. <um...@hn...> - 2005-01-24 10:12:29
|
On Mon, 24 Jan 2005, Rob Landley wrote: > Interesting. I wonder how that works? (PAE on x86 only lets you have 64G.) Thats only an limitation of the CPU support for PAE. As UML is using mmap() other limits apply and these limits is mainly set by the UML pagetable structures. > But an individual process running under UML can still only have 4 gigabytes. Not even that as we do not have a 4GB/4GB split in UML. >>> What would the pointer _be_?) >> >> What pointer? > > The virtual or physical address used to access memory. (I'm guessing > userspace programs running under UML are limited to 4 gigs, and UML is using > page indexes and is thus limited to 4 billion pages, not 4 billion bytes.) In theory UML HIGHMEM could be made to support even more pages, but it's besides the point. > Just large file support, and the ability to mmap up to 4 gigs of memory at a > time (with a starting offset potentially above 4 gigabytes), and unmap it and > map a different 4 gigs when you switch to the next process... > > Makes my brain hurt just thinking about it, but that could be caffeine > withdrawl... Thinking of HIGHMEM does hurt by design as you no longer have the equivalence of pointers to userspace in kernel land and must explicitly ask to have the needed userspace areas mapped/unmapped as needed, giving you the required pointers.. Most of this is dealt with automatically in copy_to/from_user, but some parts of the kernel uses other means to access userspace data and thus need to be very careful. Regards Henrik |
From: Bodo S. <bst...@fu...> - 2005-01-24 11:25:15
|
Henrik Nordstrom wrote: > On Mon, 24 Jan 2005, Rob Landley wrote: > >> Interesting. I wonder how that works? (PAE on x86 only lets you have >> 64G.) > > > Thats only an limitation of the CPU support for PAE. As UML is using > mmap() other limits apply and these limits is mainly set by the UML > pagetable structures. > >> But an individual process running under UML can still only have 4 >> gigabytes. > > > Not even that as we do not have a 4GB/4GB split in UML. > >>>> What would the pointer _be_?) >>> >>> >>> What pointer? >> >> >> The virtual or physical address used to access memory. (I'm guessing >> userspace programs running under UML are limited to 4 gigs, and UML is >> using >> page indexes and is thus limited to 4 billion pages, not 4 billion >> bytes.) > > > In theory UML HIGHMEM could be made to support even more pages, but it's > besides the point. I agree with nearly all of your good explanations, but have to write my objection to this one. Yes, using 3-level pagetables and mmap64, uml can access very much pagetables. But that's not all, what's needed to have *support* for big memory. Linux needs to have one struct page for each physical mem page, which in the simple case of UML are placed in one memmap array. This array needs to be accessible permanently to the kernel, i.e. it has to be in low-mem. AFAIK, one struct page is 44 bytes in size, thus the array is about 1% of the entire physical mem in size. Assume we are using skas3 on a 3GB/1GB host that provides 3GB address space for user processes (i.e. 3GB address space for UML-Kernel, too). Let the UML kernel need about 1GB for code, vmalloc-area and data, that also need to reside in lowmem. Then, there remain 2GB space for the page-structures, which limits supportable memory to about 200GB. That still is very much compared to i386, where on a 3GB/1GB host kernels address space is 1GB only. Since memmap shouldn't use more than about one third of this, i386 can support max. about 32GB of physical memory (some people talk about 16GB only). BTW: This is the reason for having the 2GB/2GB option in i386. Bodo > >> Just large file support, and the ability to mmap up to 4 gigs of >> memory at a >> time (with a starting offset potentially above 4 gigabytes), and unmap >> it and >> map a different 4 gigs when you switch to the next process... >> >> Makes my brain hurt just thinking about it, but that could be caffeine >> withdrawl... > > > Thinking of HIGHMEM does hurt by design as you no longer have the > equivalence of pointers to userspace in kernel land and must explicitly > ask to have the needed userspace areas mapped/unmapped as needed, giving > you the required pointers.. > > Most of this is dealt with automatically in copy_to/from_user, but some > parts of the kernel uses other means to access userspace data and thus > need to be very careful. > > Regards > Henrik > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting > Tool for open source databases. Create drag-&-drop reports. Save time > by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc. > Download a FREE copy at http://www.intelliview.com/go/osdn_nl > _______________________________________________ > User-mode-linux-devel mailing list > Use...@li... > https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel |
From: <st...@ni...> - 2005-01-24 11:32:55
|
> Linux needs to have one struct page for each physical mem page, which in > the > simple case of UML are placed in one memmap array. > This array needs to be accessible permanently to the kernel, i.e. it > has to be in low-mem. AFAIK, one struct page is 44 bytes in size, thus the > array is about 1% of the entire physical mem in size. You can mangle with the page-size as-well. You will break some few programs by increasing the page-size too 16K instead of 4K, but most software should respect the output mmap gives it, etc. Stian |
From: Bodo S. <bst...@fu...> - 2005-01-24 16:17:39
|
Jeff Dike wrote: > bst...@fu... said: > >>But that's not all, what's needed to have *support* for big memory. >>Linux needs to have one struct page for each physical mem page, which >>in the simple case of UML are placed in one memmap array. > > > The original poster is interested in the memory scalability of Linux, so > presumably he'll be looking into things like this :-) > > And it's not only this. It's the kernel stacks and page tables you'll have > for all the processes you'll be running on this large machine, inodes and other > file system data for all the files they have open, etc. Yes. That's the reason, why I estimated 1GB of memory for kernels internal use, additionally to the memmap. > > Jeff |