modularkernel-developer Mailing List for Modular Kernel
Status: Pre-Alpha
Brought to you by:
musteresel
Menu
▾
▴
modularkernel-developer — Developer maling list for modular kernel project
You can subscribe to this list here.
| 2010 |
Jan
|
Feb
|
Mar
|
Apr
(19) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|---|
|
From: Tomas K. <tom...@gm...> - 2010-04-24 09:22:30
|
Hi,
On Wed, Apr 21, 2010 at 4:10 PM, Daniel Oertwig
<mus...@go...> wrote:
> Hi :)
>
> Sorry for not being there doing something (visible) for such a long time.
It's ok with me.
> In fact, i had a complete reimplementation (but of course with nearly
> the same design) of the physical memory management, that is actually
> followed by a redesign of the virtual memory management and a small
> design change concerning multitasking.
How are we going to try out the (user-space?) multitasking?
AFAIK, we don't have a way to start user-space processes yet.
> So there are a lot of changes that i didn't pushed yet, i hope i can put
> it online tomorrow or the day after tomorrow.
> I will try to take as much of your comments as possible and add my own
> to make the code easier to understand this time.
Thanks.
> The last code first wasn't coded with the idea of other people working
> on it, too ;)
> Its sometimes just hard to code and comment at the same time, but i hope
> i won't forget commenting anymore ^^
I think you should comment as you see fit.
I am glad that my comments/macros do not disturb the original coder :)
> About the physical memory management:
>
> I removed the thing about the "base", from which on we were managing the
> memory. Now everything is managed. There is also a change in "using" the
> physical memory management. The lower level functions now return a value
> indicating whether they could do the job they were told to and the real
> "return" value is handled using pointer parameters.
This "return" value is something else than just an integer (like long)?
If it's only an integer, I think I would prefer to return the error code,
instead of writing it somewhere into memory. As you save space on stack frame
for passing the pointer.
> Just a note about your concern about the base+size overflow. This is now
> not possible anymore, but i think it wasn't before. That's because the
> value "size" is somehow always less than the real memory size.
> Using 8 MB i got size values which told me that i had about 7.5 MB of
> main memory.
> So this size value is somehow a minimum of available memory.
I see. Do you think we could add a check right before we start using
these two values (base, size)? This would serve both as documentation
that "we need this to be true" and as precondition check "if it is not,
we do not continue further".
> The thing about PhysicalLastFrame : I implemented this because i thought
> that we would need some sort of possibility to check, whether the frame
> we are allocating or freeing is even "there".
> I am not completely sure whether this is really necessary,
I would leave this check in there, because in theory (as mentioned above,
the "size" argument is always less than the real memory size),
such frame can be constructed.
> as error
> checks could be made on a higher level. What do you think about that?
I think we can use error/invariant/{pre|post}condition checks
during the development. It speeds up the whole development process
a lot (at least from my experience). We can also make
these checks conditional, so that we can have a #define which
will decide whether the checks get compiled in or not.
Where to place these checks is a different question.
And I do not a direct answer for this. We will need checks
on all levels of the source code hierarchy.
I place checks as I see fit.
Example 1: If it is something that I only
setup at the beginning of the process and do not change
later and seemingly, there is no memory being overwritten
near where the value is stored, then I check it only once
after it is set. Also, I do not check everything. I check stuff
that is (sort of very) important, and in which it is likely that I
will do mistake,
and which is potentially hard to debug (find out the cause of the
runtime error).
Example 2: If it is something that can be changed and place A,
and then subsequently used in places B, C, D, then I place checks
at each place (B, C, D).
> Next your question about the 4 byte arithmetic: I am using 4 byte
> arithmetic because it is much faster than 1 byte arithmetic.
> A 32 bit processor can access 4 byte at once, so only checking 1 byte at
> once would leave 3/4 of the "power" unused.
> There is of course a "overhead" of doing the right arithmetic. But these
> are only bit shifts, which aren't very expensive :)
> I don't know much about embedded systems and whether they support 4 byte
> rates, so i don't know if this could be ported to such systems.
I get it now. About the embedded world. I do not know either.
I think the important question here is how much memory the embedded
system has to have at least, so that we decide that we would like to support it.
With 16-bit, or 24-bit systems, you do not get much memory.
And to me it seems that the embedded world is related to audio/video
processing, where I would expect the need for a lot of memory.
Tomas
|
|
From: Daniel O. <mus...@go...> - 2010-04-21 14:10:41
|
Hi :) Sorry for not being there doing something (visible) for such a long time. In fact, i had a complete reimplementation (but of course with nearly the same design) of the physical memory management, that is actually followed by a redesign of the virtual memory management and a small design change concerning multitasking. So there are a lot of changes that i didn't pushed yet, i hope i can put it online tomorrow or the day after tomorrow. I will try to take as much of your comments as possible and add my own to make the code easier to understand this time. The last code first wasn't coded with the idea of other people working on it, too ;) Its sometimes just hard to code and comment at the same time, but i hope i won't forget commenting anymore ^^ About the physical memory management: I removed the thing about the "base", from which on we were managing the memory. Now everything is managed. There is also a change in "using" the physical memory management. The lower level functions now return a value indicating whether they could do the job they were told to and the real "return" value is handled using pointer parameters. Just a note about your concern about the base+size overflow. This is now not possible anymore, but i think it wasn't before. That's because the value "size" is somehow always less than the real memory size. Using 8 MB i got size values which told me that i had about 7.5 MB of main memory. So this size value is somehow a minimum of available memory. The thing about PhysicalLastFrame : I implemented this because i thought that we would need some sort of possibility to check, whether the frame we are allocating or freeing is even "there". I am not completely sure whether this is really necessary, as error checks could be made on a higher level. What do you think about that? About PhysicalKernelEnd: Yes, that's right :) Next your question about the 4 byte arithmetic: I am using 4 byte arithmetic because it is much faster than 1 byte arithmetic. A 32 bit processor can access 4 byte at once, so only checking 1 byte at once would leave 3/4 of the "power" unused. There is of course a "overhead" of doing the right arithmetic. But these are only bit shifts, which aren't very expensive :) I don't know much about embedded systems and whether they support 4 byte rates, so i don't know if this could be ported to such systems. About the problem in _use_frames() : Yes, you are right, but we won't actually use this function anymore. This problem is also connected to the question where we should do the error checking. Daniel :) |
|
From: Tomas K. <tom...@gm...> - 2010-04-20 21:54:56
|
Hi, > In fact, it gets time to think about how to implement a "hardware > abstraction" concerning the devices. How much would we like to get inspired by Linux Kernel (not just in this area)? > I mean, we must, in some way, manage all devices which are present to > the system. > We (the kernel) will only be responsible for recognizing that there are > some devices and to provide some sort of infrastructure, that allows > only one process (-tree) to use the specific device. > What's done with the device is then up to the driver, which should not > be part of the kernel and thus exchangeable. What sort of user-space application interface would we like to provide? Unix/POSIX? We will need to design and implement system calls. The ABI is much related to this, since we probably do not want to write our own libc implementation. How much modular would we like the kernel to be? Presumably, we are doing a monolithic one. We can have loadable kernel modules, and then maybe user-space drivers (and filesystem implementations). I do not yet see any usefull modularity beyond this one. (well, unless we want to play with the ability to switch to different scheduler (or whole kernel) at runtime, which I do not know how much usefull it is, just random idea). I get it that our goal is to be able to run user-space applications (with support for dynamic libraries). This means choosing compiler and stuff. Then there is also networking (tcp/ip stack). Anyway, are we going to port some existing source code (2), or do everything from scratch ( yes! :-) ? Now, in the forums, there is an idea to write an embedded OS. This is interesting from, at least, two points: - the embedded hardware might not have MMU - organizing the kernel source code and managing a decent control over the build process I think I am interested in all of it, but primarily in loadable kernel modules and user-space application execution (since they both deal with ELF). Tomas |
|
From: Tomas K. <tom...@gm...> - 2010-04-20 21:33:10
|
Hi, I have a few controling question to check my understanding. Is PhysicalSuperMapStart always going to be equal to PhysicalManagedInfoStart? The PhysicalMapsStart is where we start to store the individual map contents? What is PhysicalLastFrame? I mean, I think I get how it is computed, but I question whether it is always going to be valid. For illustration, let's say Size will be (UINT32_MAX-Base+1). I think this can occur when the machine has 4 gigabytes of ram. Now, doing Base+Size will overflow, and the PhysicalLastFrame will be zero. Which I do not think is the desired outcome. What is it that I am not seeing here? Is PhysicalLastFrame actually the index of the first unmanaged frame? One that does not exist because it is beyond the actual available physical memory? About the line "PhysicalKernelEnd = PhysicalManagedInfoEnd;" PhysicalKernelEnd is, so far, where the kernel machine code and data structures end in the physical memory. Correct? Why are we doing address arithmetic in 4-byte quantities instead of 1-byte? I mean, we need >>5 because we are using UINT*, instead of, let's say char*, right? Is it because of the bitmaps? In _use_frames(), when (Start+Count >= PhysicalLastFrame) is true, shouldn't we panic, instead of just doing nothing? Btw, I have added a few more comments to the source file. It helps me a lot to quickly refresh my understanding of the source code. I did read enough from this file for now, and I am going to read other source files. Tomas |
|
From: Daniel O. <mus...@go...> - 2010-04-10 18:20:42
|
Am 10.04.2010 01:02, schrieb Tomas Klacko: > Hi, > > in physical_memory.c, I am not quite able to figure out > the meaning of PHYSICAL_FPM_DIVISOR value. > Also the lines like "map = Frame>> PHYSICAL_FPM_DIVISOR;" in free_frame(). > > Tomas Klacko > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Modularkernel-developer mailing list > Mod...@li... > https://lists.sourceforge.net/lists/listinfo/modularkernel-developer > Hi, yes, the name of the variable is not really good chosen. In the end, all those lines lead into something like that: map = frame / frames_per_map So the PHYSICAL_FPM_DIVISOR is the value to do the above operation with a bit shift. 2^PHYSICAL_FPM_DIVISOR = frames_per_map |
|
From: Daniel O. <mus...@go...> - 2010-04-10 18:17:44
|
> Well, only unless you do not want to manage some part of the physical memory, > which I get is the nonsense :) > Exactly. I am actually working on a reimplemantation of the multitasking code (which is not really working at the moment). Perhaps (if i don't get too confused^^) i will clean up this nonsense then, too. >> Its a relict from previous code, where I thought (which was wrong, as I >> know now) that PHYSICAL memory under 1 MB is needed for DMA, but in fact >> it is VIRTUAL, so there should be no problem managing all physical >> memory in the same way :) >> > Interesting, what is your previous code about? > > Where did you get this information that memory under 1 MB is not needed for DMA? > I recall that like the first 16 MB is used for DMA, because some devices > simply cannot address higher memory for their DMA transfers. > Does it have to do with IO Ports (and a lot of RAM, like plus 4GB)? > Eh, i can't remember from where exactly i got this wrong information (i think i just misunderstood it). The code is quite about the same thing, just leaving this 1 MB free to be managed by a later (and nether existent) "DMA memory manager" How DMA will have to be implemented is another thing, about which i don't think yet. But i have some good ressources ( http://www.osdever.net/ ) In fact, it gets time to think about how to implement a "hardware abstraction" concerning the devices. I mean, we must, in some way, manage all devices which are present to the system. We (the kernel) will only be responsible for recognizing that there are some devices and to provide some sort of infrastructure, that allows only one process (-tree) to use the specific device. What's done with the device is then up to the driver, which should not be part of the kernel and thus exchangeable. :) |
|
From: Tomas K. <tom...@gm...> - 2010-04-09 23:02:33
|
Hi, in physical_memory.c, I am not quite able to figure out the meaning of PHYSICAL_FPM_DIVISOR value. Also the lines like "map = Frame >> PHYSICAL_FPM_DIVISOR;" in free_frame(). Tomas Klacko |
|
From: Tomas K. <tom...@gm...> - 2010-04-09 22:46:44
|
On Fri, Apr 9, 2010 at 11:24 PM, Daniel Oertwig <mus...@go...> wrote: > Am 09.04.2010 23:17, schrieb Tomas Klacko: >> Hi, >> >> in physical_memory.c, are PhysicalKernelStart and PhysicalManagedStart >> always going to have the same value? (0x100000) >> > They can, as it is at the moment, but it is not necessary. > The two variables describe two completely different things. > The one is the start of the kernel, which is somehow a constant thing, > because loading to 1 MB mark is the "standard" for kernels. > > The PhysicalManagedStart is the value describing at which point in > memory we start tracing whether it is used or not. So at the moment, we > cannot say whether memory before 1 MB is used or not. > > I am thinking to change that, because it is somehow nonsense to not > manage a part of the physical memory. Well, only unless you do not want to manage some part of the physical memory, which I get is the nonsense :) > Its a relict from previous code, where I thought (which was wrong, as I > know now) that PHYSICAL memory under 1 MB is needed for DMA, but in fact > it is VIRTUAL, so there should be no problem managing all physical > memory in the same way :) Interesting, what is your previous code about? Where did you get this information that memory under 1 MB is not needed for DMA? I recall that like the first 16 MB is used for DMA, because some devices simply cannot address higher memory for their DMA transfers. Does it have to do with IO Ports (and a lot of RAM, like plus 4GB)? Tomas |
|
From: Daniel O. <mus...@go...> - 2010-04-09 21:24:40
|
Am 09.04.2010 23:17, schrieb Tomas Klacko: > Hi, > > in physical_memory.c, are PhysicalKernelStart and PhysicalManagedStart > always going to have the same value? (0x100000) > > Tomas Klacko > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Modularkernel-developer mailing list > Mod...@li... > https://lists.sourceforge.net/lists/listinfo/modularkernel-developer > They can, as it is at the moment, but it is not necessary. The two variables describe two completely different things. The one is the start of the kernel, which is somehow a constant thing, because loading to 1 MB mark is the "standard" for kernels. The PhysicalManagedStart is the value describing at which point in memory we start tracing whether it is used or not. So at the moment, we cannot say whether memory before 1 MB is used or not. I am thinking to change that, because it is somehow nonsense to not manage a part of the physical memory. Its a relict from previous code, where I thought (which was wrong, as I know now) that PHYSICAL memory under 1 MB is needed for DMA, but in fact it is VIRTUAL, so there should be no problem managing all physical memory in the same way :) |
|
From: Tomas K. <tom...@gm...> - 2010-04-09 21:17:27
|
Hi, in physical_memory.c, are PhysicalKernelStart and PhysicalManagedStart always going to have the same value? (0x100000) Tomas Klacko |
|
From: Tomas K. <tom...@gm...> - 2010-04-09 20:46:06
|
On Fri, Apr 9, 2010 at 10:28 PM, Daniel Oertwig <mus...@go...> wrote: > Am 09.04.2010 22:23, schrieb Tomas Klacko: >> Hi, >> >> In Kernel/LowLevel/makefile, what is the purpose of "-m elf_i386" in >> the ldflags? >> I find the compilation to produce a functional mk.img even without it. >> > > Then I assume that you are running on a 32 bit machine. > When i would try to compile here on my laptop (64 bit) without that, it > would result in many errors of incorrect file formats. Yes actually, I am running on a 32-bit machine. I got to it by trying to compile it under cygwin. (It did not work, I am letting it be). Tomas Klacko |
|
From: Daniel O. <mus...@go...> - 2010-04-09 20:28:26
|
Am 09.04.2010 22:23, schrieb Tomas Klacko: > Hi, > > In Kernel/LowLevel/makefile, what is the purpose of "-m elf_i386" in > the ldflags? > I find the compilation to produce a functional mk.img even without it. > > Tomas Klacko > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Modularkernel-developer mailing list > Mod...@li... > https://lists.sourceforge.net/lists/listinfo/modularkernel-developer > Then I assume that you are running on a 32 bit machine. When i would try to compile here on my laptop (64 bit) without that, it would result in many errors of incorrect file formats. |
|
From: Tomas K. <tom...@gm...> - 2010-04-09 20:23:26
|
Hi, In Kernel/LowLevel/makefile, what is the purpose of "-m elf_i386" in the ldflags? I find the compilation to produce a functional mk.img even without it. Tomas Klacko |
|
From: Daniel O. <mus...@go...> - 2010-04-07 08:49:25
|
Hi, yes, that's exactly whats done there. The linked section is the section which contains code (and data in the *_data section) that must be available to every task in the later operating system. So i will link that section into every virtual address space. Perhaps it is important to add that this "linked section" is somehow different from other operating systems such as linux (or windows), because only a very small part of the kernel code is linked and not (nearly?) all of it. |
|
From: Daniel O. <mus...@go...> - 2010-04-07 08:42:06
|
Hi :) No, this address is behind the (physical location) of the .linked and .linked_data section in memory. Its the place where i write (or wrote, i think this should be removed as it makes no sense to copy the information from the variables to another place in memory) some information about the physical memory setup (e.g. the map count etc) I will remove that part... |
|
From: Daniel O. <mus...@go...> - 2010-04-07 08:36:19
|
Hi :) (sorry that it took me so long, i thought i was automatically
subscribed)
PhysicalMapSize is the size a map needs for itself in memory to contain
the information.
In the setup_ function, PhysicalMapSize gets the value
PhysicalMapSize = PHYSICAL_FPM >> 5;
PHYSICAL_FPM means "Physical Frames per Map" and indicates, how many
frames are managed by one single map. Because we "manage" every frame
with a single bit we need PHYSICAL_FPM bits to store the information.
But because a size value in bits is not very useful, i transform it into
a 4byte size value by dividing through 32 (this is what ">> 5" does)
PhysicalSuperMapSize is the same but for the supermap, which manages the
maps.
Every map is "displayed" as a single bit, so we need PhysicalMapsCount
bits, which i divide through 32 again. This time i had to add +31 before
dividing to round up. (Otherwise, if you had 33 maps, you would only use
4 bytes, which is not enough, you need with this implementation 8 bytes)
The writes are not very important at the moment. I even think about
deleting them, because the purpose of these writes (storing the physical
management information at a secure place) can be achieved by the linker
(by not putting the variables into startup_data)
But actually the information is written directly behind the end of the
kernel (as it has been loaded by grub)
Its important to notice again, that here are only physical addresses.
So the addresses from .linked and .linked_data are virtual addresses and
not (at this point) usable.
These two sections are compiled/linked as if the were loaded to the
higher memory 0xFFC00000 but in fact they are right behind the other
code (concerning their physical address)
|
|
From: Tomas K. <tom...@gm...> - 2010-04-04 21:32:27
|
Hi, I think I need a commentary on the setup_physicalmemorymanagement() function, maybe also about the intended physical memory management. What is PhysicalMapSize? What is PhysicalSuperMapSize? Looking at the writes (*wr), where in the memory are we actually writing to? Is wr related to .linked and .linked_data sections? Tomas Klacko |
|
From: Tomas K. <tom...@gm...> - 2010-04-04 20:44:47
|
Hi, in setup_physicalmemorymanagement(), the line PhysicalManagedInfoStart = (PhysicalKernelEnd+0x1000)&(~0xFFF); Is PhysicalManagedInfoStart going to contain the start of the loaded .linked section in memory? Tomas |
|
From: Tomas K. <tom...@gm...> - 2010-04-04 20:07:32
|
Hi,
I am trying to understand the following part of the linker script:
__RealStartOfLinkedMemory = ALIGN(4096);
. = 0xFFC00000;
__StartOfLinkedMemory = .;
.linked : AT (__RealStartOfLinkedMemory)
{
*(.linked)
*(.linked_data)
}
__EndOfLinkedMemory = .;
____KERNELEND = (__EndOfLinkedMemory -
__StartOfLinkedMemory)+(__RealStartOfLinkedMemory);
This is what I think it does:
- the .linked section is copied to frames immediatelly following the
.bss section.
- the .linked section is available starting at virtual address 0xFFC00000
Why is it done this way?
Is the .linked section going to contain kernel data to be shared by
every user-space process?
Tomas
|
