sharpos-developers

Re: [SharpOS Developers] MOSA / AOT

[Bruce M. <ill...@gm...>]

Okay, before anyone starts marking their scents on the ground - (I'm
stopping you so I can get mine in there first) - my point isn't what looks
pretty or doesn't.

My point is, MOSA needs to agree on how to handle (and how much to handle)
differing paradigms. Grover, I'm glad you think you agree with me - but you
didn't actually address the blunt of my point. The point being, the MOSA
forums are almost as much of a ghost town as SharpOS is. It may be a
compiler you are writing to contribute to MOSA, but MOSA as a whole (which
in and of itself, is rather undefined), needs to have an infrastructure on
agreeing on standards by which opinions or guidance can be generated in
regards to *any* compiler, including this new one being written.

Calling it a MOSA compiler may not be accurate if it has to be branched four
ways from Sunday in order to work with each project. And maybe - maybe
*thats* exactly how "MOSA" wants to do it. But there just isn't a clear
enough definition of *what MOSA is* right now, for any of us to know how to
feel about each other's compilers. (After all, we are trying to eliminate
the "mine is bigger than yours" crap so that we can get along.)

MOSA needs a domain. MOSA needs a mailing list. MOSA needs *people*. Then, I
think, MOSA can sit down to create standards. There is so much to talk
about, that I think we need some clear, defined space to do it in, so we can
track what is going on.

When you guys first started talking about MOSA, I didn't think about code at
all. But code is an understandable extension of agreed-upon specifications.
Specifications that we need to take time, space, and conversation to figure
out. The MOSA AOT is fascinating and exciting, but conversations about it
are moot for now.

Re: [SharpOS Developers] MOSA / AOT

[Chad Z. H. a. K. <Ch...@Ho...>]

> tells the compiler here's special code to emit. I've read Chad's entry
> about
> X#, but sadly think this appears nice at first, but causes a lot of
> headaches afterwards.

Actually it has worked very well for us. It's a 1:1 mapping, its not a
compiler more of a Mnemonic system. There is no way I'd go back to what we
had before or even Intel Mnemonics, AT&T or otherwise.

Re: [SharpOS Developers] MOSA / AOT

[grover <sh...@mi...>]

Hi Bruce,

Actually everything you said is my personal belief. And everything you wrote
is already in the design of the compiler. Native operations are handled by
marking properties, fields or methods with an Intrinsic attribute, that
tells the compiler here's special code to emit. I've read Chad's entry about
X#, but sadly think this appears nice at first, but causes a lot of
headaches afterwards.

You're right with the standardization needed. And I believe that this will
get going. I will personally try to drive it, once I've got more time to do
that.

Mike

P.S.: Here's an example:

    /// <summary>
    /// Provides stub methods for selected x86 native assembly instructions.
    /// </summary>
    public static class Native
    {
 	  // ...

        /// <summary>
        /// Wraps the x86 ldit instruction to load the interrupt descriptor
table.
        /// </summary>
        /// <param name="idt">A pointer to the interrupt descriptor
table.</param>
        [Intrinsic(typeof(Architecture), typeof(LditInstruction))]
        public static void Ldit(IntPtr idt) { ThrowPlatformNotSupported(); }

	  // ...
    }


Using it is as simple as Native.Ldit(ldit) - the compiler knows not to emit
a method call, if he's running on x86 architectures. The other ones will
fail with a PlatformNotSupportedException. This is live code from MOSA.


> -----Ursprüngliche Nachricht-----
> Von: sha...@li... 
> [mailto:sha...@li...] Im 
> Auftrag von Bruce Markham
> Gesendet: Dienstag, 5. August 2008 16:22
> An: sha...@li...
> Betreff: Re: [SharpOS Developers] MOSA / AOT
> 
> Well, like I said several e-mails back in this thread - MOSA 
> needs to start talking about standards. Adding another 
> compiler to the list, (whether call it a MOSA compiler or 
> not, and whether we think it will be less confusing to 
> maintain or not), doesn't solve any lasting problems.
> 
> I was working on a multi-blog aggregator this morning (with 
> hopes to pitch it in MOSA's direction eventually), using LINQ 
> and Microsoft's ASP.NET RssToolkit, and found myself reading 
> Cosmos' (Chad's) entry about "X#" 
> <http://www.gocosmos.org/Blog/20080428.en.aspx> . Its 
> definitely a more natural feeling way to express assembly 
> instructions in C# code - and I think MOSA needs to address 
> these types of issues.
> 
> What MOSA needs, I think to start with, is a standard 
> reference library of stubs to support something like "X#", 
> (or for the sake of example, stubs that look like SharpOS' 
> AOT X86 stubs) - and then begin working standards around it. 
> Standards that say "If you want an official MOSA compiler to 
> emit X86 for you, this is what you have to use." Granted, 
> thats specific to the C# subset of MOSA - but it gets us 
> moving into a spirit of standardization.
> 
> All in all, I don't care too horribly much about how the 
> stubs work. The point is nearly moot considering no matter 
> how you do it, you are going to have to do analysis of the 
> code graph to figure out what literal x86 (or other machine 
> language) the user wants you to emit. My point is that we 
> need a starting point. From there, we can talk about how we 
> want to handle the placement of the runtime ("in the kernel, 
> or in the compiler" seems to be a big issue) - and we can 
> also talk about standards for how the VES works. (For 
> example, how we want to "plug" parts of corelib.)
> 
> Notice, these are all issues that each of our projects have 
> addressed - and in rather differing ways. But if we can't 
> agree on standards, the idea of a "MOSA" compiler for them is 
> going to be useless. (I mean, the MOSA compiler could use 
> some sort of pluggable architecture so it can handle 
> different paradigms, but that would be impossible to 
> maintain, and it wouldn't be very performant.)
> 
> 

Re: [SharpOS Developers] MOSA / AOT

[Bruce M. <ill...@gm...>]

Well, like I said several e-mails back in this thread - MOSA needs to start
talking about standards. Adding another compiler to the list, (whether call
it a MOSA compiler or not, and whether we think it will be less confusing to
maintain or not), doesn't solve any lasting problems.

I was working on a multi-blog aggregator this morning (with hopes to pitch
it in MOSA's direction eventually), using LINQ and Microsoft's
ASP.NETRssToolkit, and found myself reading Cosmos' (Chad's) entry
about "X#" <http://www.gocosmos.org/Blog/20080428.en.aspx>. Its definitely a
more natural feeling way to express assembly instructions in C# code - and I
think MOSA needs to address these types of issues.

What MOSA needs, I think to start with, is a standard reference library of
stubs to support something like "X#", (or for the sake of example, stubs
that look like SharpOS' AOT X86 stubs) - and then begin working standards
around it. Standards that say "If you want an official MOSA compiler to emit
X86 for you, this is what you have to use." Granted, thats specific to the
C# subset of MOSA - but it gets us moving into a spirit of standardization.

All in all, I don't care too horribly much about how the stubs work. The
point is nearly moot considering no matter how you do it, you are going to
have to do analysis of the code graph to figure out what literal x86 (or
other machine language) the user wants you to emit. My point is that we need
a starting point. From there, we can talk about how we want to handle the
placement of the runtime ("in the kernel, or in the compiler" seems to be a
big issue) - and we can also talk about standards for how the VES works.
(For example, how we want to "plug" parts of corelib.)

Notice, these are all issues that each of our projects have addressed - and
in rather differing ways. But if we can't agree on standards, the idea of a
"MOSA" compiler for them is going to be useless. (I mean, the MOSA compiler
could use some sort of pluggable architecture so it can handle different
paradigms, but that would be impossible to maintain, and it wouldn't be very
performant.)

Re: [SharpOS Developers] True Type Fonts

[Jonathan D. <jon...@gm...>]

I said this on the Cosmos ML, I suppose I should have said it
everywhere. The point of this TTF stuff is that I will co-incide with
completion of System.Drawing namespaces (and the start of userland as
a whole). It will rely heavily on Regions and Paths and what not, we
will need those types of things first. So it is a looong way off.

Honestly, it is really on the back-burner at the moment as I have a
quite a bit of work at work, and working with TTF can drive a person
crazy. If anyone has a Hazmat suit for me maybe I could dedicate more
time to it :).

If someone wants to hack System.Drawing support on the current stuff
feel free, I will try and get this thing working faster if that's the
case.

Has anyone thought about inlining native libs (via P/Invoke) so that
we can use FreeType (which I am now considering just porting) and
Cairo? The Google Javascript OS does that and it looks like it works
pretty well... I know there is a certain philosophy with the whole C#
only idea, but this would give us a boost so that we don't have to
worry about mundane/arcane things (like TTF): we could port the libs
later on. Mike, any chance of this stuff?

On Fri, Aug 1, 2008 at 9:04 PM, Sander van Rossen
<s.v...@sy...> wrote:
> Yeah somebody did some basic graphics work, but it was a bit too early
> for it to be used.. especially since we had nothing like a driver
> architecture back then, which at least now we have a good part already
> in place (of the architecture)
>
> On Fri, Aug 1, 2008 at 4:55 PM, Phil Garcia <ph...@th...> wrote:
>> Very cool. I think someone already did some work on a VGA driver, but they
>> didn't been committed to the trunk. I'll look around for it and get it
>> implemented. Let us know why type of graphic API methods you will need since
>> obviously we won't have .NET graphic classes available for a while.
>> -Phil
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> SharpOS-Developers mailing list
> Sha...@li...
> https://lists.sourceforge.net/lists/listinfo/sharpos-developers
>



-- 
Jonathan

Re: [SharpOS Developers] MOSA / AOT

[Jonathan D. <jon...@gm...>]

I can vouch that IL2CPU is a good compiler. Only thing is that a lot
of your code doesn't 'fit' into it. I don't think you would be able to
compile your stuff without a lot of work (shy of a total re-write).

On Tue, Aug 5, 2008 at 2:11 PM, Chad Z. Hower aka Kudzu <Ch...@ho...> wrote:
>> Which could be used, but has its fair share of issues too. I'm certain
>> that
>
> I think its much farther along than you think....
>
>> design AFAIK, as there's no support for Attributes, Reflection, ...
>
> They haven't been high on the list because we haven't needed them yet.
>
>
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> SharpOS-Developers mailing list
> Sha...@li...
> https://lists.sourceforge.net/lists/listinfo/sharpos-developers
>



-- 
Jonathan

Re: [SharpOS Developers] MOSA / AOT

[grover <sh...@mi...>]

Most likely it is. I have not invested much time in it.

Mike 

> -----Ursprüngliche Nachricht-----
> Von: sha...@li... 
> [mailto:sha...@li...] Im 
> Auftrag von Chad Z. Hower aka Kudzu
> Gesendet: Dienstag, 5. August 2008 14:11
> An: sha...@li...
> Betreff: Re: [SharpOS Developers] MOSA / AOT
> 
> > Which could be used, but has its fair share of issues too. 
> I'm certain 
> > that
> 
> I think its much farther along than you think....
> 
> > design AFAIK, as there's no support for Attributes, Reflection, ...
> 
> They haven't been high on the list because we haven't needed them yet.
> 
> 
> 
> 
> --------------------------------------------------------------
> -----------
> This SF.Net email is sponsored by the Moblin Your Move 
> Developer's challenge Build the coolest Linux based 
> applications with Moblin SDK & win great prizes Grand prize 
> is a trip for two to an Open Source event anywhere in the 
> world http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> SharpOS-Developers mailing list
> Sha...@li...
> https://lists.sourceforge.net/lists/listinfo/sharpos-developers
> 

Re: [SharpOS Developers] MOSA / AOT

[Chad Z. H. a. K. <Ch...@Ho...>]

> Which could be used, but has its fair share of issues too. I'm certain
> that

I think its much farther along than you think....

> design AFAIK, as there's no support for Attributes, Reflection, ...

They haven't been high on the list because we haven't needed them yet.

Re: [SharpOS Developers] MOSA / AOT

[Chad Z. H. a. K. <Ch...@Ho...>]

> Don't the other teams, or at least the Cosmos, have compilers as well?

Yes, Cosmos has a full compiler.

> Although they're not designed as well in the basis as the AOT, maybe
> we could 'borrow' one of these compilers for the time being until the
> MOSA compiler is finished?

If you can live with our BSD license. :)

Re: [SharpOS Developers] Additional stuff regarding memory

[Sander v. R. <s.v...@sy...>]

On Tue, Aug 5, 2008 at 12:38 PM, Zachary Gorden
<dra...@gm...> wrote:
> In response to Sander:
>
> A valid point, regarding the reserved space.  However, I was thinking that
> the drivers were also restricted to the upper 2GB with the kernel.
> Actually, this is the point where you guys tell me how you want to approach
> this separation of drivers thing.  After all, sticking all the drivers into
> their own address space is not the way to go in my opinion.  Minix does this
> and they suffer a 20% performance penalty in the process.  I don't fully
> understand how Singularity does it, so, yeah.  After all, the point of
> Singularity's design was to avoid that penalty.

Right, i was still thinking about a single adress space like solution
like singularity..


> I talked to a friend of mine who has a better idea of how GCs work.
> Apparently GCs do not care whether their heaps are paged out or whether
> there's a lack of physical pages.  It's entirely possible for it to try to
> sweep heaps that are completely paged out, causing performance issues and
> etc.

Right, and i was talking about those performance issues, sorry for not
being clear.


> I expect each appdomain to have its own.
> But since the 2GB range (or whatever it ends up being) for the application's
> personal usage is separate for each appdomain anyways, then what's the
> problem with shifting all responsibility to the GC and the managed heap?

define "all responsibility" :)


> Well, originally I was thinking about an all or nothing approach.  But to be
> perfectly honest, appdomains sharing address spaces have almost nothing to
> do with the work I need to do in the lower levels.  I'm just providing the
> ability to create address spaces for each appdomain.  The MM provides the
> services and people implementing the higher level stuff can worry about
> keeping things straight.  Correct me if there is something I need to take
> into consideration though.

If multiple appdomains share the same adress space then they would
also share the 2GB limit...
I *suppose* technically it would be possible to start putting those
appdomains into multiple adress-spaces the moment we detect that that
2 GB isn't enough.
Hmm... interestingly, if we could get *that* to work, then we could
then easily extended that to -all- appdomains.
Meaning that as long as all appdomains stay within the 4GB footprint,
we'd only use one address-space...
Each appdomain has it's data already seperated from other appdomains,
so that would definitely help in this scenario..

Also, If the entire 'kernel' is always in memory, then we would also
never have to switch to another adress space when switching back and
from the kernel.

Of course all of this assumes we won't have any kernel rings.

[SharpOS Developers] Short self-introduction

[Simon W. <sim...@ya...>]

Hi everyone,

since I'm new I'll introduce myself a bit. My name is rootnode (Simon 
Wollwage)
and I'm helping grover with the MOSA compiler.
I'm a B.Sc. student of computer science and mathematics in my 5th 
semester at RWTH Aachen University, Germany. So far about my background.

Thath's all for now to keep my introduction short .

I hope I can be of help within this project.
--------------------------------------
For All Sports Lovers! SPORTS OHEN PROJECT 2008
http://pr.mail.yahoo.co.jp/yells/

Re: [SharpOS Developers] Additional stuff regarding memory

[Sander v. R. <s.v...@sy...>]

That's exactly what i was talking about before, when talking about
keeping "book-keeping information" in memory when swapping pages to
disk.
That way you know if blocks of memory (which are currently on disk)
need to be collected or not.
This way you wouldn't even have to touch the memory blocks... if
they're collected, then the page is freed in the page file, if it's
not collected.. nothing happens.
Only problem is that the book-keeping information (which specifies
which object is connected to which, and what block of memory on disk
it uses) costs memory itself.. so it should only be used on reasonably
large blocks.
Trashing to and from disk would *really* decrease the performance of
virtual memory.. and make collections a nightmare performance wise...

On Tue, Aug 5, 2008 at 1:07 PM, grover <sh...@mi...> wrote:
> Zach,
>
> You're right. Current GCs tend to trash the disk if they try to
> collect a heap that's mostly paged out. I was hoping we can
> prevent this kind of behavior by applying some smarts there. Actually
> I think the chances are actually good there as pages could be removed
> from memory on a LRU scheme - we could try to collect these specifically
> to throw them away before paging them out.
>
> Mike
>
>> -----Ursprüngliche Nachricht-----
>> Von: sha...@li...
>> [mailto:sha...@li...] Im
>> Auftrag von Zachary Gorden
>> Gesendet: Dienstag, 5. August 2008 12:39
>> An: sha...@li...
>> Betreff: Re: [SharpOS Developers] Additional stuff regarding memory
>>
>> In response to Sander:
>>
>> A valid point, regarding the reserved space.  However, I was
>> thinking that the drivers were also restricted to the upper
>> 2GB with the kernel.  Actually, this is the point where you
>> guys tell me how you want to approach this separation of
>> drivers thing.  After all, sticking all the drivers into
>> their own address space is not the way to go in my opinion.
>> Minix does this and they suffer a 20% performance penalty in
>> the process.  I don't fully understand how Singularity does
>> it, so, yeah.  After all, the point of Singularity's design
>> was to avoid that penalty.
>>
>> I talked to a friend of mine who has a better idea of how GCs
>> work.  Apparently GCs do not care whether their heaps are
>> paged out or whether there's a lack of physical pages.  It's
>> entirely possible for it to try to sweep heaps that are
>> completely paged out, causing performance issues and etc.
>> Now granted the OS can provide ways of telling whether a page
>> is in memory or not, there are other complications.  Anyways,
>> I don't expect some kind of global GC or something.  I expect
>> each appdomain to have its own.  But since the 2GB range (or
>> whatever it ends up being) for the application's personal
>> usage is separate for each appdomain anyways, then what's the
>> problem with shifting all responsibility to the GC and the
>> managed heap?  Again, that's all stuff that needs to be thought about.
>>
>> Forcing GC across the board really would be an act of
>> desperation, to be honest.  But we'll see.
>>
>> In response to Grover:
>>
>> Well, originally I was thinking about an all or nothing
>> approach.  But to be perfectly honest, appdomains sharing
>> address spaces have almost nothing to do with the work I need
>> to do in the lower levels.  I'm just providing the ability to
>> create address spaces for each appdomain.  The MM provides
>> the services and people implementing the higher level stuff
>> can worry about keeping things straight.  Correct me if there
>> is something I need to take into consideration though.
>>
>> The stuff you suggested for basic functionality the MM
>> provides through its interface will most likely be present
>> for use by whatever is managing the heap.
>>
>> In Windows, the bootloader actually handles all of the
>> mapping of the kernel into virtual memory using page tables.
>> It's one reason grub can't natively boot Windows (the other
>> being the registry).  Freeloader, the ReactOS bootloader,
>> actually would be capable of doing this too, but we'd need to
>> write the code for it to understand the SharpOS kernel.
>>
>>
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> SharpOS-Developers mailing list
> Sha...@li...
> https://lists.sourceforge.net/lists/listinfo/sharpos-developers
>

[SharpOS Developers] Hello guys

[Iraklis K. <ika...@gm...>]

I am new here, i am 16 years old. i know c, vb6, vb.net and now i am learnig
c#.
i dont know thinks about the kernel but i want to help.

so what do u suggest me?? from where to start?? thanks :)

PS
sorry for my english i speak them in the greek way :)

Iraklis Karagkiozoglou

Re: [SharpOS Developers] Additional stuff regarding memory

[grover <sh...@mi...>]

Zach,

You're right. Current GCs tend to trash the disk if they try to
collect a heap that's mostly paged out. I was hoping we can
prevent this kind of behavior by applying some smarts there. Actually
I think the chances are actually good there as pages could be removed
from memory on a LRU scheme - we could try to collect these specifically
to throw them away before paging them out.

Mike 

> -----Ursprüngliche Nachricht-----
> Von: sha...@li... 
> [mailto:sha...@li...] Im 
> Auftrag von Zachary Gorden
> Gesendet: Dienstag, 5. August 2008 12:39
> An: sha...@li...
> Betreff: Re: [SharpOS Developers] Additional stuff regarding memory
> 
> In response to Sander:
> 
> A valid point, regarding the reserved space.  However, I was 
> thinking that the drivers were also restricted to the upper 
> 2GB with the kernel.  Actually, this is the point where you 
> guys tell me how you want to approach this separation of 
> drivers thing.  After all, sticking all the drivers into 
> their own address space is not the way to go in my opinion.  
> Minix does this and they suffer a 20% performance penalty in 
> the process.  I don't fully understand how Singularity does 
> it, so, yeah.  After all, the point of Singularity's design 
> was to avoid that penalty.
> 
> I talked to a friend of mine who has a better idea of how GCs 
> work.  Apparently GCs do not care whether their heaps are 
> paged out or whether there's a lack of physical pages.  It's 
> entirely possible for it to try to sweep heaps that are 
> completely paged out, causing performance issues and etc.  
> Now granted the OS can provide ways of telling whether a page 
> is in memory or not, there are other complications.  Anyways, 
> I don't expect some kind of global GC or something.  I expect 
> each appdomain to have its own.  But since the 2GB range (or 
> whatever it ends up being) for the application's personal 
> usage is separate for each appdomain anyways, then what's the 
> problem with shifting all responsibility to the GC and the 
> managed heap?  Again, that's all stuff that needs to be thought about.
> 
> Forcing GC across the board really would be an act of 
> desperation, to be honest.  But we'll see.
> 
> In response to Grover:
> 
> Well, originally I was thinking about an all or nothing 
> approach.  But to be perfectly honest, appdomains sharing 
> address spaces have almost nothing to do with the work I need 
> to do in the lower levels.  I'm just providing the ability to 
> create address spaces for each appdomain.  The MM provides 
> the services and people implementing the higher level stuff 
> can worry about keeping things straight.  Correct me if there 
> is something I need to take into consideration though.
> 
> The stuff you suggested for basic functionality the MM 
> provides through its interface will most likely be present 
> for use by whatever is managing the heap.
> 
> In Windows, the bootloader actually handles all of the 
> mapping of the kernel into virtual memory using page tables.  
> It's one reason grub can't natively boot Windows (the other 
> being the registry).  Freeloader, the ReactOS bootloader, 
> actually would be capable of doing this too, but we'd need to 
> write the code for it to understand the SharpOS kernel.
> 
> 

Re: [SharpOS Developers] Additional stuff regarding memory

[Zachary G. <dra...@gm...>]

In response to Sander:

A valid point, regarding the reserved space.  However, I was thinking that
the drivers were also restricted to the upper 2GB with the kernel.
Actually, this is the point where you guys tell me how you want to approach
this separation of drivers thing.  After all, sticking all the drivers into
their own address space is not the way to go in my opinion.  Minix does this
and they suffer a 20% performance penalty in the process.  I don't fully
understand how Singularity does it, so, yeah.  After all, the point of
Singularity's design was to avoid that penalty.

I talked to a friend of mine who has a better idea of how GCs work.
Apparently GCs do not care whether their heaps are paged out or whether
there's a lack of physical pages.  It's entirely possible for it to try to
sweep heaps that are completely paged out, causing performance issues and
etc.  Now granted the OS can provide ways of telling whether a page is in
memory or not, there are other complications.  Anyways, I don't expect some
kind of global GC or something.  I expect each appdomain to have its own.
But since the 2GB range (or whatever it ends up being) for the application's
personal usage is separate for each appdomain anyways, then what's the
problem with shifting all responsibility to the GC and the managed heap?
Again, that's all stuff that needs to be thought about.

Forcing GC across the board really would be an act of desperation, to be
honest.  But we'll see.

In response to Grover:

Well, originally I was thinking about an all or nothing approach.  But to be
perfectly honest, appdomains sharing address spaces have almost nothing to
do with the work I need to do in the lower levels.  I'm just providing the
ability to create address spaces for each appdomain.  The MM provides the
services and people implementing the higher level stuff can worry about
keeping things straight.  Correct me if there is something I need to take
into consideration though.

The stuff you suggested for basic functionality the MM provides through its
interface will most likely be present for use by whatever is managing the
heap.

In Windows, the bootloader actually handles all of the mapping of the kernel
into virtual memory using page tables.  It's one reason grub can't natively
boot Windows (the other being the registry).  Freeloader, the ReactOS
bootloader, actually would be capable of doing this too, but we'd need to
write the code for it to understand the SharpOS kernel.

Re: [SharpOS Developers] Additional stuff regarding memory

[grover <sh...@mi...>]

Hi Zach,

Some answers inline below.

> -----Ursprüngliche Nachricht-----
> Von: sha...@li... 
> [mailto:sha...@li...] Im 
> Auftrag von Zachary Gorden
> Gesendet: Dienstag, 5. August 2008 09:39
> An: sha...@li...
> Betreff: [SharpOS Developers] Additional stuff regarding memory
> 
> Decided to get off my ass and provide some more detailed 
> thoughts about what's needed for the memory system.  So far, 
> this is more a matter of intentions than set plans.  It also 
> raises some points I don't fully understand and would 
> appreciate insight on.
> 
> SharpOS Memory Architecture Design
> 
> This is a basic outline of the low level memory management 
> system of SharpOS.  This does not include the higher level 
> garbage collection and managed memory schemes, but the basic 
> services that they will depend on.
> 
> Virtual Address Space Organization
> 
> The virtual address space in SharpOS will follow the 
> traditional 2GB split for kernel and user mode in 32bit mode, 
> however this is a mere formality, as there does not exist a 
> user mode.  It is more for organization purposes so that the 
> kernel has sufficient reserved space for its own data 
> structures.  The only thing prevent applications and other 
> unauthorized code from accessing the kernel portion will be 
> the managed nature of such code.
> 
> Using a separate page table for each appdomain will mean we 
> must change the value in CR3, so we will need to do a memory 
> context switch.  This results in increased overhead and may 
> be avoidable in 64bit mode, but will require some 
> experimentation in the future.  The page table entries 
> describing the kernel range will be shared by all page 
> tables.  Intel processors have support for this so it 
> shouldn't be problematic.

I think we can live with this, however I would not assume that
an AppDomain is a process. This may be, but its not in a traditional
.NET sense. We'd have to do lots of inter-process marshalling in a
lot of common situations if we'd move AppDomains to their own
address space. So here comes the requirement that AppDomains must
be able to share address spaces.
 
> The kernel range will be the traditional upper 2GB of the 4GB 
> virtual space.  It will be a series of preallocated paged and 
> unpaged pools of memory, specifically set aside for usage by 
> the kernel.  These preallocated data structures will need 
> some tweaking in order to ensure there is enough for each.  
> For allocations smaller than a page in size, there will be a 
> series of stacks of varying sizes.  Allocations from them 
> will be done through lockfree atomic instructions to ensure 
> better throughput.  Whether a general purpose pool will be 
> needed or even possible I'll have to look into, as 
> traditionally you cannot allocate to the heap like you would 
> in user mode.
> 
> The lower 2GB reserved for the application is where things 
> get interesting.  Because everything the application does is 
> managed, we might be able to shift almost all responsibility 
> to the GC and company.  It requires some thought, as I don't 
> know what is intended for management of application memory.
> 
> Okay, I'm lying.  What I really want to know is when people 
> keep saying "GC kicks in when memory is low," what the blazes 
> do they mean?  Are they talking about the GC being aware of 
> physical pages actually running low?  Or just this heap that 
> the GC is supposedly managing, which might as well be a range 
> of virtual addresses that might all be paged out for all we 
> know?  Until that point gets cleared up, I'm stuck as to what 
> I need to provide for the upper level services.

Like I wrote previously there's two parts to this: The actual 
allocator (a heap of some sort) and the GC itself. Heaps need
to grow and when that happens a first-gen collection is performed,
so that the crap pile of all temporary objects is freed on the 
heap. Then the allocation is tried again - if it fails this time,
the heap must grow and the heap will request new pages. So the GC
does not have much to do with you, its more the heap which needs
to be able to allocate (batches) of pages and free them again.
I see this similar to the VirtualAlloc/VirtualFree functions in
Win32, an interface of that style should be sufficient.

I would think of:
- An alloc
- A free
- A low watermark notification
- An info method to get statistics and page granularity
- Ability to set page protection attributes (read, write, execute,
no-execute)
- Page status: Reserved, Committed, Freed, (Paged, ...)
 
> The tricky part we now have to deal with is the initial 
> mapping of the kernel into virtual address space and putting 
> together a page table for it.  I still don't fully know how 
> I'm going to pull it off, but it will definitely require some 
> refactoring of the code.  I also need to pick a point early 
> enough in the loading where I won't be overwhelmed with 
> repointing all the pointers.

Does this have to do with grub? Do we need a different (managed?)
boot loader?

> Physical Memory
> 
> This is basically the paging/swapping part.  This is actually 
> fairly simple to work out, but does depend on a working 
> storage stack.  However, we can start by putting together the 
> data structures that keep track of physical pages and make 
> use of the additional values in the structures once we're 
> able to.  I honestly haven't given much thought to which 
> paging algorithm would be best, but there's so much research 
> out there we'll have plenty of help picking one.
> 
> Closing Thoughts
> 
> First and foremost, I'm hoping that we stick with an object 
> based design.  My reason is centered around security, as 
> applying security when everything is an object is much 
> easier.  That might not seem like it has anything to do with 
> memory, but trust me, it does.  But we won't actually deal 
> with it until the basic virtual memory is done.

I agree with you on this.

Re: [SharpOS Developers] MOSA / AOT

[grover <sh...@mi...>]

We can continue development with the AOT. I have never asked for anything
else. It is my opinion that we should continue with it, refactor and
redesign the existing code base and use the time to sit down and do basics
like you do with memory management.
 
Mike 


  _____  

Von: sha...@li...
[mailto:sha...@li...] Im Auftrag von
Zachary Gorden
Gesendet: Dienstag, 5. August 2008 09:45
An: sha...@li...
Betreff: Re: [SharpOS Developers] MOSA / AOT


Even if the intention is to replace the AOT, there shouldn't be any reason
to not continue development using it right now.  The code should still
compile with a new compiler.

Re: [SharpOS Developers] MOSA / AOT

[grover <sh...@mi...>]

Hi Sander,

There's only two options available IMHO:

1. IL2CPU from Cosmos

Which could be used, but has its fair share of issues too. I'm certain that
we could get further with it than the AOT, but we still can't do an advanced
design AFAIK, as there's no support for Attributes, Reflection, ... However
we would have to ask Matthjis about this.

2. Bartok

We all know what that means ;)

Ensemble doesn't have one, but Scott is contributing to the MOSA compiler
and in total there's now 4-5 people working on it directly or indirectly. A
prioritized list of features would allow us to drive the compiler into a
specific direction, e.g. support certain things earlier than later. Ensemble
is in the same position as SharpOS with this respect, except they don't even
have an AOT to work with. Further speedups in the compiler are not possible
right now, as there's some blocking points that need to be resolved before
we go ahead and add more manpower to the compiler. I'm happy with those who
contribute their time, code and mind to it.

In fact I think we should use the time to redesign specific parts of the
kernel and refactor the existing code base. I know its hard without writing
a piece of new code, but I think this is the right time to document what we
have, where we want to go and how we want to get there.

Mike

> -----Ursprüngliche Nachricht-----
> Von: sha...@li... 
> [mailto:sha...@li...] Im 
> Auftrag von Sander van Rossen
> Gesendet: Dienstag, 5. August 2008 10:42
> An: sha...@li...
> Betreff: Re: [SharpOS Developers] MOSA / AOT
> 
> Yes, but the problem is that the AOT is missing some 
> functionality, or has some bugs, which prevent us in properly 
> implementing some parts of the kernel.
> In a way, we're kinda stuck and can't go forward because of it.
> So either we fix those bugs in the AOT, and implement those 
> features that we need (which would be hard because the 
> person(s) that know the most about the AOT are no longer 
> active), or we wait until the new MOSA compiler is mature and 
> developed enough to continue with the kernel.
> 
> Just to throw in another option:
> Don't the other teams, or at least the Cosmos, have compilers as well?
> Although they're not designed as well in the basis as the 
> AOT, maybe we could 'borrow' one of these compilers for the 
> time being until the MOSA compiler is finished?
> 
> 
> On Tue, Aug 5, 2008 at 9:45 AM, Zachary Gorden 
> <dra...@gm...> wrote:
> > Even if the intention is to replace the AOT, there shouldn't be any 
> > reason to not continue development using it right now.  The code 
> > should still compile with a new compiler.
> 
> --------------------------------------------------------------
> -----------
> This SF.Net email is sponsored by the Moblin Your Move 
> Developer's challenge Build the coolest Linux based 
> applications with Moblin SDK & win great prizes Grand prize 
> is a trip for two to an Open Source event anywhere in the 
> world http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> SharpOS-Developers mailing list
> Sha...@li...
> https://lists.sourceforge.net/lists/listinfo/sharpos-developers
> 

Re: [SharpOS Developers] Additional stuff regarding memory

[Sander v. R. <s.v...@sy...>]

Cool, good to see that someone has put some time into the basics of a design.

My responses are meant as positive criticism, and meant to start a
dialog, not to force my opinion onto anyone. I -expect- people to come
up with counter arguments. Hopefully trough this dialog we can come up
with a good design.
That being said:

> Virtual Address Space Organization
>
> The virtual address space in SharpOS will follow the traditional 2GB split for kernel and user mode
> in 32bit mode, however this is a mere formality, as there does not exist a user mode.
> It is more for organization purposes so that the kernel has sufficient reserved space for its own
> data structures. The only thing prevent applications and other unauthorized code from accessing
> the kernel portion will be the managed nature of such code.

Does that still make sense with a micro-kernel? After all, we're not
talking about a monolithic kernel which uses a ton of resources, we're
talking about a lot of smaller interconnected pieces which each use a
relatively small amount of resources...
'kernel' and 'user' mode don't really exist, at least not in the
traditional sense, since we'd use a more detailed rights model to
determine what a process may or may not do.

If you consider the libraries as 'the kernel' then this split would
make more sense.
(We really need better terminology when it comes to this)
Also, we wouldn't want to expose all libraries to all appdomains..
basically only the ones they'll be using, yet we do want to share the
code-memory of libraries between appdomains.
(The memory pages containing the libraries should also be set to
read-only, but i hope that's obvious to everyone here...)


> The kernel range will be the traditional upper 2GB of the 4GB virtual space.
> It will be a series of preallocated paged and unpaged pools of memory,
> specifically set aside for usage by the kernel.

Sounds a bit much for 'the kernel'. Keep in mind all the drivers etc.
will be outside 'the kernel' and inside 'user mode'..


> The lower 2GB reserved for the application is where things get interesting.
> Because everything the application does is managed, we might be able to
> shift almost all responsibility to the GC and company.  It requires some
> thought, as I don't know what is intended for management of application memory.

That's sounds rather vague, and we'll need regular heaps for things
like inner-process communication as well.
although arguably, this would be part of 'the kernel'.
Also, we'd want a GC for every process separatly, it would allow much
quicker releasing of the memory
of a process when it shuts down, and would also keep the GC
generations much smaller and easier to manage.
Imagine having to do a system wide garbage collection! You'd have to
freeze the -entire- system and walk trough -all- the objects in -all-
the processes! every time!


> Okay, I'm lying.  What I really want to know is when people keep saying "GC
> kicks in when memory is low," what the blazes do they mean?
> Are they talking about the GC being aware of physical pages actually running low?
> Or just this heap that the GC is supposedly managing, which might as well
> be a range of virtual addresses that might all be paged out for all we know?
> Until that point gets cleared up, I'm stuck as to what I need to provide for
> the upper level services.

It's gonna be tricky, especially since we need to mix virtual memory
that's been paged to disk with memory that's used in the garbage
collector..
And we don't want to have to read all the memory back from the page
file to determine what can be collected.
We might be forced to swap whole processes to and from disk instead of
pieces of memory that haven't been used in a while.
Or restrict it to large blocks of memory (but keep some small
book-keeping information in memory to be able to walk trough the
process to see what's still 'used' and what needs to be collected)

As for when to garbage collect.. i suppose there are some thresholds
that when crossed would trigger a collection.. the most obvious ones
are when there's not enough memory available to allocate some memory
for a process, or when the system is (relatively) idle you do a
collection (to prevent a collection occurring when the system is hard
at work), or before (part of) a process is written to the page file
(to prevent memory being written to the page file which isn't actually
used anymore anyway)

We'd need some way to assign and collect pages of memory from
processes as well, and a way to force other processes to do a
collection when memory is full and a process needs more memory (and
hopefully another process would release some memory after a
collection)..


> The tricky part we now have to deal with is the initial mapping of the kernel into
> virtual address space and putting together a page table for it.  I still don't fully
> know how I'm going to pull it off, but it will definitely require some refactoring
> of the code.  I also need to pick a point early enough in the loading where
> I won't be overwhelmed with repointing all the pointers.

Don't worry to much about the refactoring at this point, we need a
good design first, and only then should we worry about making it
reality..

Re: [SharpOS Developers] MOSA / AOT

[Sander v. R. <s.v...@sy...>]

Yes, but the problem is that the AOT is missing some functionality, or
has some bugs, which prevent us in properly implementing some parts of
the kernel.
In a way, we're kinda stuck and can't go forward because of it.
So either we fix those bugs in the AOT, and implement those features
that we need (which would be hard because the person(s) that know the
most about the AOT are no longer active), or we wait until the new
MOSA compiler is mature and developed enough to continue with the
kernel.

Just to throw in another option:
Don't the other teams, or at least the Cosmos, have compilers as well?
Although they're not designed as well in the basis as the AOT, maybe
we could 'borrow' one of these compilers for the time being until the
MOSA compiler is finished?


On Tue, Aug 5, 2008 at 9:45 AM, Zachary Gorden <dra...@gm...> wrote:
> Even if the intention is to replace the AOT, there shouldn't be any reason
> to not continue development using it right now.  The code should still
> compile with a new compiler.

Re: [SharpOS Developers] MOSA / AOT

[Zachary G. <dra...@gm...>]

Even if the intention is to replace the AOT, there shouldn't be any reason
to not continue development using it right now.  The code should still
compile with a new compiler.

[SharpOS Developers] Additional stuff regarding memory

[Zachary G. <dra...@gm...>]

Decided to get off my ass and provide some more detailed thoughts about
what's needed for the memory system.  So far, this is more a matter of
intentions than set plans.  It also raises some points I don't fully
understand and would appreciate insight on.

SharpOS Memory Architecture Design

This is a basic outline of the low level memory management system of
SharpOS.  This does not include the higher level
garbage collection and managed memory schemes, but the basic services that
they will depend on.

Virtual Address Space Organization

The virtual address space in SharpOS will follow the traditional 2GB split
for kernel and user mode in 32bit mode,
however this is a mere formality, as there does not exist a user mode.  It
is more for organization purposes so that
the kernel has sufficient reserved space for its own data structures.  The
only thing prevent applications and other
unauthorized code from accessing the kernel portion will be the managed
nature of such code.

Using a separate page table for each appdomain will mean we must change the
value in CR3, so we will need to do a
memory context switch.  This results in increased overhead and may be
avoidable in 64bit mode, but will require
some experimentation in the future.  The page table entries describing the
kernel range will be shared by all page
tables.  Intel processors have support for this so it shouldn't be
problematic.

The kernel range will be the traditional upper 2GB of the 4GB virtual
space.  It will be a series of preallocated
paged and unpaged pools of memory, specifically set aside for usage by the
kernel.  These preallocated data structures
will need some tweaking in order to ensure there is enough for each.  For
allocations smaller than a page in size,
there will be a series of stacks of varying sizes.  Allocations from them
will be done through lockfree atomic
instructions to ensure better throughput.  Whether a general purpose pool
will be needed or even possible I'll have
to look into, as traditionally you cannot allocate to the heap like you
would in user mode.

The lower 2GB reserved for the application is where things get interesting.
Because everything the application
does is managed, we might be able to shift almost all responsibility to the
GC and company.  It requires some thought,
as I don't know what is intended for management of application memory.

Okay, I'm lying.  What I really want to know is when people keep saying "GC
kicks in when memory is low," what the
blazes do they mean?  Are they talking about the GC being aware of physical
pages actually running low?  Or just this
heap that the GC is supposedly managing, which might as well be a range of
virtual addresses that might all be paged
out for all we know?  Until that point gets cleared up, I'm stuck as to what
I need to provide for the upper level
services.

The tricky part we now have to deal with is the initial mapping of the
kernel into virtual address space and putting
together a page table for it.  I still don't fully know how I'm going to
pull it off, but it will definitely require
some refactoring of the code.  I also need to pick a point early enough in
the loading where I won't be overwhelmed
with repointing all the pointers.

Physical Memory

This is basically the paging/swapping part.  This is actually fairly simple
to work out, but does depend on a working
storage stack.  However, we can start by putting together the data
structures that keep track of physical pages and
make use of the additional values in the structures once we're able to.  I
honestly haven't given much thought to
which paging algorithm would be best, but there's so much research out there
we'll have plenty of help picking one.

Closing Thoughts

First and foremost, I'm hoping that we stick with an object based design.
My reason is centered around security, as
applying security when everything is an object is much easier.  That might
not seem like it has anything to do with
memory, but trust me, it does.  But we won't actually deal with it until the
basic virtual memory is done.

Re: [SharpOS Developers] MOSA / AOT

[<sh...@mi...>]

Hi Sander,

Before I left for vacation the compiler started emitting its first machine code instructions. Since then I've continued my path of refactoring and enhancing to stay with a clean design. I don't want to speculate about completion dates, but its not a matter of weeks. There's a lot still pending and one of the goals I have is being able to compile SharpOS as it is right now - which it can't do yet for various reasons.

Anyways for user mode, there's a defined API: The .NET Framework (with Mono libs) - anybody should be able to work with those and develop their user-mode apps. For kernel mode the story is a bit different and I'd like for those who want to work there to sit down and discuss what they want to do and maybe really do a design-first, document-first, prototype-first, test-first style of development (which IMHO this project really really needs.)

I'm all for meeting up and discussing various aspects and like I said previously we should try to get at least Scott to the table. We definitely are able to specify interfaces and we are able to write device emulators to get us started, the compiler will catch up quickly. Btw. that is the path that Ensemble is taking and Phil is moving a lot of his work over to MOSA so both projects can benefit here.

The compiler will steadily improve, however in order for the improvements to be measurable we need to improve the testing situation. I've started a test system using mbUnit, which allows us to test the created code right on a Windows box* reducing the turn-around times. This has worked beatifully for the integer addition and could easily be extended to other basic operations on all integral and floating point types. I'm thinking of doing a compiler status page, where  everyone can see the things supported/not supported and the progress can be monitored - this basically should clear up all the confusion if the compiler does support feature Y or not. I'd wish we could automate this with appropriate test cases and with the help of some continuous integration toolkit to automatically generate these reports for everyone to see them.

I'm planning on allowing the VES I'm developing as part of the compiler to be hosted inside the MS CLR, so we could even go as far as running code on Windows in our own runtime environment hosted inside the MS CLR. These efforts should dramatically improve development as booting up SharpOS (or any other kernel) in a VM is always a lengthy operation and I'd like to get to the point of being able to run/debug various services right on Windows/Linux as long as they don't touch real hardware. And this can work for lots of operating system services.

Mike

* I haven't started Linux support as monos gmcs compiler in version 1.9 lacks some features from the C# language I've been using, however the new 2.0 preview should compile and run the compiler fine. I'm not sure about mbUnit though and I'd need to rewrite the test suite due to manual executable memory allocations etc.

Sander van Rossen schrieb: 


I'm absolutely for an initiative such as MOSA, and think it's a
fantastic idea and we should definitely go with it.
But if we're not going to use the AOT, the next question is; how far
along is the MOSA compiler, and how long (roughly) will it take before
we can start to realistically use it?
Because there are a couple of show-stopper bugs/features missing in
the AOT (such as the interface problem that Phil emailed about
before), and until we can compile the kernel as it is, and beyond, the
whole kernel development comes to a stand-still.
And like Bruce mentioned, most people here want to develop at the
kernel or user level..
So how are we going to handle this situation?

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
SharpOS-Developers mailing list
Sha...@li...
https://lists.sourceforge.net/lists/listinfo/sharpos-developers

Re: [SharpOS Developers] MOSA / AOT

[Sander v. R. <s.v...@sy...>]

I'm absolutely for an initiative such as MOSA, and think it's a
fantastic idea and we should definitely go with it.
But if we're not going to use the AOT, the next question is; how far
along is the MOSA compiler, and how long (roughly) will it take before
we can start to realistically use it?
Because there are a couple of show-stopper bugs/features missing in
the AOT (such as the interface problem that Phil emailed about
before), and until we can compile the kernel as it is, and beyond, the
whole kernel development comes to a stand-still.
And like Bruce mentioned, most people here want to develop at the
kernel or user level..
So how are we going to handle this situation?

Re: [SharpOS Developers] Trunk Status Inquiry

[grover <sh...@mi...>]

Bruce,

I'm mrfl on sourceforge. So yes, those are my commits. They're mostly  
in my sandbox for a VFS layer and the MOSA compiler. However VFS has  
been taken over by Phil as he included a FAT driver in there.

Mike

Am 04.08.2008 um 03:13 schrieb Bruce Markham:

> Due to the lack of regularly scheduled development on the project,  
> combined with the fact that there has still been development, I was  
> hoping that a few of the people involved with changes in the trunk  
> could put together summaries of what they have been working on for  
> the last several months.
>
> The majority of commits have been done by tgiphil - and some of  
> those are larger than they appear because they are merges from  
> sandboxes. LogicalError has also been doing lots of refactoring-type  
> work (based on me just skimming through SVN logs since April). And  
> mrfl has been posting some changes. (Is that grover?)
>
> I think the best way for us to move forward is for us to get an idea  
> of what you guys have been tinkering with. I mean, I can compile and  
> run the kernel - and other than some instabilities (which we've  
> always had, but they've certainly changed over time it seems),  
> running the kernel in and of itself does not reveal much about the  
> changes ("listresources" command is fascinating, but sometimes it  
> makes QEmu lock up...). And frankly, I can't make heads or tails of  
> the architecture from looking at the code anymore. (Which is from  
> enthusiastic efforts on each of you all's parts, but largely  
> seemingly uncoordinated.)
>
> So what I'm really asking is - can each of you three, please take  
> some time to tell us about your work on SharpOS recently? We have  
> the development blog, which has sat untouched since obsethryl's  
> interview. LogicalError already has posting priveleges - so if  
> grover (mrfl?) and tgiphil could send me, privately, a Google  
> Account alias of yours, (feel free to create one, or just send me a  
> personal e-mail of yours that you want to sign in to blogger for  
> SharpOS with), I'll get you guys posting privileges.
>
> After which, I would ask that each of you spend the time, say about  
> an hour, and write up an article on what you've been working on, and  
> more importantly, why. Where it is going. What you want to see  
> happen. And then you can e-mail the mailing list to let us know when  
> you are finished, (preferably linking to the post), so that we can  
> restore an organic feeling of what is going on here.
>
> If you have time for graphics and diagrams, don't let me stop you.  
> However, I'm really thinking all we need is baseline textual  
> thoughts. Mention some classes you've written, some paradigms you  
> are trying to apply, etc. I'm hoping it will make up for a lack of  
> Redmine and regularly-attended-and-documented SharpOS meetings for  
> the last 4 months.
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's  
> challenge
> Build the coolest Linux based applications with Moblin SDK & win  
> great prizes
> Grand prize is a trip for two to an Open Source event anywhere in  
> the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/_______________________________________________
> SharpOS-Developers mailing list
> Sha...@li...
> https://lists.sourceforge.net/lists/listinfo/sharpos-developers