syllable-kernel

RE: [Syllable-kernel] Arno's kernel patches

[Kristian V. <Kri...@an...>]

Jake Hamby wrote:
> Kristian Van Der Vliet wrote:
>> On Friday 28 January 2005 03:26, Jake Hamby wrote:
>> 
>>>Also, what is the verdict as far as the stability of Arno's patches?
>>>Did the Jan. 23 patch removing the cli()/put_cpu_flags() calls make
>>>things better or worse?  I'm concerned about Vander's reports of
>>>instability during compiles and I'd much rather have a stable kernel
>>>that only works on some machines than a kernel that boots on more
>>>machines but isn't reliable.
>> 
>> 
>> It fixed booting on SMP (& HT) machines at the expense of 
>> stability on SMP and uni machines.  I wouldn't consider it a
>> suitable final patch for inclusion in the next release of
>> Syllable without further work to stabalise it.
> 
> Okay, that's what I thought.  I was a little concerned because, if 
> anything, we should be _adding_ SMP sychronization mutexes, 
> not removing them, unless they are obviously unnecessary and not
> in the critical startup path.

The odd thing is that it fixes the SMP boot problems.  Quite why removing
syncronisation primitives should have that effect is still somewhat of a
mystery.  However Arno did say that he removed them from functions which
were already being called with interupts disabled, so they were redundent.

> In our case we have AFS attributes, and any app is free to use XML or 
> any other format to store settings so, outside of creating an 
> equivalent to the hidden "Application Data" and "Local Settings" folders 
> in Windows (perhaps ".config" if this hasn't already been decided?) in 
> each user's home directory for personal settings, we should be okay on
> that end. 

See os::Settings, which stores configuration data in
$HOME/Settings/<appname>/Settings  This settings file is a flattened
os::Message object streamed to disc, which in a way is a simple
key::value model with no tree structure.  It works very well.

> But for communications between the kernel and user programs, we need 
> something a little different..<snip>
> 
> Anyway, I will look into the capabilities of the registrar as well as 
> read through the d-bus mailing list archives to see how it's 
> being used, and see if perhaps we can begin to use the registrar for 
> communication of information between user processes and the kernel.

One idea I had was to add an events API to the kernel.  Any code, either
kernel space or user-space, could call event() which would queue up an
event in a message queue.  A user space process E.g. the Registrar, could
then pull these simple messages out of the queue.  Applications could
register with the Registrar to receive events that they are interested
in.

So E.g. inserting a USB device would cause the following to happen:

o USB HCI driver gets interupt, finds new hardware, raises an event with
  a "Hardware inserted" class.
o The Registrar sees the event in it's queue and checks to see which
  applications are waiting for "Hardware inserted" events.
o The Registrar constructs an os::Message object with the details from
  the event and posts the message to every Application currently waiting
  for "Hardware inserted" events.

> I would very much like for Syllable to have an equivalent to the Windows
> task manager (I want to write my own in order to learn the syllable API),
> and I don't want us to continue passing information in fixed length 
> structures, as we're currently doing in get_system_info().

Well there is already Syllable Manager, which offers a process list and
some basic process control.

What you seem to be proposing (& correct me if I'm wrong) is some way to
pass a dynamic tree of key::value pairs between application and kernel space
via. a consistent API.  Which is a brilliant idea which would simply the
entire
process of passing data between the kernel and user-space much much easier.
Just think, a /proc type system without the need to mess with parsing
human-readable text in the userspace process.  Forward *and* backward
compatability.  Complex data structures flowing seamlessly...mmm. If you
coupled this with the event model I describe above, just think of the
possibilities of kernel and user-space integration we could achieve!

-- 
Vanders
http://www.syllable.org
http://www.liqwyd.com

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________

RE: [Syllable-kernel] Arno's kernel patches

[Kristian V. <Kri...@an...>]

Arno Klenke wrote:
> The kernel patch is now available. It changes the following things:
> 
> *Fixes the acpi table parsing errors and adds a kernel option 
> to disable 
> the acpi support
> *Removes the unnecessary cli()/put_cpu_flags() calls
> *Removes the reflect_irq_to_realmode() call

I'm curious about this change:

-	g_sSysBase.ex_GDT[0x40 << 3].desc_lmh &= 0x8f;
+	g_sSysBase.ex_GDT[0x40 >> 3].desc_lmh &= 0x8f;

I seem to recall yourself or Jake mentioning an issue with the GDT
descriptor, I assume this is the fix?  What effect does this have
E.g. does it fix stability on SMP systems?

I'm guessing that code hasn't changed since AtheOS..how did it ever
work correctly in the first place?

-- 
Vanders
http://www.syllable.org
http://www.liqwyd.com


______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________

RE: [Syllable-kernel] Arno's kernel patches

[Kristian V. <Kri...@an...>]

Jake Hamby wrote:
> Speaking of bad performance, I'm frustrated that there are a 
> couple of threads that chew up a big chunk of CPU time on my system:
> 
> dbterm:  main thread (14%), read thread (7%)
> media_server_flush:  21.7%
> ata_cmd_thread:  up to 17% for each thread
> appserver: keyb_thread: 7.1%
> appserver: input_thread:  2.6%

dbterm is almost certainly because it has to poll the kernel for new
data on the kernel debug port.
ata_cmd_thread may currently have some tight loops in it where it is
impossible to do anything else but poll for a status change.  The ATA
controller only raises an interupt at certain times.  If you're just
waiting for E.g. STATUS_BUSY to go low you must poll the status register
until it happens.  The best you can do is snooze() during the loop.

-- 
Vanders
http://www.syllable.org
http://www.liqwyd.com

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________

Re: [Syllable-kernel] Arno's kernel patches

[Jake H. <jh...@po...>]

Kristian Vandervliet wrote:
> 
> The odd thing is that it fixes the SMP boot problems.  Quite why removing
> syncronisation primitives should have that effect is still somewhat of a
> mystery.  However Arno did say that he removed them from functions which
> were already being called with interupts disabled, so they were redundent.

I think there is some dark voodoo involved in bringing up the additional 
CPU's correctly.  Wouldn't disabling interrupts on the active CPU with 
cli() prevent that processor from being able to receive any IPIs 
(inter-processor interrupts) that it needs to work correctly?  Perhaps 
that's the problem.

> See os::Settings, which stores configuration data in
> $HOME/Settings/<appname>/Settings  This settings file is a flattened
> os::Message object streamed to disc, which in a way is a simple
> key::value model with no tree structure.  It works very well.

Cool.  Simple BeOS style.  Yeah, app settings are not exactly rocket 
science to do at the libsyllable level, which is why it is so annoying 
that Microsoft has had so much trouble with the registry over the years.

> One idea I had was to add an events API to the kernel.  Any code, either
> kernel space or user-space, could call event() which would queue up an
> event in a message queue.  A user space process E.g. the Registrar, could
> then pull these simple messages out of the queue.  Applications could
> register with the Registrar to receive events that they are interested
> in.

My understanding is this is basically what d-bus is about.  udev, the 
Linux user-mode /dev management demon and replacement for devfs, 
interfaces with the hardware using hotplug scripts and can fire events 
to d-bus.  An alternative API for routing events is the POSIX Tracing 
API, described here and implemented by RTL-POSIXTRACE as an add-on for 
RT-Linux:

http://www.opengroup.org/onlinepubs/000095399/functions/xsh_chap02_11.html

Since POSIX tracing isn't supported by Solaris or any of the other 
commercial UNIX's and is only available as an obscure add-on for 
RT-Linux, it's not a very useful "standard" for us to support.  OTOH, 
while d-bus seems to have some momentum, integration between it and 
GNOME or KDE is still pretty minimal, as far as I can tell.  Here's a 
good technical article on what d-bus is about, with some sample code:

http://www-106.ibm.com/developerworks/linux/library/l-dbus.html

> Well there is already Syllable Manager, which offers a process list and
> some basic process control.

Yeah, I remember seeing that but forgot what it was called.  Is it 
installed in the default Syllable install or do I have to download it 
from somewhere?

> What you seem to be proposing (& correct me if I'm wrong) is some way to
> pass a dynamic tree of key::value pairs between application and kernel space
> via. a consistent API.  Which is a brilliant idea which would simply the
> entire process of passing data between the kernel and user-space much much easier.

Exactly.  It's a great idea and we can take advantage of the existing 
D-BUS codebase for Linux in addition to the registrar code we have now.

> Just think, a /proc type system without the need to mess with parsing
> human-readable text in the userspace process.  Forward *and* backward
> compatability.  Complex data structures flowing seamlessly...mmm. If you
> coupled this with the event model I describe above, just think of the
> possibilities of kernel and user-space integration we could achieve!

Yeah, /proc is gross.  Something like d-bus could even be used, if fast 
enough, by a debugger to inspect and manipulate processes, instead of 
using /proc or ptrace().

There are a lot of security implications of course, but no different 
from the permission issues of a filesystem or any other resource.  I 
would assume that as part of the SylSec project, we will have some sort 
of global security token system where a process will have permission to 
access a certain branch of the hierarchy or a particular key/value pair 
based on the object's ACL and the process's credentials.  Slightly 
off-topic but I have found Windows NT's implementation of security in 
e.g. Active Directory or NTFS that supports BOTH "allow" and "deny" bits 
for many different operations, and supports different ways of 
propagating permissions up the hierarchy, extremely confusing!

Another aspect that may be challenging is how these security tokens and 
credentials could be manipulated and transferred between the kernel and 
user-mode for implementing objects and events in user-space.  One 
possibility is adding a special system call to allow one process to 
authenticate another over a message port, for example FreeBSD allows you 
to send the pid, uid, and group ID of the sending process to its peer on 
the other end of the socket (only over UNIX local domain sockets, not 
through TCP/IP) with SCM_CREDS (the Linux version is called SO_PEERCRED) 
using a special unspoofable message flag.  Sending credentials between 
machines would require something like Kerberos.  In the case of peer 
authenticating across a socket, you are not actually transferring any 
privileges between processes, just allowing a process to know for sure 
the uid/gid/pid of the process it's talking to.  A more useful thing 
would be a fully transferrable key system such as KeyKOS or EROS:

http://www.cis.upenn.edu/~KeyKOS/
http://www.cis.upenn.edu/~eros/

-Jake

Re: [Syllable-kernel] Arno's kernel patches

[Michael S. <mi...@as...>]

On Fri, 28 Jan 2005, Jake Hamby wrote:
>
> > Well there is already Syllable Manager, which offers a process list
> > and some basic process control.
>
> Yeah, I remember seeing that but forgot what it was called.  Is it
> installed in the default Syllable install or do I have to download it
> from somewhere?

It's in the default install -- IIRC just entering SlbMgr in a terminal
will fire it up. Failing that, go to the Dock menu and About, and there's
an 'Advanced' (or similar) button which brings it up.

Along with processes it also displays some resource stats.

M

-- 
Michael Saunders
www.aster.fsnet.co.uk

Re: [Syllable-kernel] Arno's kernel patches

[Jake H. <jh...@po...>]

Michael Saunders wrote:
> 
> It's in the default install -- IIRC just entering SlbMgr in a terminal
> will fire it up. Failing that, go to the Dock menu and About, and there's
> an 'Advanced' (or similar) button which brings it up.
> 
> Along with processes it also displays some resource stats.

Cool.  In that case I will probably start from that codebase instead of 
trying to write my own GUI from scratch.  I am mostly interested in 
fleshing out the additional capabilities of Windows task manager not yet 
implemented and in using something like D-BUS to abstract this type of 
system info utility from the specific version of the kernel being used.

-Jake

Re: [Syllable-kernel] Arno's kernel patches (part 2)

[Jake H. <jh...@po...>]

The link for EROS in my last e-mail was incorrect.  It should have been:

http://www.eros-os.org/

Since I didn't have a good example of d-bus/GNOME integration, here is 
one from the IBM article:

==
Even though D-BUS is relatively new, it has been adopted very quickly. 
As I mentioned earlier, udev can be built with D-BUS support so that it 
sends a signal when a device is hot-plugged. Any application can listen 
to these events and perform actions when they are received. For example, 
gnome-volume-manager can detect the insertion of a USB memory stick and 
automatically mount it; or, it can automatically download photos when a 
digital camera is plugged in.

A more amusing but far less useful example is the combination of 
Jamboree and Ringaling. Jamboree is a simple music player that has a 
D-BUS interface so that it can be told to play, go to the next song, 
change the volume, and so on. Ringaling is a small program that opens 
/dev/ttyS0 (a serial port) and watches what is received. When Ringaling 
sees the text "RING," it uses D-BUS to tell Jamboree to turn down the 
volume. The net result is that if you have a modem plugged into your 
computer and your phone rings, the music is turned down for you. This is 
what computers are for!
==

Pretty cool hack!  Even cooler would be to detect that your cellphone is 
ringing using some sort of Bluetooth notification, or you could try to 
detect if anyone's phone is ringing using a microphone and an FFT 
analysis to listen for "ringing" sounds.  Then you would only need to 
write a replacement for Ringaling and Jamboree, or in our case Coldfish, 
would continue to work unchanged.  A further hypothetical:  this "ring" 
notification app could be wired by the user to implement an arbitrary 
user or system-wide policy such as lowering the volume on all audio 
output devices, perhaps by triggering a script or a plugin dll.  Or 
there might be multiple apps that want to listen for this event.

On that note, I discovered another article describing a way in which you 
could abstract services using d-bus where it wouldn't matter which 
application you were using to do something.  It sounds somewhat like how 
Amiga apps could be scripting using ARexx or Mac apps could be scripted 
with AppleScript and OSA, but more generic between applications.

http://code.eikke.com/DesktopAbstractionLayer/presentation-1.txt

I had a CS professor at my university, Dr. Craig Rich, who had a pretty 
neat layout of his applications and documents on his Mac (this was 
before OS X).  He renamed all of his applications from whatever their 
original names were to generic names, for example Netscape was renamed 
to "Web Browser" and Eudora became "E-Mail Client".  In the Mac world, 
renaming the executable generally doesn't break anything.  He could then 
change e-mail clients, for example, and of course the icon would change, 
but the name would be the same and so the app would be alphabetized into 
the same place in the Apple quick launch menu.

Also notice that Windows only has user associations for certain types of 
Internet locations (HTML, e-mail, news) plus file extensions and events 
such as inserting a blank CD-R or a DVD movie.  You can choose the 
default audio output device or what happens when you double-click on a 
.mp3 file, but if there is a certain activity that isn't tied to a 
particular data file or URL, for example the default "TV tuner" app, I'm 
not sure how you would specify that except through some special registry 
key or something.

-Jake

Re: [Syllable-kernel] Arno's kernel patches

[Daniel G. <da...@fp...>]

On Fri, 2005-01-28 at 06:29 -0800, Jake Hamby wrote:
> My understanding is this is basically what d-bus is about.  udev, the 
> Linux user-mode /dev management demon and replacement for devfs, 
> interfaces with the hardware using hotplug scripts and can fire events 
> to d-bus. 

This is basically correct.  Hotplug is migrating to kevents, rather than
callout scripts, but other than that, that's correct.

>  OTOH, 
> while d-bus seems to have some momentum, integration between it and 
> GNOME or KDE is still pretty minimal, as far as I can tell. 

Not quite true.  Gnome 2.10 will have a considerable amount of dbus
integration, and will require it to be installed to work properly.  For
example, you mentioned Gnome Volume Manager.  There's also
NetworkManager, which listens for interface events (loss of carrier, new
wireless AP, etc.) and automatically configures networking.  Evolution
will emit various dbus messages based on state of mail/calenders, etc.
For example, it emits a new-mail message, and uses dbus to notify a
watcher that a scheduled event has fired from the calender.  There are
other examples as well.  By Gnome 2.12/3.0, dbus should be completely
integrated into Gnome, and be a full requirement.  I don't follow KDE,
so I don't know what it's doing, but dbus is the wave of the future for
Gnome, and generally will be required for all Linux.

Daniel
(Who's happily running dbusified gnome 2.9.x)

Re: [Syllable-kernel] Arno's kernel patches

[Saem G. <sae...@gm...>]

On Fri, 28 Jan 2005 17:37:06 -0500, Daniel Gryniewicz <da...@fp...> wrote:
> >  OTOH,
> > while d-bus seems to have some momentum, integration between it and
> > GNOME or KDE is still pretty minimal, as far as I can tell.
> 
> Not quite true.  Gnome 2.10 will have a considerable amount of dbus
> integration, and will require it to be installed to work properly.  For
> example, you mentioned Gnome Volume Manager.  There's also
> NetworkManager, which listens for interface events (loss of carrier, new
> wireless AP, etc.) and automatically configures networking.  Evolution
> will emit various dbus messages based on state of mail/calenders, etc.
> For example, it emits a new-mail message, and uses dbus to notify a
> watcher that a scheduled event has fired from the calender.  There are
> other examples as well.  By Gnome 2.12/3.0, dbus should be completely
> integrated into Gnome, and be a full requirement.  I don't follow KDE,
> so I don't know what it's doing, but dbus is the wave of the future for
> Gnome, and generally will be required for all Linux.

KDE has had DCOP it's own messaging system.  DBUS integration is
planned, basically AFAIK DCOP will sit on top of DBUS as a legacy
while everyone else will migrate to DBUS, KDE 4.0 is supposed to
herald this change.  There is HUGE momentum behind this stuff.