squeak-vmdev — CVS-related Virtual Machine Development Issues

[Squeak-VMdev] re: simulating current snapshots

[Dan I. <Da...@Sq...>]

Hi, Ned -

>On Friday 09 July 2004 12:33 pm, Dan Ingalls wrote:
>
>> I am now at the point where the 64-bit system works essentially as well as
>> the 32-bit system in simulation, but that is not good enough.  There are
>> failures in both during morphic display, and these become fatal as they
>> recur during attempts to put up error windows.
>
>Where are you seeing the problems?

As I mentioned in that message,
> Most of my current failures stem from BalloonEngine>>#primInitializeBuffer:
However I think some other things may be failing that shouldn't.

> > Moreover the latest system,
>> even when running MVC, uses outline fonts, and there are failures in
>> simulating these.
>
>It doesn't have to use outline fonts. Just remove the TTCFont and set the
>system fonts to not use them.

Yes, and that will be useful for people who want small MVC images, but my goal is no longer to avoid the problems but to deal with them.

> > I need to know some very basic information, which has changed since I was
>> last in control of the VM and simulation.
>>
>> 1.  If everything worked right, is it the case that a current system should
>> be able to simulate itself?  What this means is that, provided that the
>> plugin interface all works (and fails) properly, does all the necessary
>> failure code exist such that the system will run completely and with
>> integrity?  If this is not so, then a further question is, apart from
>> whatever fixes may be needed, how much new code would need to be written?
>
>Did you get the various simulation changes that I and others posted (I think I
>mailed some to you)? I don't think they have gotten back into the image (but
>they should), because they needed more testing.

I thought that Tim had folded them into the VMMaker that he put out after that time.  If he didn't, then that is unfortunate, and I'm not sure that they are still additive.  This is why it would be so useful for me to get one consistently working system.  I could then fairly easily force that latest release and the latest VMMaker into full compliance.  [but the short answer to your question is no;  do you know if they still work with the latest VMMaker?]

> > 2.  Assuming the answer to (1) is yes (and I sure hope it is), then the
>> question is, does anyone anywhere have a recent version of Squeak that can
>> simulate itself successfully in Morphic?  This means displaying the mouse (
>> ideally), and a browser in which one can at least type 3+4, and print the
>> value.  If so, then this is all I need, as I can use that to find whatever
>> problems there are in the latest 3.7 and VMMaker that I am using for the
>> basis of the 64-bit work.
>
>I had this going at one time, as I recall.

That's what Craig said ;-) ;-).  Seriously, it would be *wonderful* if you could produce any one recent image that can simulate itself with integrity.

> > I would be happy to supply further debugging info, and you can, of course,
> > see it all for yourself if you have time to load up VMMaker in the latest
> > 3.7, and try simulating itself.
>
>What are you loading besides the basic image?

I run the latest 3.7 (5963).  I am using VMM 37b2.1 plus the two-method fix needed to make it work since Tim's latest changes.  That's where I test it.  Then there's a raft of Ian's type reconciliation changes and my 64-bit changes, plus Anthony's Systemtracer2 and my subclass for 64-bit images as well.  but all I need is a current 3.7 with a current VMMaker plus changes that make them able to simulate itself fully and, as I said, I'd settle for one that is not current from which I could repair the current one by side-by-side comparison.

Thanks for responding, Ned

	- Dan

Re: [Squeak-VMdev] Request for help with simulation in 3.7

[Ned K. <ne...@sq...>]

On Friday 09 July 2004 12:33 pm, Dan Ingalls wrote:

> I am now at the point where the 64-bit system works essentially as well as
> the 32-bit system in simulation, but that is not good enough.  There are
> failures in both during morphic display, and these become fatal as they
> recur during attempts to put up error windows.

Where are you seeing the problems?

> Moreover the latest system, 
> even when running MVC, uses outline fonts, and there are failures in
> simulating these.

It doesn't have to use outline fonts. Just remove the TTCFont and set the 
system fonts to not use them.

> I need to know some very basic information, which has changed since I was
> last in control of the VM and simulation.
>
> 1.  If everything worked right, is it the case that a current system should
> be able to simulate itself?  What this means is that, provided that the
> plugin interface all works (and fails) properly, does all the necessary
> failure code exist such that the system will run completely and with
> integrity?  If this is not so, then a further question is, apart from
> whatever fixes may be needed, how much new code would need to be written?

Did you get the various simulation changes that I and others posted (I think I 
mailed some to you)? I don't think they have gotten back into the image (but 
they should), because they needed more testing.

> 2.  Assuming the answer to (1) is yes (and I sure hope it is), then the
> question is, does anyone anywhere have a recent version of Squeak that can
> simulate itself successfully in Morphic?  This means displaying the mouse (
> ideally), and a browser in which one can at least type 3+4, and print the
> value.  If so, then this is all I need, as I can use that to find whatever
> problems there are in the latest 3.7 and VMMaker that I am using for the
> basis of the 64-bit work.

I had this going at one time, as I recall.

> I would be happy to supply further debugging info, and you can, of course,
> see it all for yourself if you have time to load up VMMaker in the latest
> 3.7, and try simulating itself.

What are you loading besides the basic image?

-- 
Ned Konz
MetaMagix
Stanwood WA
(360) 629-1091

[Squeak-VMdev] re: simulating current snapshots

[Dan I. <Da...@Sq...>]

>Hi Dan--
>
>	I was doing other things yesterday, but I've just now started
>regression-testing the simulator with a current system (3.7b.5969). I'm
>loading VMMaker 3.7b5, etc. Hopefully this exercise will take only a few
>minutes... (famous last words :). I intend to fix whatever brokenness I
>encounter.

Many thanks, Craig -

As you will see, I just sent out a plea to the VM list as well.

I'm CC'ing your response to that list as well, just so we don't get too much duplication.  Perhaps you could CC your findings as well, whether good or not.

As soon as something works right, I'll use that as my base and release the updated version with Ian's changes (much better controlled types) and my changes for 64 bits (now able to simulate through most of browser and text display).

	- Dan

[Squeak-VMdev] Request for help with simulation in 3.7

[Dan I. <Da...@Sq...>]

Hi, Guys -

In starting to work on the 64-bit system, i was never able to get any current version of Squeak to simulate itself fully.  Tim Rowledge had a couple of work-arounds that at least allowed some windows to paint, and an early MVC image to run, so I took those and have been using them, in hopes that by the time I got well through the 64-bit work I would find a solid base to work from.

I am now at the point where the 64-bit system works essentially as well as the 32-bit system in simulation, but that is not good enough.  There are failures in both during morphic display, and these become fatal as they recur during attempts to put up error windows.  Moreover the latest system, even when running MVC, uses outline fonts, and there are failures in simulating these.

I need to know some very basic information, which has changed since I was last in control of the VM and simulation.

1.  If everything worked right, is it the case that a current system should be able to simulate itself?  What this means is that, provided that the plugin interface all works (and fails) properly, does all the necessary failure code exist such that the system will run completely and with integrity?  If this is not so, then a further question is, apart from whatever fixes may be needed, how much new code would need to be written?

2.  Assuming the answer to (1) is yes (and I sure hope it is), then the question is, does anyone anywhere have a recent version of Squeak that can simulate itself successfully in Morphic?  This means displaying the mouse ( ideally), and a browser in which one can at least type 3+4, and print the value.  If so, then this is all I need, as I can use that to find whatever problems there are in the latest 3.7 and VMMaker that I am using for the basis of the 64-bit work.

I would be happy to supply further debugging info, and you can, of course, see it all for yourself if you have time to load up VMMaker in the latest 3.7, and try simulating itself.

Most of my current failures stem from BalloonEngine>>#primInitializeBuffer:, which either was written to fail, or fails as a side-effect of other work-arounds.  Unfortunately I cannot figure out what should happen here.  I am similarly in the dark about, eg, transformations in the 3DMatrix plugin, but I believe that I'm getting by by making things fail.

I have asked Craig Latta about this, since he claims to have achieved full simulation of 3.6 in 3.6, but now he's not sure.  I think you two are the only others who may know more or may have gone farther at vairous times in the past, but I'm also CC'ing the VM list in case someone else has more info.

I don't need much here, I don't think.  Either a single working system, or some documentation that relates to simulating the complex plugins.  I can supply lots of traces, and diagnostic info from the simulator transcript if that would be of use.

Thanks in advance for any guidance you can offer.

	- Dan

[Squeak-VMdev] [VICTORY] (was Re: Linux building woes...)

[<gor...@bl...>]

Used the wrong list address, resending:
------------------

Hi Ian and all!

Now I have been battling autoconf/automake/m4 etc for a few days and MY
GOD it can drive you nuts. :) But... it seems I have managed to get
Ian's build process working with automake 1.8.5 and autoconf 2.59 -
these are AFAIK the latest releases of these packages.

I hacked and slashed and dug and messed around A LOT. Why did I start
this torture? After all - Ian's tar-balls (3.6 and 3.7b-5) build fine on
my box! Well - those tar's have the configure script included...

But then I wanted to build from Sourceforge CVS and noticed that it
doesn't have the configure script in CVS. Naturally. Anyway, I use Lunar
Linux and it didn't have automake 1.7.x available - which was the hint
Ned pointed out in order to be able to make the configure script.

First I made a Lunar module for automake 1.7.9 (the latest in the 1.7
series) and modified the Makefile to call "aclocal-1.7.9 -I
/usr/share/aclocal" instead of just "aclocal". That worked after messing
with a few softlinks here and there. I also spent a lot of time messing
with autoreconf but eventually dropped that. Well, it all felt like a
boring solution - it would be nicer if we could get automake 1.8.5 to
work of course!

After staring at weird M4 errors for hours and testing, trying to debug
Perl code (shudder) etc, I finally struck gold. In the end it turned out
that *the only thing* making 1.8.5 not working is this damn line in
configure.ac:

sinclude(acplugins.m4)

It is approximately in the middle of configure.ac and means "silent
include". The idea is that it includes the previously (by mkacinc)
assembled file acplugins.m4. But for some WEIRD reason this makes M4 go
bananas. It starts complaining about AC_ARG_WITH ?!, yadda yadda. I have
NO IDEA why. But whatever - I swiftly "solved" this with some manual
"inclusion" instead. I took configure.ac and split it up into
configure-1.ac and configure-2.ac. Then I changed the Makefile to this:

configure : .force
        cp configure-1.ac configure.ac
        ./mkacinc >> configure.ac
        cat configure-2.ac >> configure.ac
        aclocal
        autoconf
.force :

And... voila! It suddenly works just bloody fine. The configure script
produced in the end is identical to the once produced with 1.7.9 on my
box. I have tried this on Ian's 3.7b tar - and it works. And it also
works fine with SF CVS (HEAD).

Well - I will not commit anything to SF - it seems wiser to leave it up
to Ian to figure out how and if this change should be done. Obviously
sinclude() isn't really up to snuff. Haven't googled the subject
extensively yet.

regards, Göran

PS. What about that CVS move people? Are you all on vacation? :)

[Squeak-VMdev] Linux building woes...

[<gor...@bl...>]

Hi guys!

I am struggling on with my VM-building exercises, this time I am trying
to build Ned's branch, checked out from SqF (may be slightly outdated,
but my issue is probably not related - see below).

I have successfully built Ian's 3.6 and 3.7b sources before.

According to a hint from Ned that I found I explicitly used aclocal-1.7
instead of just aclocal - that made it work. But I had to manually hack
/usr/bin/aclocal-1.7 so that it doesn't try to call find_configure_ac,
because I have no idea where that function is! Anyway, this made it work
with autoconf 2.59 and automake 1.8.5, which are the latest releases
AFAIK.

But... my VM src is from a 3.7b-5969, with this essentially just latest
VMMaker and OSProcess installed from SM. When it compiles the VM itself
it barfs at the "fptr" like this:

	gnu-interp.c:741: error: syntax error before "const"

I am using gcc 3.3.3. Not sure how to handle this, any hints
appreciated.

regards, Göran

PS. Still not sure what status we have on the SF HEAD vs SF-ned-branch
vs Ian's source tarballs. We really, really ought to try to get this
settled in a good way. Since we are now (I assume) going to do the move
from SF to SqF it seems like a good time to get this in "order". :)

[Squeak-VMdev] Re: CVS? SF? cvs.squeakfoundation.org? Confusion! :)

[<gor...@bl...>]

Hi VM hackers and all!

Ned Konz <ne...@sq...> wrote:
> On Monday 05 July 2004 12:56 pm, gor...@bl... wrote:
> > After using Ian's 3.7b-5 sources (works kinda fine) I decided to instead
> > use the CVS sources from SF. But I kept getting a weird error on
> > checkout - which apparently others get too (permission denied about
> > creating some /tmp/cvs-yaddayadda directory - on the server that is).
> 
> It works OK if you use your SF login and don't use the anon CVS.
> 
> That is:
> 	cvs -d :ext:ne...@cv...:/cvsroot/squeak -z3 co squeak

Ok.

> > And then Ken, Cees and I got chatting on IRC about the CVS move to sqf.
> > And Cees said - he has rigged CVS. Fine! So I thought - perhaps we
> > should take the chance and make the move.
> 
> Sure. It's clear that SF doesn't want us.
> 
> > And more importantly - do people KNOW? :)
[about the fact that the contents at SF had already been copied over] 
> I didn't.
> 
> > So if anyone has any good answers to the above - please enlighten us. IF
> > the move has been done properly - then we need to shut down SF ASAP -
> > because otherwise we end up in a MESS.
> 
> We are already:
[Ned posted some diffs showing that SF has already newer commits since
the copy was made]

Sigh... I do wonder when and who did the move it was of course helpful -
but no real point in doing it unless we all decide at the same time and
shut down SF too. :)

Ok, I think we should re-make the move. Anyone against? :) I am guessing
not.

The move that had been done seemed like a proper one (full history
preserved etc) so it would be nice if we could just redo it. :)

Anyone up to it? I am not sure about SSH access, etc - it was a long
while since I fiddled on SF.

regards, Göran

PS. I am on IRC, more or less all week. :)

Re: [Squeak-VMdev] Newest VMMaker to try out

[Tim R. <ti...@su...>]

> The first work that we (mainly Ian right now) are doing is to
> generalize them kernel interpreter so that it can run any combination
> of 32- or 64-bit image on 32- or 64-bit machine.  This means being
> careful about the types of pointers, small integers and other
> quantities such as bytes, 16-, 32- and 64-bit ints.
>
It would be kind of useful to know just what changes are being
proposed here. Having worked on the previous generation change from 16
to 32 bits some eighteen years ago I might even have some useful
suggestions and experience.

tim
--
Tim Rowledge, ti...@su..., http://sumeru.stanford.edu/tim
Death is a nonmaskable interrupt.

Re: [Squeak-VMdev] Newest VMMaker to try out

[Tim R. <ti...@su...>]

In message <9DE...@sm...>
          John M McIntosh <jo...@sm...> wrote:

> Hi, when I saw this, what happen was that os-x decided to load the 1GB  
> image at the 2GB boundary, versus the 1GB or earlier location.
> This was a choice made by the OS, which then caused immediate failure.
Likewise - startofmemory was &7xxxxxxxx and endofmemory was well into
&8xxxxxxxxx territory. The good news is that the image segment loading
is the only thing I've noticed failing because of that, though it's not
impossible that occasional mysterious crashes are due to the problem.
As with OSX, it's an OS memory allocation choice that isn't
controllable by the application.

tim
--
Tim Rowledge, ti...@su..., http://sumeru.stanford.edu/tim
It said, "Insert disk #3," but only two will fit!

Re: [Squeak-VMdev] Newest VMMaker to try out

[John M M. <jo...@sm...>]

Hi, when I saw this, what happen was that os-x decided to load the 1GB  
image at the 2GB boundary, versus the 1GB or earlier location.
This was a choice made by the OS, which then caused immediate failure.

On May 21, 2004, at 10:26 AM, Dan Ingalls wrote:

>> How's the 64bit stuff going? Any changes that are worth putting into
>> general release VMMaker?
>
> Not at this point.  There will be a huge pile after a while.  I hope  
> to put out a general update in a week or 10 days.
>
>> I got bitten by the oop>2gb problem this week. Your(?) code for
>> checking for acceptable classes when loading an image segment goes  
>> nuts
>> if fed an oop it thinks is -ve. Don't suppose you've done any work  
>> that
>> might fix some of these places? Hate to waste time duplicating work.
>
> The first work that we (mainly Ian right now) are doing is to  
> generalize the kernel interpreter so that it can run any combination  
> of 32- or 64-bit image on 32- or 64-bit machine.  This means being  
> careful about the types of pointers, small integers and other  
> quantities such as bytes, 16-, 32- and 64-bit ints.
>
> Other than through general code review, we would not find such a  
> problem as that to which you allude, as it is particular to images  
> that are close to their maximum oop size.  Your 32-bit image is a  
> valuable example, and it's not likely that we'll find a 64-bit image  
> with this problem for a while.
>
> Therefore:  please *do* look into fixing it or forwarding it to me for  
> attention, and we'll make sure to incorporate it with out other  
> changes.
>
> Thanks
> 	- Dan
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: Oracle 10g
> Get certified on the hottest thing ever to hit the market... Oracle  
> 10g.
> Take an Oracle 10g class now, and we'll give you the exam FREE.
> http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
> _______________________________________________
> Squeak-VMdev mailing list
> Squ...@li...
> https://lists.sourceforge.net/lists/listinfo/squeak-vmdev
>
>
--
======================================================================== 
===
John M. McIntosh <jo...@sm...> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
======================================================================== 
===

Re: [Squeak-VMdev] Newest VMMaker to try out

[Dan I. <Da...@Sq...>]

>How's the 64bit stuff going? Any changes that are worth putting into
>general release VMMaker?

Not at this point.  There will be a huge pile after a while.  I hope to put out a general update in a week or 10 days.

>I got bitten by the oop>2gb problem this week. Your(?) code for
>checking for acceptable classes when loading an image segment goes nuts
>if fed an oop it thinks is -ve. Don't suppose you've done any work that
>might fix some of these places? Hate to waste time duplicating work.

The first work that we (mainly Ian right now) are doing is to generalize the kernel interpreter so that it can run any combination of 32- or 64-bit image on 32- or 64-bit machine.  This means being careful about the types of pointers, small integers and other quantities such as bytes, 16-, 32- and 64-bit ints.

Other than through general code review, we would not find such a problem as that to which you allude, as it is particular to images that are close to their maximum oop size.  Your 32-bit image is a valuable example, and it's not likely that we'll find a 64-bit image with this problem for a while.

Therefore:  please *do* look into fixing it or forwarding it to me for attention, and we'll make sure to incorporate it with out other changes.

Thanks
	- Dan

[Squeak-VMdev] Historical VMs

[Andreas R. <and...@gm...>]

Hi Guys,

You might have seen (or not - it seems like squeak-dev is down) that I have
started to assemble a history of Squeak at

    http://isgwww.cs.uni-magdeburg.de/~raab/squeak/history

However, I am lacking VMs for the various versions of Squeak. I'd appreciate
it if some of you could drop me VMs that are capable of running the "major
versions" of Squeak - e.g., *one* VM for all of the 1.x images, *one* VM for
all of the 2.x images and *one* VM for all of the 3.x images. I'd like to
complete the history by providing precompiled VMs for most of the major
platforms just so people can play with it.

So if you have old VMs that are known to work well with the above please
drop me a note with link so that I can integrate them into the above.

Oh, and one more "wish": If possible, *please* try to use "original VMs" as
far as possible - I'd like the history to be as authentic as possible so if
you happen to have a "historic VM" I'd prefer that one over one which was
"compiled today" based on the historical sources.

Thanks,
  - Andreas

[Squeak-VMdev] Newest VMMaker to try out

[Tim R. <ti...@su...>]

Dudes, dudettes, dunces and other interested sophonts -
there is a new version of VMMaker that I hope will be suitable for the
final 3.7 release. This latest version stores the primitive function
addresses in the MCache so that calling a prim within
#internalExecuteNewMethod can branch pretty directly instead of a
switch statement. It's not so much a performance thing (yet) as a
flexibility thing, since we can alter #functionPointerFor:inClass: to
specialise the chosen prim to suit the class at some later stage.

The not-so-inline method activations still go via the table of prim
function addresses for now. I'd be interested to know if anyone can see
any reason or way to use the MCache entry in those cases.

It's been tested on RISC OS & winXP and appears to be reliable - not to
mention 2.2 times faster on RISC OS than the 3.6 VMs. Sadly the winXP
results only show about 5% improvement but why would I care about that
:-)

I still don't have a decent way to upload to cvs so you'll have to use
the attached copy of sq.h - or upload it for everyone's convenience -
in order to get a good build. The new #define for
dispatchFunctionPointer is the only change from thelast version.

tim
--
Tim Rowledge, ti...@su..., http://sumeru.stanford.edu/tim
Strange OpCodes: DSO: Do Something or Other

[Squeak-VMdev] security plugin issues

[Tim R. <ti...@su...>]

I finally got to implementing the RISC OS security plugin; I'm sure
you're all greatly relieved to hear that...

I'm puzzled to see that the Mac, Win & unix VMs all directly call
ioInitSecurity() from the core VM - not to mention surprised that they
_can_ call it like that. It's called by the module init code anyway so
it's quite redundant and unless it is a fluke of having the plugin
internal I can't see why the linker doesn't get upset.

On RISC OS at least it all works just fine as an external plugin so I'm
wondering at the reason for the direct call.

tim
--
Tim Rowledge, ti...@su..., http://sumeru.stanford.edu/tim
It said, "Insert disk #3," but only two will fit!

Re: [Squeak-VMdev] New Object Header Proposal

[John M M. <jo...@sm...>]

Dan I did notice, mind we've lots of bits, but that in #2 you've

> 	9      indexable 64-bit fields
> 	10      indexable 32-bit fields
> 	11      indexable 16-bit fields
> 	12      indexable 8-bit fields

which could be 2 bits where

00	indexable 8-bit fields
01	indexable 16-bit fields
10	indexable 32-bit fields
11	indexable 64-bit fields


On Apr 13, 2004, at 10:25 PM, Dan Ingalls wrote:

--
======================================================================== 
===
John M. McIntosh <jo...@sm...> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
======================================================================== 
===

[Squeak-VMdev] New Object Header Proposal

[Dan I. <Da...@Sq...>]

Folks -

I've been scrutinizing our old object header format, contemplating 64-bit variants, speculating on the demise of compact classes, and thinking about object header types.

I've come up with a proposal that I believe maximizes compatibility with 64-bit images, and minimizes the number of changes that need to be made in the code.  You will also find a variant proposal and some discussion at the end.

Please look it over and tell me what you think.  Before proposing any radical departures, please consider them from the standpoint of your having to write all the new code and get it to work flawlessly.

Thanks
	- Dan
----------------------------------------

As a refresher course, here's the current object header format, as documented in class ObjectMemory:
	3 bits	reserved for gc (mark, old, dirty)
	12 bits	object hash (for HashSets)
	5 bits	compact class index
	4 bits	object format
	6 bits	object size in 32-bit words
	2 bits	header type (0: 3-word, 1: 2-word, 2: forbidden, 3: 1-word)

and here is the current encoding of the "object format" field:
	0      no fields
	1      fixed fields only (all containing pointers)
	2      indexable fields only (all containing pointers)
	3      both fixed and indexable fields (all containing pointers)
	4      both fixed and indexable weak fields (all containing pointers).
	5      unused
	6      indexable word fields only (no pointers)
	7      unused
 
	8-11      indexable byte fields only (no pointers) (low 2 bits are low 2 bits of size)
	12-15     compiled methods:
	     # of literal oops specified in method header,
	     followed by indexable bytes (same interpretation of low 2 bits as above)


Here is Proposal #1 (my preference) for Version 4...

Memory management bits (high 3 bits)
In actual fact, we currently only use the top two bits.  I propose keeping these as they are, and reserving the third one for immutability.

Header type bits
With the demise of compact classes, header type zero goes away.  Currently the garbage collector uses type 2.  My proposal is to leave these as is, except to relabel type 2 as gcOnly, and type 3 as forbidden.  I thought of changing the order of 0 and 1 for convenient logic, but I don't think this matters and it all works right now.

Object size
This field, with low bits masked off, gives the object's size in host memory words, whether 32- or 64-bit words.  The value is a don't-care for objects with large headers, but it will be given a value of zero in that case.

Object format
My proposal here is to add a bit to this field, and slightly reorganize it as follows:
	0      no fields -- actually unused
	1      fixed fields only (all containing pointers)
	2      indexable fields only (all containing pointers)
	3      both fixed and indexable fields (all containing pointers)
	4      both fixed and indexable weak fields (all containing pointers).
	5      unused
	6      unused
	7      unused

	8      unused
	9      indexable 64-bit fields only (no pointers)
	10-11      indexable 32-bit fields only (no pointers) (low 0-1 bits are low 0-1 bits of size)
	12-15      indexable 16-bit fields only (no pointers) (low 1-2 bits are low 1-2 bits of size)
	16-23      indexable 8-bit fields only (no pointers) (low 2-3 bits are low 2-3 bits of size)
	24-31     compiled methods:  [becomes unused with NCMs]
	     # of literal oops specified in method header,
	     followed by indexable bytes (same interpretation of low 2-3 bits as above)

Here's my thinking:  We need an extra bit on all the non-pointer arrays when we go to 64 bits.  The current logic is fairly simple and solid, so let's just add a bit to it where needed.  Once we do this, there is the possibility of cleaning up the hacks that are currently used for arrays of 16-bit objects, and doing a reasonably uniform thing for 32-bit non-pointer arrays in a 64-bit image.

[note there may be no real value to cleaning up 16-bit values at this point, especially since the current hacks work with signed values -- IIABDFI]

Compact class index
...goes away.

Object hash
With the demise of compact classes, we have 5 bits to redistribute.  Since we already took one for the format, we have 4 left to add to the object hash, bringing it up to 16 bits.

64-bit images
Here the format would be almost identical, except we would stretch the hash field by 14 bits to 30 bits (my choice: highest positive smallInt value), and stretch the object size field by 18 bits, thus all but eliminating 3-word headers in the 64-bit world.


Proposal #2
This is an alternative that I have seriously considered  because it puts all the size bits together.  I don't like it because it uses one more bit, and because it requires a shift of the word size.

Object size
Here we would combine all the size bits in one 8-bit field.  It would be encoded (as now) so that the high bits are the size in words and the low bits are the low bits of the size-1.  Thus you can mask to get the size in words, or add a constant to get the Squeak byte size.  Unfortunately, since the low bits are included, a right shift is required to get around the header type.

Object format
In this scheme, 4 bits is still plenty for format, since the low bits of size are elsewhere.
	0      no fields -- actually unused
	1      fixed fields only (all containing pointers)
	2      indexable fields only (all containing pointers)
	3      both fixed and indexable fields (all containing pointers)
	4      both fixed and indexable weak fields (all containing pointers).
	5      unused
	6      unused
	7      unused

	8      unused
	9      indexable 64-bit fields
	10      indexable 32-bit fields
	11      indexable 16-bit fields
	12      indexable 8-bit fields
	13     compiled methods:  [becomes unused with NCMs]
	     # of literal oops specified in method header,
	     followed by indexable bytes
	14   unused
	15   unused


Discussion
The problem with header type bits is that they have to be both on the word pointed to by the oop, and the first word encountered in a scan of memory.  Our current scheme is nice because it puts them where they can be masked out of the class word.  But that means that they are in the way if we want to integrate the size bits without having to shift them.

A somewhat radical change would be to put the type bits at the top of the word, and change the field order so that the class word followed the header word or words.  This would be a good thing, but it's another potentially large change.  Moreover the GC usage of the forbidden type as a flag would have to be addressed as well.  If this change had been done already, then Proposal #2 would become my preference.

Considering all the above, I am inclined to proceed with Proposal #1 with a willingness to adopt #2 if someone (possibly me) comes up with all the changes needed to make it simple.  It would definitely be nice to more the class word after the header regardless, because then memory scans would usually find the header word they want, rather than having to look a the next word every time (this was different with compact classes).

Re: [Squeak-VMdev] An object format proposal

[Anthony H. <ajh...@ya...>]

One thing I forgot is the isLarge flag has to be repeated in the -8
word, so when going to the next object you know if the header is two
or three words.  So here's the format again.

 -8	isLarge flag, n if large (this field does not exist if not large)
  4	isLarge flag, identityHash, other flags, isOops flag, n if small
  0	class pointer, gc flags (in low 2 bits)
  4	(for class objects only) primitive type, isBytes & isWeak flags, m
  ..
  n

isLarge can be the high bit or the low bit, it just has to be the same
in both words.  Also, the two low bits of class oop can be used for
flags, and I added "other" flags for immutable or whatever we decide.

Cheers,
Anthony


__________________________________
Do you Yahoo!?
Yahoo! Tax Center - File online by April 15th
http://taxes.yahoo.com/filing.html

Re: [Squeak-VMdev] An object format proposal

[Anthony H. <ajh...@ya...>]

Anthony Hannan <aj...@co...> wrote:
> n serves as the identity hash for variable structures (usually variable
> structures sit inside tuples so their identityHashes won't be used
> much).

I forgot about Symbols which are variable structures.  I would 
make Symbols tuples with one field pointing to its string, or with
internationalization the one field would probably be an index into a
String table.  Also, I think selectors someday might be more than just a
name, for example to contain type information (message signatures).

But there is another problem.  Become won't work between variable
structures of different sizes because there identity hashes would be
different (you can no longer substitute one hash for the other).

One solution to this problem is to simply declare that variable structures
don't have identity, ie. they can't be put into identity hash tables nor
can they be compared using == (only =).

Another solution is to change the proposed object format by either adding
a third header word for variable structures containing the identity hash,
or to narrow the size (n), so the first header word encodes 3 fields
instead of 2, ie. the size, the identity hash, and flag bits for gc (and
immutable, etc. if desired) and a flag to indicate if the variable
structure is so large that its size is stored in a third header word.

I like this last solution.  So to recap here is my complete object format
proposal #2:

 -8	n if large otherwise this field does not exist
 -4	identityHash, gc flags, isOops flag, isLarge flag, n if small
  0	class oop
  4	(for class objects only) primitive type, isBytes & isWeak flags, m
  ..
  n

primitive type is Code (00), Data (01), Array (10), or Tuple (11).
isOops is 1 for Arrays and Tuples, and 0 for Code and Data (redundant
with primitive type in class but might make garbage collection faster).
m is number of fixed fields for Arrays and Tuples, m = n for Tuples.

This proposal is actually not much different from the current format.
The difference is replacing object format with isOops, replacing header
type with isLarge, and removing the compact class index as already
suggested.

Cheers,
Anthony


__________________________________
Do you Yahoo!?
Yahoo! Tax Center - File online by April 15th
http://taxes.yahoo.com/filing.html

[Squeak-VMdev] An object format proposal

[Anthony H. <aj...@co...>]

What do you guys think of this simple object format.

Five primitive object types:  SmallInteger, Tuple, Array, Data, and
Code.  The last four are called structures, and the last three are
called variable structures.

Object encoding:  An oop is a word interpreted either as a SmallInteger
or a pointer to a structure, depending on its low bit (eg. 0 for
SmallInteger, 1 for pointer).  A structure is 2 + n contiguous words: 
The first word is a SmallInteger, the second word is a pointer to its
class, and the remaining n words are interpreted either as oops (for
Arrays and Tuples), machine code (for Codes), or raw data (for Datas). 
The SmallInteger in the first word encodes flags for gc, etc, plus
either n for variable structures or an identity hash for Tuples.  n is
the same for all tuples of the same class so n is stored in the class. 
n serves as the identity hash for variable structures (usually variable
structures sit inside tuples so their identityHashes won't be used
much).  Finally, a class is a tuple where the first field is a
SmallInteger that encodes its primitive type (Tuple, Array, Data, or
Code), flags like isBytes/isWeak, and n if its primitive type is Tuple
(or number of fixed fields if its primitive type is Array, although the
VM shouldn't care).

Cheers,
Anthony

Re: [Squeak-VMdev] Versiojn 4 changes

[Tim R. <ti...@su...>]

In message <014301c41dbb$6f406ec0$6688b8d9@R22>
          "Andreas Raab" <and...@gm...> wrote:

> Tim,
>
> The idea was that you would specify primitives with arguments (like FFI
> and/or SmartInterpreterplugin) and then glue gets generated for the
> interpreter. So it would actually look somewhat along the lines of:
OK, that would help prepare for a real translator and it might be
argued that it would make writing prims simpler but I can't see any
particular performance benefit in our current VM. I guess we could
generate wrappers that do extra stuff on occasion as well -perhaps
debugging? Of course, he wrappers need generating somehow. The CCodeGen
could probably create static C code ones with a bit of futzing; doing it
in machine code dynamically might be more than we want to do right now?

tim
--
Tim Rowledge, ti...@su..., http://sumeru.stanford.edu/tim
Oxymorons: Small crowd

Re: [Squeak-VMdev] SMACC [was: DING!]

[Marcus D. <de...@ac...>]

Am 09.04.2004 um 21:01 schrieb Anthony Hannan:

> Marcus Denker <de...@ac...> wrote:
>>
>> We need to really put some work into cleaning up the old compiler and
>> make everything use the new parsetrees.
>
> I think we should just get rid of the old compiler and use the closure
> compiler.

Yes, that's what I meant: clean up by throwing the old compiler away ;-)

     Marcus

--
Marcus Denker de...@ac...

Re: [Squeak-VMdev] SMACC [was: DING!]

[Anthony H. <aj...@co...>]

Marcus Denker <de...@ac...> wrote:
> 
> We need to really put some work into cleaning up the old compiler and 
> make everything use the new parsetrees.

I think we should just get rid of the old compiler and use the closure
compiler.

> The SmaCC-based compiler uses the RB Parsenodes. They are really nice 
> and a clear improvement, IMHO.
> But some changes are needed to make everthing use them. We need to make 
> both Slang and eToys to work
> with the new Parsenodes instead of the old ones. (I think Anthony 
> already did some work on Slang).

No, not really.  I just made a couple of fixes way back when.

Re: [Squeak-VMdev] SMACC [was: DING!]

[Anthony H. <aj...@co...>]

Dan Ingalls <Da...@Sq...> wrote:
> 
> 1.  I haven't played with SMACC.  I assume it's good.  I gather Marcus has also used this to do some fun work with Parse Trees.
> 
> 2.  Is there a pretty clean package that converts a current Squeak to use SMACC with decent reliability?
> 2a.  Is this a part of Anthony's Closure package?

Yes, see the closure compiler
(http://minnow.cc.gatech.edu/squeak/ClosureCompiler).  It does not
convert existing methods, instead both compilers live together with a
preference indicating which one to use.  Of course one can recompile
methods if desired.  Marcus Denker has added parts of the closure
compiler into 3.7, so it is in a state of flux so I don't think the
package loads correctly in 3.7.  But it does load into 3.6.

> 3.  Is there general agreement on this list that SMACC is the way to go?

I believe so.  And it worked fine for me.

Cheers,
Anthony

Re: [Squeak-VMdev] SMACC [was: DING!]

[Marcus D. <de...@ac...>]

Am 09.04.2004 um 19:00 schrieb Dan Ingalls:

>> [Anthony's closure work]
>>> Is this a modified St-80 Compiler, or one based on SmaCC?
>>
>> SmaCC.
>
> Hi, Guys -
>
> Just anticipating the release process...
>
> 1.  I haven't played with SMACC.  I assume it's good.  I gather Marcus 
> has also used this to do some fun work with Parse Trees.
>
I did some experiments regarding scripting syntax last year. That is, 
my Squeak accept happily a method like

function testForIn2() {
		 var a,c,i;
		 a = new OrderedCollection;
		 a.add(1);
		 a.add(2);
		 c = 0;
		 for (i in a) { c += i; }
		 this.assert(c == 3);
	  }

or

to exampleTurtle
	repeat 36 [repeat 4 [
		fd 90
		rt 90
	] rt 10 ]
end

or

def testWhile():
	a = 1
	while a < 10:
		a = a + 1
		b = a
	self.assert(b == 10)


> 2.  Is there a pretty clean package that converts a current Squeak to 
> use SMACC with decent reliability?
>
Yes. Anthony provides all the changesets needed.

I used this while hacking on AOStA, no problems so far.

The new compiler is installed in a way that it can be disabled via a 
preference.

We need to really put some work into cleaning up the old compiler and 
make everything use
the new parsetrees.

The SmaCC-based compiler uses the RB Parsenodes. They are really nice 
and a clear improvement, IMHO.
But some changes are needed to make everthing use them. We need to make 
both Slang and eToys to work
with the new Parsenodes instead of the old ones. (I think Anthony 
already did some work on Slang).

> 2a.  Is this a part of Anthony's Closure package?
>
Yes.
Parts of the closure package already were added to 3.7 (e.g. the 
IRBuilder). I need to make a
package with all the rest.

> 3.  Is there general agreement on this list that SMACC is the way to 
> go?
>
I vote for this.

> ... If so, then I want to work with the Guides to get SMACC into 
> 3.8alpha asap, so that it's not an added cause of confusion later, 
> when we merge the other (lower-level) V4 stuff with 3.8.
>
Ok. I'l help with that.

            Marcus

--
Marcus Denker de...@ac...

Re: [Squeak-VMdev] DING! End of Round One

[Dan I. <Da...@Sq...>]

>>This is another class I ought to have a filein for but damned if I can
>>find it.  Still, not a large class IIRC. The only trick will be
>>substituting it for ordinary Rectangle in the tracer but we really
>>ought to be able to handle that ok.

Bert asked...

>Why wouldn't this work with a normal changeset? Does the VM know about Rectangles?

Ooh.  You're right, Bert.  I forgot that this came up back when John and I were working on the original Morphic, and we wanted rectangle intersection to run faster.

It would be nice to change BitBlt correspondingly to save time and objects (we'd have to make temporary points for it after the change), but yes, I believe the whole thing can be done with a changeSet or two.

This may be a mere distraction from the more important things on the table.

	- Dan