sml-implementers — Standard ML implementers

RE: [Sml-implementers] compilation management

[Andrew K. <ak...@mi...>]

I totally agree. Sorry I wasn't clearer in my message: I was
just offering a description of our own compilation model as
another *example* not as a proposal for adoption. I like Bob's
analogy with type inference - we need a common means of expressing
each compiler's particular model.

- Andrew.

> -----Original Message-----
> From: Robert Harper [mailto:Rob...@cs...]=20
> Sent: Wednesday, September 19, 2001 10:28 PM
> To: 'Matthias Blume'; Andrew Kennedy
> Cc: Daniel C. Wang; Stephen Weeks;=20
> sml...@li...
> Subject: RE: [Sml-implementers] compilation management
>=20
>=20
> Just to be sure we're on the same page, let me say that I=20
> would not want to standardize on a TOOL, but rather a=20
> LANGUAGE for expressing separate compilation.  In particular,=20
> saying that CM could be made to generate one huge source file=20
> for those of us that don't implement CM is a non-starter.
>=20
> An anology: CM is like type inference; we want to specify the=20
> underlying explicitly typed syntax.
>=20
> We will send a brief summary of our compilation mechanism to=20
> this list shortly.
>=20
> Bob
>=20
> -----Original Message-----
> From: Matthias Blume [mailto:bl...@re...]
> Sent: Wednesday, September 19, 2001 11:09 AM
> To: Andrew Kennedy
> Cc: Daniel C. Wang; Stephen Weeks;=20
> sml...@li...
> Subject: Re: [Sml-implementers] compilation management
>=20
>=20
> Andrew Kennedy wrote:
> >=20
> > (A) Agree on some way to map top-level SML Module identifiers (for=20
> > structures, functors and signatures) to full file names identifying=20
> > the file that contains the single binding for that Module entity=20
> > (structure, functor or signature).
>=20
> Sorry.  I will NEVER agree to that!
>=20
> We had this discussion before in an OCaml vs. SML context.  A=20
> naming convention such as the above can be used by an=20
> implementation, but it should _not_ be the common ground on=20
> which we all live.  It is trivial to map implicit file naming=20
> such as the one you suggest to explicit naming, but not vice versa.
>=20
> As I said, I will try to come up with a simple (although=20
> perhaps verbose) and very explicit description format that we=20
> all can implement.  Systems such as CM or your implementation=20
> can take whatever scheme they use and _generate_
> the explicit format.   It is unlikely that we will ever agree on a
> high-level
> format (I for one will never agree to a modulename->filename=20
> mapping scheme, and others seem to think that such as scheme=20
> is the only way they can accept), so abstracting from this=20
> issue is the only way to go.
>=20
> Matthias
>=20
> _______________________________________________
> Sml-implementers mailing list Sml...@li...
> https://lists.sourceforge.net/lists/listinfo/sml-implementers
>=20

Re: [Sml-implementers] compilation management

[John H. R. <jh...@re...>]

In message <C70...@EX...>, Robert 
Harper writes:
>
> Just to be sure we're on the same page, let me say that I would not want to
> standardize on a TOOL, but rather a LANGUAGE for expressing separate
> compilation.  In particular, saying that CM could be made to generate one
> huge source file for those of us that don't implement CM is a non-starter.
> 
> An anology: CM is like type inference; we want to specify the underlying
> explicitly typed syntax.

This is what Matthias is proposing.  An explicit syntax for describing
how to assemble libraries/applications from sources.  CM can be used to
generate such a description from a list of sources.

	- John

RE: [Sml-implementers] compilation management

[Robert H. <Rob...@cs...>]

Just to be sure we're on the same page, let me say that I would not want to
standardize on a TOOL, but rather a LANGUAGE for expressing separate
compilation.  In particular, saying that CM could be made to generate one
huge source file for those of us that don't implement CM is a non-starter.

An anology: CM is like type inference; we want to specify the underlying
explicitly typed syntax.

We will send a brief summary of our compilation mechanism to this list
shortly.

Bob

-----Original Message-----
From: Matthias Blume [mailto:bl...@re...]
Sent: Wednesday, September 19, 2001 11:09 AM
To: Andrew Kennedy
Cc: Daniel C. Wang; Stephen Weeks;
sml...@li...
Subject: Re: [Sml-implementers] compilation management


Andrew Kennedy wrote:
> 
> (A) Agree on some way to map top-level SML Module identifiers (for
> structures, functors and signatures) to full file names identifying the
> file that contains the single binding for that Module entity (structure,
> functor or signature).

Sorry.  I will NEVER agree to that!

We had this discussion before in an OCaml vs. SML context.  A naming
convention
such as the above can be used by an implementation, but it should _not_ be
the common ground on which we all live.  It is trivial to map implicit file
naming such as the one you suggest to explicit naming, but not vice versa.

As I said, I will try to come up with a simple (although perhaps verbose)
and very explicit description format that we all can implement.  Systems
such
as CM or your implementation can take whatever scheme they use and
_generate_
the explicit format.   It is unlikely that we will ever agree on a
high-level
format (I for one will never agree to a modulename->filename mapping scheme,
and others seem to think that such as scheme is the only way they can
accept),
so abstracting from this issue is the only way to go.

Matthias

_______________________________________________
Sml-implementers mailing list
Sml...@li...
https://lists.sourceforge.net/lists/listinfo/sml-implementers

Re: [Sml-implementers] compilation management

[Ken F. L. <kf...@it...>]

Stephen Weeks <sw...@in...> writes:
> I would be interested to hear from other groups (Moscow ML, Poly/ML, ML Kit,
> TILT) about the compilation management system they use.

Let me reply about what Moscow ML and (partly) ML Kit use.

Moscow ML traditionally uses ordinary make and the distribution
includes a simpleminded dependency analysis tool.  But if you are
willing to commit to GNU make it is possible to make a generic
Makefile that will "just work" in most projects (it does take quite a
bit of make hacking).

I dislike make as much as any other sane person.  But it is standard
tool that many programmers know how to use.  And it is easy to
integrate components writen in another programming language such as C.

But Moscow ML 2.00 also came with mosmlpm[1], which is a project manager
that accepts the same project files (PM files) as the ML Kit (almost).
PM files are quite simple, the grammar is:

   pm ::=  "import" imports "in" body "end" 
       |   body

   imports ::= path.pm imports
            |  empty
  
   body ::=  path.sml body
         |   path.sig body
         |   "local" body "in" body "end" body
         |   empty

Where path is a unix-style path (much to Doug Currie's grief ;-), and
we also allow SML-style comments.  Example:

  import Ast.pm Lexer.pm Parser.pm in
     Typechecker.sml
     local Optimizer.sml in
       Compiler.sml
     end
  end

I like PM because it is *simple* and it is simple to explain what the
semantics of a PM file is (even with respect to side-effects!).  And
it is simple to implement.

I think PM is a good candidate for a common exchange format.  The only
flows I know of is:

  * No way to specify compiler options for a specific file (or a whole
    .pm file).  My suggestion is to extend the format to allow
    something like:

        | RedBlack.sml [mosml : "-orthodox", mlkit : "-inline 40 -fix-reduce"]

  * No way to make a big library pm file that exports other pm files
    (for example for the Basis Lib).


Martin Elsmann is the designer behind PM and I'll let him correct any
mistake I've made in the description above.


Cheers,

--Ken

[1] Unfortunately is mosmlpm still unannounced and a bit hard to find.
But those who are interested can find a quick prototype in
mosml/src/mosmlpm in the Moscow ML distribution.

Re: [Sml-implementers] compilation management

[Doug C. <e...@fl...>]

At 10:18 AM -0700 9/19/01, Stephen Weeks wrote:
>I would be interested to hear from other groups (Moscow ML, Poly/ML, ML Kit,
>TILT) about the compilation management system they use.

Porting Moscow ML 1.x to the Mac was hindered by the use of UNIX make 
files in the distribution. Ultimately I made several additions and 
changes to the Mac version of Moscow ML to enable full bootstrapping 
capability without external tools. I integrated the Moscow ML 
dependency generator and linker into the toplevel, along with a new 
make function that uses the dependencies to compute a compile 
ordering, and use file modification times to implement conditional 
compilation. The make function processes a directory at a time, with 
additional directory arguments as a search path for dependencies. 
This directory at a time approach follows from the organization of 
the distribution. With the new functions, Mac Moscow ML can rebuild 
itself from sources in less than a page of code.

Integrating the compile management and link mechanisms into the 
toplevel required very close coordination. There is a lot of state 
maintained by each of these pieces (toplevel, compiler, and linker) 
that needs to be kept independently and dynamically wound in and out 
as each of the pieces is invoked. Nevertheless, I found the end 
result invaluable in maintaining Moscow ML on a platform without the 
standard UNIX style development environment.

The tools in Moscow ML 1.x enforced a one to one correspondence 
between signatures (structures) and identically named .sig (.sml) 
files. This greatly simplified the dependency generator and linker.

More recent versions of Moscow ML (>= 2.0) support a "structure" mode 
wherein this correspondence is broken. We have experimented with a 
compilation manager that takes the "recursive list of files" 
approach, and a separate tool to discover the inter-file dependencies 
and compute these lists. This compilation manager abstracted the 
interface to the compiler; there are at least two concrete 
implementations: one uses the "system" call to invoke the compiler, 
and the other uses a compiler integrated into the toplevel, as with 
the Mac make function.

e

Re: [Sml-implementers] compilation management

[Stephen W. <sw...@in...>]

I would be interested to hear from other groups (Moscow ML, Poly/ML, ML Kit,
TILT) about the compilation management system they use.

Re: [Sml-implementers] compilation management

[John H. R. <jh...@re...>]

In message <BCD...@re...rp.microso
ft.com>, "Andrew Kennedy" writes:
>
> Our model consists of the following:
> 
> (A) Agree on some way to map top-level SML Module identifiers (for 
> structures, functors and signatures) to full file names identifying the 
> file that contains the single binding for that Module entity (structure,
> functor or signature).

I'm strongly opposed to requiring a fixed mapping from module names to
file names.  But it isn't necessary to have such a requirement on the
interchange format to support implementations that impose such a requirement.
If we have a mapping from module names to source files in the project
file, a simple script can generate a symbolic link "RedBlackSetFn" to
"red-black-set-fn.sml".  Of course, there are broken operating systems
that don't have symbolic links, but renaming would work for those cases.

I think that it is important the the interchange format not impose
unnecessary restrictions on library and application authors.  It should
be designed to make it as easy as possible to port existing code to
use the format.

	- John

Re: [Sml-implementers] compilation management

[Daniel C. W. <dc...@ag...>]

At 10:27 AM 9/19/2001 -0400, Matthias Blume wrote:
{stuff deleted}

>Obviously, I have a natural bias in favor of CM. :-)  But I actually agree with
>Steve (and others) to some degree.  Getting everyone to adopt CM is a bit too
>ambitious, I think.  More realistic is a simple exchange format that specifies
>precisely which files get compiled in the context of what bindings imported from
>where.  I have ideas as to what this format could look like.  As you might imagine,
>CM internally computes all this information anyway, so dumping it into an
>ASCII file will not be hard at all.
>
>It is my plan to design a low-level format that I believe all SML implementors could
>easily support without having to do fancy things such as dependency analysis etc.
>The format will also provide the necessary information for the transformation into
>pure SML that we described in our TOPLAS paper (where it was also used to define the
>dynamic semantics of CM).

I guess there are two issues.
 1. A portable low-level build script mechanism that is easy for any 
    implementation to adopt, whose goal is to reliably reproduce a build. 
    It really should not be too much fancier than "use" with 
    symbol import and export info. It would be useful if these build scripts 
    can be composed in a limited way like sources.cm files can be.

 2. A standard way to generate these "build-scripts" from source files.

The answer to question 1, should be easy to agree on.

For 2, the long run answer should be CM, because I see no reason to adopt
any other model when the CM model. It is quite well tested and superior to
all the existing approaches in all the existing ML implementations, from
a user standpoint. 

I feel #1 is only a stop-gap until CM becomes a standalone SML program that
can generates these build scripts for all implementations. At that point
the build-script language is just some silly intermediate form that end
users should be completely oblivious to.

In the not to distant future, I hope their is one user visible build mechanism
and it should look a lot like CM. Point 1 should be addressed first  
but at the end of the day point 2 is the only one that matters to end users.



 

Re: [Sml-implementers] compilation management

[Matthias B. <bl...@re...>]

Andrew Kennedy wrote:
> 
> (A) Agree on some way to map top-level SML Module identifiers (for
> structures, functors and signatures) to full file names identifying the
> file that contains the single binding for that Module entity (structure,
> functor or signature).

Sorry.  I will NEVER agree to that!

We had this discussion before in an OCaml vs. SML context.  A naming convention
such as the above can be used by an implementation, but it should _not_ be
the common ground on which we all live.  It is trivial to map implicit file
naming such as the one you suggest to explicit naming, but not vice versa.

As I said, I will try to come up with a simple (although perhaps verbose)
and very explicit description format that we all can implement.  Systems such
as CM or your implementation can take whatever scheme they use and _generate_
the explicit format.   It is unlikely that we will ever agree on a high-level
format (I for one will never agree to a modulename->filename mapping scheme,
and others seem to think that such as scheme is the only way they can accept),
so abstracting from this issue is the only way to go.

Matthias

RE: [Sml-implementers] compilation management

[Andrew K. <ak...@mi...>]

For comparison purposes I present below the compilation model we use=20
in MLj and SML.NET. As new versions of both these compilers are
currently=20
unavailable I guess that not many people are familiar with our model.

Personally I dislike even having to list explicitly the files making up
a
project, though I recognise that for large projects the module-level=20
namespace management and library support etc that CM provides are very=20
valuable.

Our model consists of the following:

(A) Agree on some way to map top-level SML Module identifiers (for=20
structures, functors and signatures) to full file names identifying the=20
file that contains the single binding for that Module entity (structure,

functor or signature).

(B) Supply a set of "root" SML structure identifiers.

(C) Let the SML compiler figure out the rest i.e. do sufficient parsing,
mapping of top-level identifiers to filenames, and dependency analysis
to figure out which files to compile and in what order.

For simple projects whose files all reside in a single directory and
whose
names conform to standard conventions, all the user needs to do is type=20
"make Main" (or whatever) and the compiler does the rest. For more
complex
projects the mapping (A) breaks down into

(1) A PATH-like list of directories.
(2) A suffix convention (.sig for signatures, .sml for structures and
functors).
(3) A means of overriding the convention to map particular identifiers
to
particular files.
(4) A further means of overriding the convention to supply file names
that may
contain multiple top-level module bindings (as CM allows).

We have successfully applied the scheme to the compiler itself, the most
unpleasant
aspect being the use of SML/NJ libraries whose files have an informal
naming=20
convention which could very nearly be formalized but which currently
must be mapped=20
explicitly by the programmer e.g. IntRedBlackSet |->
int-redblack-set.sml.

- Andrew.

> -----Original Message-----
> From: Matthias Blume [mailto:bl...@re...]=20
> Sent: Wednesday, September 19, 2001 3:28 PM
> To: Daniel C. Wang
> Cc: Stephen Weeks; sml...@li...
> Subject: Re: [Sml-implementers] compilation management
>=20
>=20
> "Daniel C. Wang" wrote:
> >=20
> > At 05:01 PM 9/18/2001 -0700, Stephen Weeks wrote:
> > >{stuff deleted}
> > >Here is my main requirement for a standard compilation management=20
> > >system that specifies how to build applications:
> > >
> > >>From a description of an application, it should be easy=20
> to extract=20
> > >>an SML
> > >program consisting of all and only the modules that are needed for=20
> > >the application.  (This extraction may require renaming=20
> modules.)  In=20
> > >fact, that is one way to define the dynamic semantics of the=20
> > >compilation management system -- by translating the description of=20
> > >the application into an SML program.
> > >
> > >By "easy", I mean that the extraction should be possible without=20
> > >looking at any SML sources, only by looking at the application and=20
> > >library description files.
> >=20
> > Let's just keep things simple, and get everyone to adopt CM. I've=20
> > talked to Matthias in the past about having a "dump=20
> everything to one=20
> > big SML file" option for CM. It's not that hard to do. Making CM=20
> > independent of the underlying SML/NJ compiler will probably=20
> take some=20
> > more work, but much less work than trying to reinvent yet another=20
> > compilation system.
> >=20
> > This suggestion violates Steven's "easy" requirement.=20
> However, I think=20
> > that by making things "easy" for the tool it must make=20
> things harder=20
> > for the user. I'm happy as things are now.. where I just=20
> list a bunch=20
> > of SML files in any old order in my sources.cm and type CM.make().
>=20
> Obviously, I have a natural bias in favor of CM. :-)  But I=20
> actually agree with Steve (and others) to some degree. =20
> Getting everyone to adopt CM is a bit too ambitious, I think.=20
>  More realistic is a simple exchange format that specifies=20
> precisely which files get compiled in the context of what=20
> bindings imported from where.  I have ideas as to what this=20
> format could look like.  As you might imagine, CM internally=20
> computes all this information anyway, so dumping it into an=20
> ASCII file will not be hard at all.
>=20
> It is my plan to design a low-level format that I believe all=20
> SML implementors could easily support without having to do=20
> fancy things such as dependency analysis etc. The format will=20
> also provide the necessary information for the transformation=20
> into pure SML that we described in our TOPLAS paper (where it=20
> was also used to define the dynamic semantics of CM).
>=20
> > Even for very larger systems, I see no advantage to having the user=20
> > provide any more information than a list of files and perhaps an=20
> > export list of identifiers.
>=20
> Right.  I see no reason to give up CM or its model in SML/NJ.=20
>  But I can modify its implementation in such a way that a=20
> more explicit low-level interchange format can be produced=20
> (and even used internally) by it.
>=20
> > I am a CM bigot... so perhaps there are some strong=20
> concrete technical=20
> > reasons I'm ignoring... but unless there are I don't see any reason=20
> > not to just adopt CM*.
>=20
> I am all for adopting CM.  But the current implementation=20
> will probably be hard to port because it "knows" an awful lot=20
> about SML/NJ.  Plus, the bulk of the code is not concerned=20
> with simple dependency analysis anyway.  (I can imagine=20
> ripping out the dependency analyzer, but that's on the order=20
> of less than 500 lines of code anyway.)
>=20
> Matthias
>=20
> _______________________________________________
> Sml-implementers mailing list Sml...@li...
> https://lists.sourceforge.net/lists/listinfo/sml-implementers
>=20

Re: [Sml-implementers] compilation management

[Matthias B. <bl...@re...>]

"Daniel C. Wang" wrote:
> 
> At 05:01 PM 9/18/2001 -0700, Stephen Weeks wrote:
> >{stuff deleted}
> >Here is my main requirement for a standard compilation management system that
> >specifies how to build applications:
> >
> >>From a description of an application, it should be easy to extract an SML
> >program consisting of all and only the modules that are needed for the
> >application.  (This extraction may require renaming modules.)  In fact, that is
> >one way to define the dynamic semantics of the compilation management system --
> >by translating the description of the application into an SML program.
> >
> >By "easy", I mean that the extraction should be possible without looking at any
> >SML sources, only by looking at the application and library description files.
> 
> Let's just keep things simple, and get everyone to adopt CM. I've talked to
> Matthias in the past about having a "dump everything to one big SML file"
> option for CM. It's not that hard to do. Making CM independent of the
> underlying SML/NJ compiler will probably take some more work, but much less
> work than trying to reinvent yet another compilation system.
> 
> This suggestion violates Steven's "easy" requirement. However, I think that
> by making things "easy" for the tool it must make things harder for the user.
> I'm happy as things are now.. where I just list a bunch of SML files in any
> old order in my sources.cm and type CM.make().

Obviously, I have a natural bias in favor of CM. :-)  But I actually agree with
Steve (and others) to some degree.  Getting everyone to adopt CM is a bit too
ambitious, I think.  More realistic is a simple exchange format that specifies
precisely which files get compiled in the context of what bindings imported from
where.  I have ideas as to what this format could look like.  As you might imagine,
CM internally computes all this information anyway, so dumping it into an
ASCII file will not be hard at all.

It is my plan to design a low-level format that I believe all SML implementors could
easily support without having to do fancy things such as dependency analysis etc.
The format will also provide the necessary information for the transformation into
pure SML that we described in our TOPLAS paper (where it was also used to define the
dynamic semantics of CM).

> Even for very larger systems, I see no advantage to having the user provide
> any more information than a list of files and perhaps an export list
> of identifiers.

Right.  I see no reason to give up CM or its model in SML/NJ.  But I can modify its
implementation in such a way that a more explicit low-level interchange format can
be produced (and even used internally) by it.

> I am a CM bigot... so perhaps there are some strong concrete technical reasons
> I'm ignoring... but unless there are I don't see any reason not to just adopt CM*.

I am all for adopting CM.  But the current implementation will probably be hard to port
because it "knows" an awful lot about SML/NJ.  Plus, the bulk of the code is not
concerned with simple dependency analysis anyway.  (I can imagine ripping out the
dependency analyzer, but that's on the order of less than 500 lines of code anyway.)

Matthias

Re: [Sml-implementers] compilation management

[Daniel C. W. <dc...@ag...>]

At 05:01 PM 9/18/2001 -0700, Stephen Weeks wrote:
>{stuff deleted}
>Here is my main requirement for a standard compilation management system that
>specifies how to build applications:
>
>>From a description of an application, it should be easy to extract an SML
>program consisting of all and only the modules that are needed for the
>application.  (This extraction may require renaming modules.)  In fact, that is
>one way to define the dynamic semantics of the compilation management system --
>by translating the description of the application into an SML program.
>
>By "easy", I mean that the extraction should be possible without looking at any
>SML sources, only by looking at the application and library description files.

Let's just keep things simple, and get everyone to adopt CM. I've talked to
Matthias in the past about having a "dump everything to one big SML file" 
option for CM. It's not that hard to do. Making CM independent of the 
underlying SML/NJ compiler will probably take some more work, but much less
work than trying to reinvent yet another compilation system.

This suggestion violates Steven's "easy" requirement. However, I think that 
by making things "easy" for the tool it must make things harder for the user.
I'm happy as things are now.. where I just list a bunch of SML files in any
old order in my sources.cm and type CM.make(). 

Even for very larger systems, I see no advantage to having the user provide 
any more information than a list of files and perhaps an export list 
of identifiers.

Anyway, as a first cut someone should take some version of CM and modify it so
that it compiles independently of SML/NJ and simply dumps one big SML source file.
Perhaps, later on we can figure out how to disentangle the current CM from SML/NJ
and have others integrate it into the guts of their systems.

I am a CM bigot... so perhaps there are some strong concrete technical reasons 
I'm ignoring... but unless there are I don't see any reason not to just adopt CM*.

*A question arises about which CM the "classic" version that ships with the stable 
SML release or the new CM. I think, it would be prudent to just stick with the
"classic" version as a first cut, and migrate to the "new" CM later on.

 

[Sml-implementers] compilation management

[Stephen W. <sw...@in...>]

I like Bob's division into four pieces.  To get the ball rolling on 1 ... I
agree with John when he says

> The important problem is to have a standard mechanism for specifying how to
> build libraries and applications from sources.

Here is my main requirement for a standard compilation management system that
specifies how to build applications:

From a description of an application, it should be easy to extract an SML
program consisting of all and only the modules that are needed for the
application.  (This extraction may require renaming modules.)  In fact, that is
one way to define the dynamic semantics of the compilation management system --
by translating the description of the application into an SML program.

By "easy", I mean that the extraction should be possible without looking at any
SML sources, only by looking at the application and library description files.

Re: [Sml-implementers] extensions to SML

[John H. R. <jh...@re...>]

In message <C70...@EX...>, Robert 
Harper writes:
>
...
> 
> 1. Standardization of a separate compilation mechanism.  This would entail
> defining what are compilation units, including how imports and exports are
> described, and what is the semantics of linking.  At a high level this
> should not be exceptionally hard to work out, but the devil is in the
> details.  For example, since interfaces for compilation units cannot be
> accurately expressed in the language, there is a fundamental distinction
> between "true" separate compilation (with specified interfaces for imports)
> and incremental recompilation (with inferred signatures obtained by
> scheduling the elaboration of units in dependency order).

I think the debate about "true" separate compilation is a Red Herring.  The
important problem is to have a standard mechanism for specifying how to
build libraries and applications from sources.

If we had such a mechanism, then we could have substantial amounts
of library code available to all implementations for a fairly small
amount of work.

	- John

RE: [Sml-implementers] extensions to SML

[Robert H. <Rob...@cs...>]

It's good to see such an animated discussion about possible revision and
extension of Standard ML.  What quickly emerges from the discussion is the
need to separate concerns.  Before considering these, let me say that I
don't like to see various efforts described as "valid" or "invalid", or for
them to be divided into "camps".  These descriptions are needlessly
antagonistic to our common goals.

Broadly speaking, we can consider two distinct enterprises:

1. Revision and extension of Standard ML to better support current and
readily foreseen needs.  The emphasis here is on maintaining continuity and
enhancing utility, rather than experimentation and exploration.

2. Drawing on experience with ML to explore new territory in language design
and implementation.  The emphasis here is on developing the "next great
language" in the ML lineage, with diminished continuity and compatibility
constraints.

Personally, I've put effort into both enterprises, which led to the 1997
revision of Standard ML and also to research on new languages and
implementations.

What is clear to me is that it is more than time to renew emphasis on
revision and extension of Standard ML, for several reasons.  First, there is
a crying need, as is clearly evidenced by this discussion.  Second, many of
the proposed revisions are entirely do-able within a reasonable time frame.
Third, it is essential for Standard ML to remain a viable language.

I propose, therefore, that we focus our efforts on (1), leaving aside
discussion of more ambitious projects (which I, among many, wish to
undertake) for another day.  I take it from the discussion that this will
not be a controversial point.

Reviewing the discussion, I can see a few major topics for immediate
consideration.

1. Standardization of a separate compilation mechanism.  This would entail
defining what are compilation units, including how imports and exports are
described, and what is the semantics of linking.  At a high level this
should not be exceptionally hard to work out, but the devil is in the
details.  For example, since interfaces for compilation units cannot be
accurately expressed in the language, there is a fundamental distinction
between "true" separate compilation (with specified interfaces for imports)
and incremental recompilation (with inferred signatures obtained by
scheduling the elaboration of units in dependency order).

2. Standardization of a substantial set of libraries.  Quite obviously this
goes far beyond the meager standard basis library that we currently share.
There are lots of hard problems to be solved here, but given the substantial
code base already in place, I'm sure we can formulate a reasonable plan.  We
would need to consider how libraries interact with separate compilation (the
work on CM is highly relevant here), formulating some interface standards
(we all probably have our own to contribute), and choice of which libraries
to include (the more the merrier, but some harmonization would be required).

3. Standardization of a foreign function interface.  I have not thought very
much about this issue, but I can certainly see difficulties with
compatibility not only across implementations (the various compilers out
there), but also across platforms (primarily, Unix vs Windows).  Fundamental
issues such as the semantics of "int" will arise, since implementations
differ, even on the same hardware and software platform, and since external
code will impose its own requirements.  The need for an FFI is largely
pragmatic --- we cannot re-invent the world ourselves in our own way.
However, as someone pointed out, buying into foreign code will certainly
limit portability across platforms and quite possibly across
implementations.

4. Extensions and modifications to the language itself.  These include
relatively trivial things like denigrating obsolete mechanisms (such as
abstype), re-considering the semantics of structure sharing (which started
this discussion), and adding support for new features (eg, updateable
records, lazy evaluation, vector expressions, richer patterns, hierarchical
extensible sums).  I think there are strong arguments for all of these
changes.  We might also consider ways to improve the syntax while providing
a path for porting old code.

I propose that we confine ourselves to these four categories for immediate
discussion.  (If I've overlooked something, I hope we can quickly agree on
what that is and whether to consider it now.)  It might make sense to form
sub-groups who take charge of specific topics, and report back to the full
group with their proposal.  Once a solid, but informal, proposal is in
place, we can evaluate it by examining its semantics and its implications
for implementation.  Presumably this will lead to revision, but will also
lead rather quickly to a solid revision or extension.

My experience has been that even very modest revisions are very hard to
make.  One reason is that we all have a very substantial commitment to the
language (in the abstract), its semantics, and its implementation.  It's a
tribute to the language that we all have such passionate views about it, and
have contributed so much of our time and energy to it.  It can also be an
obstacle to consensus.  Perhaps it is worthwhile to state a few principles
that I hope can guide us.

1. Standard ML exists independently of its implementations.  The language
should continue to have a formal definition to which implementations agree
to conform.

2. Revisions must be guided as much by the experience of users and
implementors as by the demands of a clean formal definition.  IMO the 1997
revision was hobbled by an excessive emphasis on the needs of The Definition
without due consideration of implementation or application.

3. It is important to achieve a rough consensus, but complete agreement on
all issues may be impossible to achieve.  We will need to have a mechanism
for reaching a decision in the face of disagreement.

Let the discussion begin!

Bob Harper


PS: I, among many, have ideas about new language designs that would take us
beyond the charter outlined above.  It might make sense, if there is
interest, to fork off a separate discussion of these issues.  For example, I
would consider the discussion about automatic generation of equality
functions to fall within this category, as would the proposal I mentioned
for re-working datatypes.  (In fact these fit together nicely.)

Re: [Sml-implementers] extensions to SML

[Daniel C. W. <dc...@ag...>]

At 09:49 AM 9/18/2001 +0200, Andreas Rossberg wrote:
>"Daniel C. Wang" wrote:
>> 
>> Another semantics would be for (op =) to be a partial function i.e.
>>   'a * 'a -> bool option  (or it could raise an exception of some sort)
>
>That is problematic. Consider
>
>        (2, ....fn x => x....) = (3, ....fn x => x....)
>
>where "...." is meant to indicate some deeply nested data structure.
>Now, for the semantics of equality to be well-defined short-cutting the
>comparison after seeing 2<>3 is no longer possible - you always have to
>traverse the whole structure. That implies a serious performance
>penalty. Same for using options.
>
> 

If you're "cleaver" you can avoid this particular problem, and introduce
others. Every value can have an extra "poison" bit, which you set at 
allocation time. If you know the "eqness" of an object at 
compile time because the type is known, then you can set the poison bit 
appropriately. If the eqness of a type is unknown because it is polymorphic  
(i.e. a list) then you set the bit at runtime by inspecting the bit of its 
argument or the type at runtime if you're passing type info at runtime. 
Since, every value now has a "poison" bit. You can first check the poison 
bit and return NONE or else run the standard short-circuiting poly-eq test. 

Your poly-eq function is probably examining some type tags at runtime, so you 
can piggy back the extra bit on your existing tags. If you're doing 
something fancy like dictionary passing to implement poly-eq changing 
your system to implement this trick is pretty easy too. If you're doing 
whole program compilation you statically know when you should just return 
NONE. 

Anyway, you can easily implement this in a way that returns NONE when 
comparing two non-equality types without having to examine the entire 
object. It's just like the partial unboxing tricks used in FLINT and other 
compilers.

I'm not so concerned about the implementation details about this. I think, 
the bigger problem is that it has ugly semantics, from a user point of view. 
This proposal just pushes problems at the type-level into the 
runtime/compile time level.

Re: [Sml-implementers] Structure sharing and type abbreviations

[Andreas R. <ros...@ps...>]

Dave Berry wrote:
> 
> At 12:46 16/09/2001, Derek R. Dreyer wrote:
> >To clarify: the transparent interpretation of datatypes is one in which
> >datatype specifications expose the implementation of a datatype as
> >a recursive sum type.
> 
> Thank you.  That was what I was alluding to; I apologise for using a
> garbled example.
> 
> >I'm pretty sure O'Caml also has an opaque interpretation of datatypes.

Yes, it has. Datatypes are generative in OCaml.

> This brings up a separate point.  Perhaps people could announce new
> SML-related papers on this mailing list?  Would other people find this useful?

A very good idea!

	- Andreas

-- 
Andreas Rossberg, ros...@ps...

"Computer games don't affect kids; I mean if Pac Man affected us
 as kids, we would all be running around in darkened rooms, munching
 magic pills, and listening to repetitive electronic music."
 - Kristian Wilson, Nintendo Inc.

Re: [Sml-implementers] extensions to SML

[Andreas R. <ros...@ps...>]

"Daniel C. Wang" wrote:
> 
> Another semantics would be for (op =) to be a partial function i.e.
>   'a * 'a -> bool option  (or it could raise an exception of some sort)

That is problematic. Consider

	(2, ....fn x => x....) = (3, ....fn x => x....)

where "...." is meant to indicate some deeply nested data structure.
Now, for the semantics of equality to be well-defined short-cutting the
comparison after seeing 2<>3 is no longer possible - you always have to
traverse the whole structure. That implies a serious performance
penalty. Same for using options.

Alternatively, you could try to specify the exact traversal order for
all kinds of values in the language but that would be complicated,
pretty arbitrary and probably unintuitive (consider records, which are
basically unordered).

OCaml does have an exception raising (=) without specifying anything, so
its behaviour is essentially non-deterministic in the case of
higher-order data structures. But I don't think we would want that for
SML.

	- Andreas

-- 
Andreas Rossberg, ros...@ps...

"Computer games don't affect kids; I mean if Pac Man affected us
 as kids, we would all be running around in darkened rooms, munching
 magic pills, and listening to repetitive electronic music."
 - Kristian Wilson, Nintendo Inc.

Re: [Sml-implementers] extensions to SML

[Dave B. <da...@ta...>]

At 14:20 17/09/2001, Matthias Blume wrote:
>I agree with Bob and others who would rather drop polymorphic equality.  

One possibility would be to keep the notion of equality type, and to
replace the polymorphic equality operation with a function overloaded on
equality types.  This would remove the notion of "equality type variable",
and make the language more uniform.

Daniel suggested the opposite approach: keeping polymorphic equality but
getting rid of eqtypes (as in OCaml).  I don't like this at all -- I think
it's one of the weak points of OCaml.  As far as I'm aware, if I define a
set type using unsorted lists in OCaml, the equality operator compares the
representation of two values.  In SML I can use abstraction to make the
type non-eqtype, which seems much preferable.  OCaml's equality can even
return different results when comparing floating point numbers (when NaN),
depending on whether the compiler can deduce the types involved!

To be honest, I doubt that changing the current definition is worthwhile,
compared to other improvements we could be doing.

Dave.

Re: [Sml-implementers] Structure sharing and type abbreviations

[Dave B. <da...@ta...>]

At 12:46 16/09/2001, Derek R. Dreyer wrote:
>To clarify: the transparent interpretation of datatypes is one in which 
>datatype specifications expose the implementation of a datatype as 
>a recursive sum type. 

Thank you.  That was what I was alluding to; I apologise for using a
garbled example.

>I'm pretty sure O'Caml also has an opaque interpretation of datatypes. 

O'Caml restricts datatypes in that constructors take an explicit number of
arguments, and can only be applied to that number of arguments.  E.g. they
cannot be applied to a tuple instead; nor can they be treated as
first-class functions.  This is to simplify the optimisation where the
constructor tag is included in the record.  As you say, this is done in a
non-type-preserving compiler, and I have no idea whether it has any bearing
on the opaque/transparent question.

>We have a new way
>of dealing with the opaque interpretation in TILT that clears up these
>problems.  We are in the process of writing up a new TR with more compelling
>examples of the "permissiveness gap" between the opaque and transparent
>interpretations and with a description of our approach to implementing the
>opaque interpretation in a type-directed compiler.  Stay tuned.

I look forward to reading that.

This brings up a separate point.  Perhaps people could announce new
SML-related papers on this mailing list?  Would other people find this useful?

Dave.

RE: [Sml-implementers] extensions to SML

[Stephen W. <sw...@in...>]

> Portability will result only if the libraries are well designed, and 
> implemented carefully in SML, not in C. All too often I have been 
> thwarted from using SML code because it depends, e.g., on 
> non-portable C code, on a UNIX system call, on a specific pathname 
> syntax, or on system specific line termination.
> 
> Real portability should be a goal if we want SML to succeed. We want 
> SML to be used on a wide variety of systems of the future, e.g., 
> multi-processor servers, PDAs, wearable computers, cell phones, etc., 
> not just on researchers' workstations.

I agree with your assessment that writing as much code as possible in SML and
careful library design will help.  I also agree with your goal of portability.
I would like to point out that we were using "portability" in two different
senses: across-compiler and across-platform.  I was talking about the former
while you were talking about the later.  As I only run on a single platform, I
tend to forget about across-platform, so thanks for reminding me.  I think that
both senses of portability are important.

Unfortunately, there is a lot that can not be done in SML + the standard basis
library, and even if we add more libraries, this will always be the case.  So,
an across-compiler portable C FFI is important.

Re: [Sml-implementers] Structure sharing and type abbreviations

[Dave B. <da...@ta...>]

At 10:09 17/09/2001, Matthias Blume wrote:
>I disagree with this view.  If you prohibit redefinitions of names, you get 
>a serious modularity problem:   Any program that has a global or local name
>"foo" can no longer be linked with a library that exports a binding under the
>name "foo".  

I'm assuming that each library is a separate namespace.  Clearly we need a
way to distinguish a binding in one library from that in another.

>As for the interactive toplevel:  I agree with you that having it is pretty 
>useless for anyone outside the theorem-proving community.  
>
>On the other hand, I do not agree with your assessment that there is 
>something fundamentally flawed about the way the interactive toplevel
works in ML.

I think your first statement is a strong counter-argument to your second.
I would like a system that allows interactive program development, not just
a shell for theorem provers.  But tastes differ.

>Anyway, I can live with the idea of not having an interactive toplevel. On 
>the other hand, not being able to redefine (locally or at the
(non-interactive)
>toplevel) a globally bound name is absolutely unacceptable IMO.

Please note that I explicitly stated that rebindings at inner scope would
be permitted.  I didn't mention libraries, but it is certainly my intention
to allow an identifier to be defined in different libraries.

Dave.

RE: [Sml-implementers] extensions to SML: Foreign function interf ace

[David M. <Dav...@pr...>]

On Sunday, September 16, 2001 3:19 PM, Dave Berry
[SMTP:da...@ta...] wrote:
> At 16:03 14/09/2001, Stephen Weeks wrote:
> If you have a standard interface to C and a standard for compilation
> management, then it becomes much easier to write libraries.  Any SML
> programmer will be able to write libraries that run on any SML
compiler.
> That seems a "base level" that everyone on this list should aim for.
> 
> I suggest an IDL interface for the C FLI.  People should not have to
write
> any C themselves.  I know SML/NJ were working on this, as were the
MLWorks
> team.  

Dave,
Have you looked at the foreign function interface that Nick Chapman did
for Poly/ML several year ago? (See
http://www.lfcs.ed.ac.uk/software/polyml/docs/CInterface.html ).  It's
been used for several projects but particularly the Windows interface
which has several hundred functions in it.  It works for dynamic
libraries in Unix or DLLs in Windows and doesn't require the SML
programmer to write anything but ML.

As an example, the following code creates a function to put up a message
box using the Windows MessageBox function.  MessageBoxA is a function
within the user32 DLL and is normally called from C or Visual Basic code
but it's just as easy to give it an ML type and call it with ML
arguments. 

local
	open CInterface
	fun userDLL name = load_sym (load_lib "user32.dll") name
in
	val MB_OKCANCEL           = 0x00000001
	val MessageBox = call4 (userDLL "MessageBoxA") (INT, STRING,
STRING, INT) INT
end;

MessageBox is now a function with type    int * string * string * int ->
int  and can be called as
MessageBox(0, "Is this OK?", "The title", MB_OKCANCEL);

The converters INT and STRING are defined in the library but it's easy
to define new converters for arbitrary types.

David.

RE: [Sml-implementers] extensions to SML

[David M. <Dav...@pr...>]

I've been thinking of how to respond to the suggestions for extensions
and changes to the language.  Most of all I've been trying to work out
why there is the desire for change.  What is it trying to achieve?  

It seems to me that there are three groups of people involved in SML who
may have different views on the desirability of change: language
designers, implementers and users.  Of course, in practice these groups
overlap and at various times I've been a member of each.  For a language
designer adding new features and making changes are the whole point of a
language.  A stable language represents a language with no interest.
For implementers change represents new challenges although it's possible
to find interest and challenges in finding novel ways to implement a
stable language.  Users are probably the most conservative of the
groups, especially users with a substantial investment in code.  For
them any change is probably a bad idea and if there has to be change it
must be done in such a way as to provide backwards compatibility.

I assume we are all in favour of encouraging the use of SML and the
differences are all in how best to achieve this.  Will we encourage more
use by "improving" the language, by giving them a better, cleaner
language to program in?  My feeling is not, that rather by making
changes we will discourage more users than we draw in.  Existing users
may decide that it's not worth going through their code adapting it to
the latest version of the language.  I feel that if we have to make any
changes to the language they should be minor changes which don't break
existing code.

There seem to be a number of interesting suggestions for language
changes and improvements being made.  It seems to me that the best way
of experimenting with them is in the context of a new language, freed
from the constraints of needing to be compatible with SML.  That isn't
to say that this will require starting from scratch.  Many of the SML
compilers are open source and if the new language is similar to SML it
should be possible to adapt an existing compiler to support the new
language.  Who knows, if the new language is successful we may end up
adapting all the existing SML implementations to support it?  We do,
though, have a duty to existing users not to disrupt their SML code.

David.

Re: [Sml-implementers] extensions to SML

[Doug C. <e...@fl...>]

At 12:00 PM -0700 9/17/01, Stephen Weeks wrote:
>I agree with Dave Berry that if we focus on getting a portable C FFI
>and compilation management system, then the libraries are much more
>likely to follow.  Therefore, we should spend our effort there.
>Designing and specifying a new set of libraries is too much effort.
>It would be better to let libraries evolve from code that people
>write.

If real portability is a goal, then this is the wrong approach. 
Portability will result only if the libraries are well designed, and 
implemented carefully in SML, not in C. All too often I have been 
thwarted from using SML code because it depends, e.g., on 
non-portable C code, on a UNIX system call, on a specific pathname 
syntax, or on system specific line termination.

Real portability should be a goal if we want SML to succeed. We want 
SML to be used on a wide variety of systems of the future, e.g., 
multi-processor servers, PDAs, wearable computers, cell phones, etc., 
not just on researchers' workstations.

e