Re: [gobo-eiffel-develop] Introspection, persistence, debugger

[Eric B. <er...@go...>]

Hi Wolfgang,

The first rule to follow in the Gobo team is to reply
to the mailing-list. Can you please make sure to do that
in the future? Thank you.

Wolfgang Jansen wrote:
> Eric Bezault wrote:
>> Wolfgang Jansen wrote:
>>> Eric Bezault wrote:
>>>> Wolfgang Jansen wrote:
>>>>> Eric Bezault wrote:
>>>>>> Hi Wolfgang,
>>>>>>
>>>>>> Wolfgang Jansen wrote:
>>>>>>> I would like to contribute the following
>>>>>>> features/parts to the GEC compiler:
>>>>>>> - introspection (in particular, class INTERNAL)
>>>>>>>  and persistence closure (in particular, class STORABLE),
>>>>>>>  see persist.pdf
>>>>>>> - debugger, see debug.pdf
>>>>>>>
>>>>>>> In the course of recent years I implemented those parts
>>>>>>> for the SE-1.* compiler (not integrated into the
>>>>>>> official release), and the last version is almost
>>>>>>> completely written in Eiffel. Thus, it should be
>>>>>>> possible to integrate the stuff into the GEC as well.
>>>>>>> The integration of introspection and persistence closure
>>>>>>> (at least the core parts) seems to be straight forward
>>>>>>> whereas integrating the debugger is much more involved.
>>>>>>>
>>>>>>> Most of the development still to be done will be my work
>>>>>>> (help about the meaning of existing compiler classes
>>>>>>> and their features may be necessary from time to time).
>>>>>>> Moreover, the development is rather separated from
>>>>>>> the main lines of the development of the compiler,
>>>>>>> therefore, conflicts should be rare. On the other hand,
>>>>>>> things may take some time since it's not my main work.
>>>>>>>
>>>>>>> What do you think about the topic?
>>>>>>
>>>>>> I just finished reading your documents, and here are my
>>>>>> remarks:
>>>>>>
>>>>> The PDF description was rather a user's guide, not a developer's 
>>>>> guide.
>>>>> To answer your remarks it will be wise to sketch the implementation.
>>>>>>
>>>>>> INTERNAL
>>>>>> --------
>>>>>>
>>>>>> In order to take full advantage of the dynamic type set
>>>>>> mechanism implemented in gec, my plan was to implement
>>>>>> class TYPE first (introduced in ECMA), and then add to
>>>>>> class TYPE the features currently available in INTERNAL.
>>>>>> The features of INTERNAL could be implemented using those
>>>>>> from TYPE in order to allow backward compatibility for
>>>>>> applications currently compiled with ISE. Let's consider
>>>>>> that we have in ANY:
>>>>>>
>>>>>>    generating_type: TYPE [like Current]
>>>>>>
>>>>>> and in class TYPE:
>>>>>>
>>>>>>    attribute_count: INTEGER
>>>>>>         external
>>>>>>              "built_in"
>>>>>>
>>>>>> we could implement `field_count' in INTERNAL as follows:
>>>>>>
>>>>>>    Result := object.generating_type.attribute_count
>>>>>>
>>>>>> and likewise for the other features in INTERNAL. In your
>>>>>> document you mentioned a class INTRO_TYPES. Is it similar
>>>>>> to class TYPE above? In the generated C code there is
>>>>>> already an array 'getypes' whose purpose will be to return
>>>>>> a TYPE object indexed by type-id.
>>>>>>
>>>>>>
>>>>> As mentioned in the description, class INTERNALS is just a wrapper 
>>>>> class.
>>>>> The heart of introspection is a couple of classes INTRO_TYPE,
>>>>> INTRO_FIELD etc. which mirror information contained in compiler 
>>>>> classes
>>>>> (RUN_CLASS, ... in case of SE, possibly ET_DYNAMIC_TYPE, ...
>>>>> in case of GEC) as far as needed during runtime. Moreover, the classes
>>>>> have features needed during runtime only (e.g. creating an object
>>>>> of the type in question). The compiler's part is to strip the 
>>>>> appropriate
>>>>> information from RUN_CLASS, ... objects to make INTRO_TYPE, ...
>>>>> objects and to write the result to C code.
>>>>>
>>>>> The approach should be open to implement class TYPE by means
>>>>> of class INTRO_TYPE.
>>>>
>>>> Does that mean that interoperability with ISE will be broken?
>>> I don't know, my knowledge of ISE is rather poor.
>>>> Currently class TYPE is in FreeELKS, and the idea is that both
>>>> ISE and Gobo use the same class for interoperability. The trick
>>>> is to use 'external "built_in"' to let the each compiler plug
>>>> its own runtime. My feeling if we want interoperability with ISE
>>>> is that all features needed to implement introspection need to
>>>> be in FreeELKS. This mean that either everything needed is in
>>>> class TYPE (or INTERNAL) and your classes INTRO_TYPE/INTRO_FIELD
>>>> are based on that. Or your classes INTRO_TYPE/INTRO_FIELD
>>> Is it a joke? Do you really want to base elementary classes
>>> on complex ones?
>>
>> Well, I know nothing about your classes except that you
>> want to use them to implement other parts of your contribution.
> The appendix contains the short form (still SE generated) of the classes.

I cannot see it in the document you sent me.

>> I don't even know what might be the difference between TYPE
>> and INTRO_TYPE. So it's hard to say why one is more complex
>> than the other. I can see that one might be more object-oriented,
>> but more basic/complex? The only thing that I know is that
>> if we want to have some interoperability with ISE, either
>> we add what is needed in class TYPE because it seems to be
>> the chosen mechanism by ECMA to have an embryo of introspection.
>> Or we decide that your approach is better/more elegant/more
>> object-oriented and we discuss with ISE and ECMA to replace
>> TYPE by your solution.
>>
> My solution is more elementary since class INTRO_TYPE
> is not generic. 

I don't see why not being generic makes it more elementary.
The advantage of having TYPE generic is that we can have
`new_instance' which returns an object whose type is known
at compilation time. Apart from that, why is the fact that
INTRO_TYPE is not generic making it more elementary?

> It is more object-orientated since it provides
> different classes, including inheritance if appropriate, for
> different abstractions. It is less elegant, at least less programmer
> friendly (the generic parameter of class TYPE lets the programmer
> express the type of interest). It is more flexible since it is
> our decision which features to add, remove, or modify.

If these features require some compiler support, then it means
that we gain flexibility but lose interoperability with other
Eiffel compilers.

> Is it better? I don't know, too many criteria are to be considered.
> 
> I'm more and more convinced that class TYPE may be
> implemented on base of class INTRO_TYPE.
> For example:
> 
>   class TYPE[G]
> 
>   create set_internal
> 
>   feature {NONE} -- Initialization
> 
>      set_internal(i: like internal) is
>         do
>            internal := i
>         end
> 
>   feature
> 
>      name: STRING is
>         do
>            Result := internal.base_class.name
>         end
> 
>      -- other features
> 
>   feature {NONE} -- Implementation
> 
>      internal: INTRO_TYPE
> 
>   end

But the reverse could also be true. I don't see why we could
not implement INTRO_TYPE with features of TYPE. Why couldn't
TYPE be equipped with features that would provide information
about its base class, about its attributes, ...

> One point is, when to generate an object of, say, TYPE[STRING].
> Will be there exactly one object for each type occurring in the system,

Yes.

> possibly from the program's startup? Having more objects per
> generic parameter is meaningless since each describes the same thing.
> Another point is, having an object of TYPE[STRING] then
> also TYPE[STRING] is an alive type and we need type
> TYPE[TYPE[STRING]] either. You see, we immediately
> enter an infinite recursion.

The infinite recursion only occurs when you want to introspect
on objects of type TYPE.

> To the first point: my solution provides one type, namely
> INTRO_TYPE, and exactly one object of that type for each
> type in the system. Well, there is also an object for INTRO_TYPE
> itself, but this does not cause infinite recursion.
> If based on this solution then it should be possible to implement
> class TYPE such that exactly one object exists for each necessary
> generic parameter value, simply by mapping the objects of
> INTRO_TYPE to those of TYPE[G]. The advantage of is
> that classes INTRO_TYPE etc. already exist.
> What remains to do is to create objects of these types by extracting
> information from objects of ET_DYNAMIC_TYPE etc.,
> and then, when this works, to implement class TYPE.
> 
> To the second point: Of course, class TYPE needs a special
> treatment to avoid the infinite recursion. Here, the approach
> to implement TYPE on top of INTRO_TYPE does not help.

The recursive recursion is something that has already been
discussed at ECMA and is being addressed. I don't think that
this is a show-stopper, knowing class TYPE is known by the
compiler.

> There is no question whether to follow ECMA or not, and it is
> no question that class TYPE should _not_ be replaced by classes
> INTRO_TYPE etc. 

Well, the question is about interoperability. And this is where
ECMA, ISE and FreeELKS are concerned. I don't think it is good
for Eiffel to have libraries which only work with SmartEiffel
or only work with ISE. Let's try not to repeat these mistakes
in Gobo. I keep coming back to class TYPE because that's the
only entry point that we currently have for interoperability
in the introspection world. Here I speak about TYPE as a basic
class (equipped with 'external "built_in"' features), not as
something based on even more basic classes. As I already said,
if you think that your classes are/should be more basic that TYPE
(or that TYPE cannot achieve the kind of introspection that you
have in mind), then we should discuss that with ECMA/ISE/FreeELKS
at all means rather than introducing interoperability issues.

> The question is, whether to use these classes
> as a basis of the implementation of the former one.
> And the question is, whether to use these classes directly
> to implement the persistence closure or indirectly via class TYPE.
> I think that the direct use is more efficient.
> If class TYPE is implemented on the base of class INTRO_TYPE
> then the final question is, whether this should be a solution
> for eternity or merely a temporary solution for now.
>>> What makes sense is to implement class TYPE
>>> on base of classes INTRO_TYPE etc. Where to locate the classes,
>>> together with class TYPE in FreeEKLS or in a more hidden cluster,
>>> is a secondary question.
>>>> (by the way, I'm not a big fan of these class names) should be
>>> By the way, INTRO_TYPE etc. are the _current_ class names.
>>> Changing them is no problem and will be done for the GEC version.
>>> Please, do not misunderstand the final state for SE to be also
>>> the final state for GEC.
>>>> added to FreeELKS with the proper 'external "built_in"' routines
>>>> to let other compilers like ISE be able to plug their implementation.
>>>> In either case, I think that it is better to discuss with ISE
>>>> about what we could do with FreeELKS in order to make available
>>>> in an interoperable way the features that you need to implement
>>>> introspection.
>>>>
>>>>>> STORABLE
>>>>>> --------
>>>>>>
>>>>>> It would indeed be interesting to plug your implementation
>>>>>> of STORABLE in gec. I have two concerns though. First, are
>>>>>> you aware that ISE also implemented a version of Storable
>>>>>> in Eiffel:
>>>>>>
>>>>>> http://dev.eiffel.com/cgi-bin/viewvc.cgi/trunk/Src/library/base/ise/serialization/ 
>>>>>>
>>>>>>
>>>>> I knew only older variants (it's at least three years ago that I 
>>>>> played with ISE),
>>>>> and I played also with serialization but did not analyze it.
>>>>>> As far as I understood, they only use INTERNAL. So I wonder
>>>>>> whether you should try to use the experience you gained when
>>>>>> developing your Storable for SmartEiffel and try to improve
>>>>>> ISE's SED library rather than providing a competing library.
>>>>>>
>>>>> I developed the stuff from scratch (I started in 2001 or 02  when
>>>>> ISE was not yet open). What I've just read, so has ISE's approach
>>>>> a similar class hierarchy but the underlying implementation
>>>>> seems to be quite different.
>>>>> In any case, my approach is bases on classes INTRO_TYPE, etc.
>>>>> not on class INTERNALS.
>>>>>> The other concern that I have is interoperability. It would
>>>>>> be nice if gec could at least have a way to read and write
>>>>>> the file formats produced by the old ISE's STORABLE mechanism
>>>>>> (at least independent_store). And why not also the file format
>>>>>> produced by SE 2.*, although I'm less interested in this one ;-)
>>>>>> It may be sound odd to try to be interoperable with ISE's old
>>>>>> STORABLE, but that's something I will need at work (we have a
>>>>>> lot of Storable files like that). And probably others currently
>>>>>> using ISE Eiffel will have the same need. So this will have
>>>>>> to be implemented at some stage.
>>>>>>
>>>>> Yes, I understand. The point was that I did not know ISE's
>>>>> store format. (The appendix of file "persist.pdf" contains the BNF
>>>>> of the store format I use. Are you aware of an analog description
>>>>> of ISE's store format? Knowing this would be a great help.)
>>>>
>>>> Each time I asked ISE I was told that I should read the source
>>>> code.
>>>>
>>> I downloaded the stuff yesterday. Maybe that I have time to study it
>>> on weekend. It will be rather hard to understand it.
>>>>> On the other hand, I know the store format of SE. In fact,
>>>>> it is possible to store in that format, but not to read from.
>>>>>
>>>>> The implementation is based on a class PERSISTENCE_CLOSURE
>>>>> managing the traversal through the persistence closure and
>>>>> on deferred classes PERSISTENCE_SOURCE and
>>>>> PERSISTENCE_TARGET for handling elementary data.
>>>>> Effective descendants of the latter two are combined
>>>>> with the first to get the desired effect. As sources exist
>>>>> reading from memory and from a file written by the corresponding
>>>>> target class. The question whether ISE's store format will
>>>>> be readable is the question whether it will be possible
>>>>> to implement a new source class while keeping class
>>>>> PERSISTENCE_CLOSURE as it is.
>>>>> On the other hand, there are much more effective target
>>>>> classes,in particular those which generate output that is
>>>>> not devoted to be re-read by PERSISTENCE_CLOSURE.
>>>>> Among those target classes is one that generates SE's
>>>>> store format and one that writes, at the end of compilation,
>>>>> the INTRO_TYPE, ... objects to C code. As for ISE,
>>>>> the question whether the SE store format can be read
>>>>> is the question whether an appropriate heir of class
>>>>> PERSISTENCE_SOURCE can be implemented.
>>>>> And here I know the answer: it is not possible.
>>>>>
>>>>> I final remark: classes SERIALIZER and SERIALIZE
>>>>> mentioned in the description are merely wrapper classes
>>>>> to make programmer's life easier.
>>>>>> deep_twin
>>>>>> ---------
>>>>>>
>>>>>> You didn't mention it in your message, but it is mentioned
>>>>>> in your document. None of the deep features are implemented
>>>>>> yet in gec. Even though I don't like to use deep_twin myself
>>>>>> (I prefer to control the level of duplication I want by
>>>>>> redefining `copy'), one of the programs I currently try to
>>>>>> compile with gec at work uses it. So it will need to be
>>>>>> implemented soon.
>>>>>>
>>>>> In fact, I didn't try it so far, but I'm sure it will work.
>>>>> The only bad thing is that the general approach causes overhead
>>>>> in space and time slowing down the `deep_twin' by some factor.
>>>>> Of the other routines, `deep_copy' will be quite similar to
>>>>> `deep_twin' whereas `deep_is_equal' is different:
>>>>> we have two sources and no target. Maybe that one can
>>>>> misuse the target as one another source.
>>>>>
>>>>>>
>>>>>> Debugger
>>>>>> --------
>>>>>>
>>>>>> This topic is something that I'm not very good at. So I really need
>>>>>> to rely on others to implement a good debugging mechanism for gec.
>>>>>> I have not really thought about it. My only requirement is that
>>>>>> it should be user-friendly and should do what people expect from
>>>>>> a debugger. So people who want to contribute in this area will
>>>>>> be free to do pretty much what they want, also I will be interested
>>>>>> in following their progress.
>>>>>>
>>>>>> But gec is not as advanced as SmartEiffel with respect to
>>>>>> debugging. There is currently only one compilation mode, with
>>>>>> no stack trace, no assertion monitoring, no exception. So
>>>>>> everything needs to be done in this area. If this is not
>>>>>> frightening you, then that would be great.
>>>>>>
>>>>> The debugger started as a by-product of introspection/persistence
>>>>> closure: to implement an "evaluate" (or "print") command
>>>>> one needs introspection. Since the SE debugger did that time
>>>>> (yes, it is long ago) not support evaluation even of the most
>>>>> simple expressions, I added such a command to the debugger.
>>>>> And since I'm a friend of debuggers, I continued the work
>>>>> to make the debugger more user friendly, finally,  to implement
>>>>> it in Eiffel.
>>>>>
>>>>> Expression evaluation of the debugger is based on the
>>>>> introspection discussed above (whether INTRO_TYPE
>>>>> directly or indirectly via TYPE is of minor concern).
>>>>> Well, expression evaluation is just a part of debugging,
>>>>> but an inevitable part. Thus, whether I can add the debugger
>>>>> to GEC is also the question whether my approach of
>>>>> introspection will been added.
>>>>>
>>>>> In any case, adding the debugger (probably any debugger)
>>>>> to GEC will have much more impact on the compiler and
>>>>> the generated C code than introspection. This will cause
>>>>> more discussion and negotiation.
>>>>
>>>> But since the compilation mode for debugging does not exist
>>>> yet (only the equivalent of SE's boost mode is implemented),
>>>> I think that it should be less controversial than if such
>>>> compilation mode already existed and needed to be adapted.
>>>> Here I wonder if we could simply have a class ET_DEBUG_C_GENERATOR
>>>> which inherits from ET_C_GENERATOR. Or have something like
>>>> that:
>>>>
>>>>                    ET_C_GENERATOR*
>>>>                      ^       ^
>>>>                     /         \
>>>> ET_FINALIZE_C_GENERATOR    ET_WORKBENCH_C_GENERATOR
>>> Yes, it was my idea, too, to put much of the work into code generation
>>> (and in case of GEC, in contrast to SE, it is well concentrated).
>>> Nevertheless, a class ET_WORKBENCH_C_GENERATOR
>>> will be not sufficient.
>>> First of all, the debugger is written in Eiffel but the user written
>>> program does not contain an object of class DEBUGGER
>>> (or which class name you like). Thus, the class will normally not
>>> be compiled and the executable will not have debugging facility.
>>> What's needed is to compile class DEBUGGER additionally to
>>> the user's system as a second root class and to create an object
>>> of that class during runtime as a second root object.
>>> Second, during compilation one has to decide what to do with
>>> classes and features that occur twice: generating them twice
>>> and giving them different C names, or reusing in the debugger
>>> class what has been generated outside of them (causing that
>>> the debugger will debug itself, possibly an infinite recursion).
>>> The duplication of the root and possibly other code is
>>> what I mean by large impact onto the compiler.
>>>>
>>>>>>
>>>>>> You spoke about your availability to work on this project.
>>>>>> INTERNAL/TYPE and deep_twin are something that I will need
>>>>>> relatively soon. So I guess I should start implementing
>>>>>> them myself. Of course it does not mean that you cannot
>>>>>> help or that I won't need your help. For STORABLE,
>>>>>> what I will need will be something similar to ISE's
>>>>>> independent store. I don't think that any of us can
>>>>>> implement that in short amount of time. I guess that
>>>>>> you would prefer to work on your own Storable format (which
>>>>>> looks interesting just by reading you document), although it
>>>>>> would be nice if you could have a look at ISE's SED and see
>>>>>> if/how it could be improved. The part of your contribution
>>>>>> offer that I would be the most interested in, considering
>>>>>> my need and your availability, would be the debugger part.
>>>>>> Indeed, this is not something that I need in the short term,
>>>>>> so you have plenty of time to work on it without too much
>>>>>> constraints. There is not that much conflicting parts with
>>>>>> what I currently work on in the compiler. And this is something
>>>>>> that I don't think I will be able to implement myself anyway.
>>>>>> And everything needs to be done from scratch. So if you are
>>>>>> not frightened by that, this could be an interesting experience.
>>>>>>
>>>>> As already mentioned, the core part of introspection/persistence
>>>>> (i.e introspection of types and attributes, not routines
>>>>> and local variables, not class TYPE; persistence without
>>>>> class ACTIONABLE, not ISE's format) should be not very
>>>>> difficult. Besides the general task to replace SE specific classes
>>>>> and features by GOBO specific ones, there remains only
>>>>> the extraction of relevant information from classes
>>>>> ET_DYNAMIC_TYPE etc.
>>>>> I think that I can tell you tomorrow evening
>>>>> how much time will be needed.
>>>>>> Now I should mention that people contributing to the Gobo
>>>>>> project need to follow some programming rules. It is difficult
>>>>>> to follow them at first because most of them are not explicitly
>>>>>> stated. Some are here:
>>>>>>
>>>>>>    http://www.gobosoft.com/eiffel/gobo/guidelines/
>>>>>>
>>>>>> Some are just implicit (just look at how the other classes
>>>>>> in Gobo look like). The idea is to have a Gobo package where
>>>>>> all classes follow the same programming style so that we have
>>>>>> some sort of uniformity, instead of being a set of independently
>>>>>> developed parts.
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
> 
> 


-- 
Eric Bezault
mailto:er...@go...
http://www.gobosoft.com