czt-devel — For programmers using the CZT code

Re: [CZT-Devel] Reducing memory usage: Signatures are the biggest culprit?

[Anthony H. <an...@an...>]

Dear Andrius

 

This is brilliant!

 

Since I had a profiler on CZT running for catching some leftover debugging code, I thought I would give a quick look into the CZT memory usage as well.

 

I have discovered that most of the large amount of ZNames that are created are referenced from typechecker Signatures (via NameTypePairs). This is the case because when creating type signatures during typechecking, the Signatures are duplicated in a lot of places.

 

A prominent example of this is in net.sourceforge.czt.typechecker.z.ExprChecker:94 (method visitRefExpr). This bit of code calculates the type of a RefExpr. If a reference is to a Schema, it duplicates the Schema signature (assigns new IDs to all its names) and then uses the new signature within the power type.

 

As a quick test, I tried removing this duplication and instead reuse the original schema signature (with original ZName ids): `Signature sig = signature;`

 

This single change reduced the memory consumption of typechecked `spec.tex` by about 60%!

Excellent!

 

This lead me to thinking - do signatures really need to be duplicated everywhere? I imagine that the signature of schema A would be the same everywhere? Can we reuse the signature, or is it important to duplicate and assign new IDs to the name?

 

The massive space saving becomes quite obvious when you think about it. If we have schema references, every RefExpr would duplicate the whole schema definition when typechecked, hence creating a lot of new ZName instances and consuming all this memory.

 

Leo advised that Tim would be the person to ask about the Signature duplication in typechecker? Is it really necessary, or can we reuse the objects?

Andrius, this is exactly the sort of thing I hoped might be possible. I don’t know how much of the previous discussion you’re aware of, but at first sight it does look as if objects are being copied many, many times and the total number of objects created seems to be bigger than should be needed, not by a few percent, but by orders of magnitude.

 

If Tim can confirm that these objects and others like them are indeed immutable, then it seems not only more efficient but also the Right Thing simply to copy the object reference. If there are a few more places you can find savings of this sort of percentage then the problem would be fixed. (Two more 60% and you’d have an order of magnitude already!)

 

I have a selfish follow up question. Assuming Tim can confirm that this, or something like it, is acceptable then would you be able to do a bit of work on this? It’s true that I had very diffidently volunteered to look at this area, but it would be clearly much more effective and efficient for you, who obviously know what you are doing, to look at it than for me to spend a lot of time groping around and wasting your time asking dumb questions. Of course I’d be extremely happy to help with any testing, especially if you get to the point where typechecking large specifications becomes feasible (at the moment it simply isn’t possible to typecheck the whole of the iFACTS spec in CZT on any machine I’ve been able to try.)

 

Anyway, I’m really encouraged by this result and thank you so much for doing this – I’m delighted (and impressed!)

 

Anthony

 

 

Re: [CZT-Devel] Memory allocation with ArrayLists; arrays; etc.

[Mark U. <ma...@cs...>]

Andrius,

Very interesting that duplication of big ints and signatures etc. is
consuming so much memory.

Assuming that these are immutable after they have been created, it sounds
like it would be worth using a factory design pattern that caches the
constructed objects. It would need to be a weak cache though, so the
garbage collector could reclaim structures that are in the cache but are
not used elsewhere.  Like the Google Guava Interner (
http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/collect/Interner.html
)

Cheers
Mark



On 13 April 2013 01:40, Andrius Velykis <and...@ne...>wrote:

> Yes, the big integers will get large. But in your previous e-mail you said
> "given most of them are shared"? So I wanted to object, that I think they
> are not actually shared.
>
> -16..16 comes from BigInteger.valueOf() method - check the Java sources. I
> wanted to say that the only sharing of BigInteger objects is in this range..
>
> So I just wanted to say that BigIntegers do consitute a significant part
> of CZT memory usage :)
>
> Andrius
>
>
> On Fri, Apr 12, 2013 at 4:28 PM, Leo Freitas <leo...@ne...>wrote:
>
>>  Hi Andrius,
>>
>>  I don't think so. The big integers are used for the line / column /
>> buffer length numbers no?
>> So if you have  large files they will get large. Or am I mistaken? Where
>> did the -16..16 came from?
>>
>>  Leo
>>
>>  On 12 Apr 2013, at 09:12, Andrius Velykis <
>> and...@ne...> wrote:
>>
>>  Hi Leo,
>>
>>  (reviving and old post..)
>>
>>   As is, the only major problem with memory consumption is the
>>> duplication (e.g., ArrayList/Object[]).
>>> The presence of various (4-6) BigIntegers in LocAnn is also an issue,
>>> but given most of them are shared
>>> it's not such a big deal.
>>>
>>
>>  From what I gather, BigIntegers are only shared when in the range of
>> [-16, 16] -- looking at BigInteger.valueOf()? Or am I mistaken? If so, most
>> of the BigIntegers would not be shared in CZT..
>>
>>  Andrius
>>
>>
>> On Wed, Jan 25, 2012 at 6:33 AM, Leo Freitas <leo...@ne...
>> > wrote:
>>
>>> Hi Tim,
>>>
>>> Yes, after parsing. And yes, I found that rather odd too. There is
>>> something funny going on
>>> in Garbage Collection somewhere. I guess the trouble is that there are
>>> certain structures
>>> (e.g., TokenSequence iterators) that end up retaining the biggest part
>>> of the object memory.
>>>
>>> Having said that, I was now trying to find the sources of the problem(s)
>>> using more intrusive
>>> snapshot points (AKA memory dumps at particular execution points).
>>>
>>> As is, the only major problem with memory consumption is the duplication
>>> (e.g., ArrayList/Object[]).
>>> The presence of various (4-6) BigIntegers in LocAnn is also an issue,
>>> but given most of them are shared
>>> it's not such a big deal.
>>>
>>> I've tweaked the PerformanceSettings constant(s) a little bit more to
>>> keep arraylists/object[] capacity 0 at creation,
>>> and that just about halved the memory used (!)... possibly at some speed
>>> expense given it will take some time to
>>> increase capacity.
>>>
>>> At the profiling sessions done, this wasn't a problem: I've put initial
>>> capacity at 0, 1, and 10 (e.g., only very few go
>>> beyond that, mainly on type info), and the CPU increase was marginal
>>> (e,g., with 10 it was 300ms - 2.75% - quicker; with 1 100ms quicker),
>>> yet  the memory gain was significant (e.g., with 10 and 1 it was it was
>>> twice the heap at 100MB instead of 50MB for 0).
>>>
>>> One nice thing would be to have this as a configuration file or a
>>> SectionManager option.... once all key bottleneck points are identified.
>>> By default I will keep it at 0, as the performance penalty in CPU time
>>> seems marginal/acceptable (e.g., 2-3% penalty for halving memory needs).
>>>
>>> Suggestions / comments?
>>>
>>> Best,
>>> Leo
>>>
>>> On 24 Jan 2012, at 23:11, Tim Miller wrote:
>>>
>>> > Nice work!
>>> >
>>> > When you say you added "parser = null", I assume you mean after you do
>>> > the parsing? This would mean that, prior to your change, a reference to
>>> > the parser is being kept even after the variable scope of the variable
>>> > "parser" ends? That seems odd.
>>> >
>>> > On 25/01/12 01:22, Leo Freitas wrote:
>>> >> Tim,
>>> >>
>>> >> I've forgot to mention below that I was talking about parsing and
>>> typechecking only.
>>> >>
>>> >> I've just run the same profiling setup on the VCG for these larger
>>> examples and yes, the worst ones were
>>> >> Object[] (24%), char[] (17.3%), ArrayList[] (9%) and BigInteger
>>> (2-3%).
>>> >>
>>> >> Surprising ones were int[] (6.5%) and short[] (6%) arrays, since they
>>> are not explicitly created anywhere in CZT.
>>> >> LocalAnn took 3%; Iterators were also a surprise: both iterator() and
>>> listIterator() cover about 10%, where more
>>> >> obvious ones like String (0.9%) and ZName (1.7%) were quite low.
>>> >>
>>> >> I've run the profiling sessions about 3-5 times on each example
>>> taking the average.
>>> >>
>>> >> ----
>>> >>
>>> >> Searching for potential sources of such unexpected types, I managed
>>> to find a few successful candidates:
>>> >>
>>> >> a) ParseUtils
>>> >>
>>> >> char[], short[], and int[] are mostly present in the Java CUP
>>> low-level classes to represent the internal parsing tables from the grammar.
>>> >> I simply added an explicit "parser = null" to the main
>>> ParseUtils.parse method. These arrays combined account for about 30% of
>>> memory.
>>> >>
>>> >> after the change, these arrays now account for 1.1% (!!!) that's
>>> quite an improvement.
>>> >>
>>> >> b) ListIterator and Iterator
>>> >>
>>> >> These appear in various places, but places that should influence all
>>> runs are only on SmartScanner (and possibly PrettyPrinter).
>>> >> After this change, they went from a combined (ListIterator +
>>> Iterator) 49% footprint to 17%; another good improvement.
>>> >>
>>> >> d) ArrayList and Object[]
>>> >>
>>> >> There are various places and identifying where are the ones of most
>>> significance is tedious/time-consuming.
>>> >> Instead, I've done a thorough search through various projects and
>>> changed default constructors to more sensible values, when possible,
>>> >> or to a parameterised default when not (e.g., PerformanceSettings
>>> interface in util project).
>>> >>
>>> >> This led to a decrease in the memory footprint of these objects of....
>>> >>
>>> >> e) BigInteger
>>> >>
>>> >> Why do we need big integers within LocAnn and other places? I guess
>>> because long would be potentially small?
>>> >> But would there really be something bigger than 2^32 or 2^64 in
>>> number of lines of source in a file say? Hum...
>>> >>
>>> >> In terms of memory footprint, it varied between 0.7% to 3%, depending
>>> on the example. I won't change this one for now.
>>> >>
>>> >> f) Garbage collection time
>>> >>
>>> >> Firstly it was about 16-20% now it was about 46.1% of time taken.
>>> Don't know if that's related to changes, but looks like it.
>>> >>
>>> >> ======
>>> >>
>>> >> Changes a) and b) led to a drastic memory footprint decrease from 8MB
>>> to 0.91MB;
>>> >> Other changes led to a minor improvement in comparison (i.e., 0.80MB).
>>> >>
>>> >> That's despite the fact that changes in d) were the most numerous -
>>> they didn't amount to much improvement, but some.
>>> >>
>>> >> Hopefully this will enable much better performance on larger specs. I
>>> am committing it now to see how it goes.
>>> >>
>>> >> Best,
>>> >> Leo
>>> >>
>>> >> On 23 Jan 2012, at 14:00, Leo Freitas wrote:
>>> >>
>>> >>> Hi Tim,
>>> >>>
>>> >>> That's interesting. I';ve been doing similar (profiling) tests over
>>> specs of some size (e.g., Mondex, Tokeneer, Xenon, IEEE float point unit);
>>> >>> although I guess they are smaller than iFACTS - apart from Xenon,
>>> which is quite large.
>>> >>>
>>> >>> On the profiling sessions, the worst culprit was "char[]" arrays,
>>> mostly from the java_cup lexer. The Object[] and ArrayList[] were about
>>> 2-3% each,
>>> >>> for what the char[] was 90% (!)... Similarly, on profiling CPU, it
>>> was the IO operations on zzRefill within the java_cup scanner that took the
>>> largest chunk
>>> >>> of the time (27%). Smart scanning (e.g., lookahead) has taken only
>>> about 2-3%.
>>> >>>
>>> >>> I wonder... what was the profilling setup that you used to get to
>>> the creation of Object[] ArrayList as the main problem?
>>> >>> Although the change you refer to below shouldn't be relatively
>>> simple to change, as Petra pointed out.
>>> >>>
>>> >>> I just want to get the right picture to tackle such performance
>>> problem for larger specs.
>>> >>>
>>> >>> Best,
>>> >>> Leo
>>> >>>
>>> >>> On 5 Jan 2012, at 23:51, Tim Miller wrote:
>>> >>>
>>> >>>> Hi everyone,
>>> >>>>
>>> >>>> Anthony Hall and I have been discussing some memory problems that
>>> CZT
>>> >>>> has when parsing large specifications. Anthony has been trying to
>>> >>>> typecheck the iFacts specification, but without much luck due to the
>>> >>>> large memory resources.
>>> >>>>
>>> >>>> We've each been playing around with VisualVM, and Anthony pointed
>>> out is
>>> >>>> that the two largest memory hogs are Object[] and ArrayList, taking
>>> up
>>> >>>> around 23% and 10% of the heap respectively.
>>> >>>> Most of the object arrays and ArrayLists contain exactly 10 items,
>>> and
>>> >>>> almost all items in these lists are null.
>>> >>>>
>>> >>>> After some poking around, I discovered that when ArrayList is
>>> created
>>> >>>> using the default constructor, it allocates 10 items initially. This
>>> >>>> appears to be where the 10 items come from in each case. I suspect
>>> this
>>> >>>> may also be contributing to some of the memory problems, considering
>>> >>>> that CZT has so many empty annotation lists, etc. Creating
>>> ArrayLists
>>> >>>> with an initial capacity of 0 or 1 (using the constructor
>>> ArrayList(int
>>> >>>> initialCapacity)) may give us some substantial space savings
>>> >>>>
>>> >>>> It appears that most ArrayLists are created in the gnast-generated
>>> code.
>>> >>>> Petra, how difficult would it be to create these lists using this
>>> other
>>> >>>> constructor?
>>> >>>>
>>> >>>> Regards,
>>> >>>> Tim
>>> >>>>
>>> >>>>
>>> ------------------------------------------------------------------------------
>>> >>>> Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a
>>> complex
>>> >>>> infrastructure or vast IT resources to deliver seamless, secure
>>> access to
>>> >>>> virtual desktops. With this all-in-one solution, easily deploy
>>> virtual
>>> >>>> desktops for less than the cost of PCs and save 60% on VDI
>>> infrastructure
>>> >>>> costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox
>>> >>>> _______________________________________________
>>> >>>> CZT-Devel mailing list
>>> >>>> CZT...@li...
>>> >>>> https://lists.sourceforge.net/lists/listinfo/czt-devel
>>> >>>
>>> >>>
>>> >>>
>>> ------------------------------------------------------------------------------
>>> >>> Try before you buy = See our experts in action!
>>> >>> The most comprehensive online learning library for Microsoft
>>> developers
>>> >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
>>> MVC3,
>>> >>> Metro Style Apps, more. Free future releases when you subscribe now!
>>> >>> http://p.sf.net/sfu/learndevnow-dev2
>>> >>> _______________________________________________
>>> >>> CZT-Devel mailing list
>>> >>> CZT...@li...
>>> >>> https://lists.sourceforge.net/lists/listinfo/czt-devel
>>> >>
>>> >>
>>> >
>>> >
>>> >
>>> ------------------------------------------------------------------------------
>>> > Keep Your Developer Skills Current with LearnDevNow!
>>> > The most comprehensive online learning library for Microsoft developers
>>> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
>>> MVC3,
>>> > Metro Style Apps, more. Free future releases when you subscribe now!
>>> > http://p.sf.net/sfu/learndevnow-d2d
>>> > _______________________________________________
>>> > CZT-Devel mailing list
>>> > CZT...@li...
>>> > https://lists.sourceforge.net/lists/listinfo/czt-devel
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Keep Your Developer Skills Current with LearnDevNow!
>>> The most comprehensive online learning library for Microsoft developers
>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
>>> Metro Style Apps, more. Free future releases when you subscribe now!
>>> http://p.sf.net/sfu/learndevnow-d2d
>>> _______________________________________________
>>> CZT-Devel mailing list
>>> CZT...@li...
>>> https://lists.sourceforge.net/lists/listinfo/czt-devel
>>>
>>
>>
>>
>
>
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> CZT-Devel mailing list
> CZT...@li...
> https://lists.sourceforge.net/lists/listinfo/czt-devel
>
>

Re: [CZT-Devel] Memory allocation with ArrayLists; arrays; etc.

[Andrius V. <and...@ne...>]

Yes, the big integers will get large. But in your previous e-mail you said
"given most of them are shared"? So I wanted to object, that I think they
are not actually shared.

-16..16 comes from BigInteger.valueOf() method - check the Java sources. I
wanted to say that the only sharing of BigInteger objects is in this range..

So I just wanted to say that BigIntegers do consitute a significant part of
CZT memory usage :)

Andrius


On Fri, Apr 12, 2013 at 4:28 PM, Leo Freitas <leo...@ne...>wrote:

>  Hi Andrius,
>
>  I don't think so. The big integers are used for the line / column /
> buffer length numbers no?
> So if you have  large files they will get large. Or am I mistaken? Where
> did the -16..16 came from?
>
>  Leo
>
>  On 12 Apr 2013, at 09:12, Andrius Velykis <
> and...@ne...> wrote:
>
>  Hi Leo,
>
>  (reviving and old post..)
>
>   As is, the only major problem with memory consumption is the
>> duplication (e.g., ArrayList/Object[]).
>> The presence of various (4-6) BigIntegers in LocAnn is also an issue, but
>> given most of them are shared
>> it's not such a big deal.
>>
>
>  From what I gather, BigIntegers are only shared when in the range of
> [-16, 16] -- looking at BigInteger.valueOf()? Or am I mistaken? If so, most
> of the BigIntegers would not be shared in CZT..
>
>  Andrius
>
>
> On Wed, Jan 25, 2012 at 6:33 AM, Leo Freitas <leo...@ne...>wrote:
>
>> Hi Tim,
>>
>> Yes, after parsing. And yes, I found that rather odd too. There is
>> something funny going on
>> in Garbage Collection somewhere. I guess the trouble is that there are
>> certain structures
>> (e.g., TokenSequence iterators) that end up retaining the biggest part of
>> the object memory.
>>
>> Having said that, I was now trying to find the sources of the problem(s)
>> using more intrusive
>> snapshot points (AKA memory dumps at particular execution points).
>>
>> As is, the only major problem with memory consumption is the duplication
>> (e.g., ArrayList/Object[]).
>> The presence of various (4-6) BigIntegers in LocAnn is also an issue, but
>> given most of them are shared
>> it's not such a big deal.
>>
>> I've tweaked the PerformanceSettings constant(s) a little bit more to
>> keep arraylists/object[] capacity 0 at creation,
>> and that just about halved the memory used (!)... possibly at some speed
>> expense given it will take some time to
>> increase capacity.
>>
>> At the profiling sessions done, this wasn't a problem: I've put initial
>> capacity at 0, 1, and 10 (e.g., only very few go
>> beyond that, mainly on type info), and the CPU increase was marginal
>> (e,g., with 10 it was 300ms - 2.75% - quicker; with 1 100ms quicker),
>> yet  the memory gain was significant (e.g., with 10 and 1 it was it was
>> twice the heap at 100MB instead of 50MB for 0).
>>
>> One nice thing would be to have this as a configuration file or a
>> SectionManager option.... once all key bottleneck points are identified.
>> By default I will keep it at 0, as the performance penalty in CPU time
>> seems marginal/acceptable (e.g., 2-3% penalty for halving memory needs).
>>
>> Suggestions / comments?
>>
>> Best,
>> Leo
>>
>> On 24 Jan 2012, at 23:11, Tim Miller wrote:
>>
>> > Nice work!
>> >
>> > When you say you added "parser = null", I assume you mean after you do
>> > the parsing? This would mean that, prior to your change, a reference to
>> > the parser is being kept even after the variable scope of the variable
>> > "parser" ends? That seems odd.
>> >
>> > On 25/01/12 01:22, Leo Freitas wrote:
>> >> Tim,
>> >>
>> >> I've forgot to mention below that I was talking about parsing and
>> typechecking only.
>> >>
>> >> I've just run the same profiling setup on the VCG for these larger
>> examples and yes, the worst ones were
>> >> Object[] (24%), char[] (17.3%), ArrayList[] (9%) and BigInteger (2-3%).
>> >>
>> >> Surprising ones were int[] (6.5%) and short[] (6%) arrays, since they
>> are not explicitly created anywhere in CZT.
>> >> LocalAnn took 3%; Iterators were also a surprise: both iterator() and
>> listIterator() cover about 10%, where more
>> >> obvious ones like String (0.9%) and ZName (1.7%) were quite low.
>> >>
>> >> I've run the profiling sessions about 3-5 times on each example taking
>> the average.
>> >>
>> >> ----
>> >>
>> >> Searching for potential sources of such unexpected types, I managed to
>> find a few successful candidates:
>> >>
>> >> a) ParseUtils
>> >>
>> >> char[], short[], and int[] are mostly present in the Java CUP
>> low-level classes to represent the internal parsing tables from the grammar.
>> >> I simply added an explicit "parser = null" to the main
>> ParseUtils.parse method. These arrays combined account for about 30% of
>> memory.
>> >>
>> >> after the change, these arrays now account for 1.1% (!!!) that's quite
>> an improvement.
>> >>
>> >> b) ListIterator and Iterator
>> >>
>> >> These appear in various places, but places that should influence all
>> runs are only on SmartScanner (and possibly PrettyPrinter).
>> >> After this change, they went from a combined (ListIterator + Iterator)
>> 49% footprint to 17%; another good improvement.
>> >>
>> >> d) ArrayList and Object[]
>> >>
>> >> There are various places and identifying where are the ones of most
>> significance is tedious/time-consuming.
>> >> Instead, I've done a thorough search through various projects and
>> changed default constructors to more sensible values, when possible,
>> >> or to a parameterised default when not (e.g., PerformanceSettings
>> interface in util project).
>> >>
>> >> This led to a decrease in the memory footprint of these objects of....
>> >>
>> >> e) BigInteger
>> >>
>> >> Why do we need big integers within LocAnn and other places? I guess
>> because long would be potentially small?
>> >> But would there really be something bigger than 2^32 or 2^64 in number
>> of lines of source in a file say? Hum...
>> >>
>> >> In terms of memory footprint, it varied between 0.7% to 3%, depending
>> on the example. I won't change this one for now.
>> >>
>> >> f) Garbage collection time
>> >>
>> >> Firstly it was about 16-20% now it was about 46.1% of time taken.
>> Don't know if that's related to changes, but looks like it.
>> >>
>> >> ======
>> >>
>> >> Changes a) and b) led to a drastic memory footprint decrease from 8MB
>> to 0.91MB;
>> >> Other changes led to a minor improvement in comparison (i.e., 0.80MB).
>> >>
>> >> That's despite the fact that changes in d) were the most numerous -
>> they didn't amount to much improvement, but some.
>> >>
>> >> Hopefully this will enable much better performance on larger specs. I
>> am committing it now to see how it goes.
>> >>
>> >> Best,
>> >> Leo
>> >>
>> >> On 23 Jan 2012, at 14:00, Leo Freitas wrote:
>> >>
>> >>> Hi Tim,
>> >>>
>> >>> That's interesting. I';ve been doing similar (profiling) tests over
>> specs of some size (e.g., Mondex, Tokeneer, Xenon, IEEE float point unit);
>> >>> although I guess they are smaller than iFACTS - apart from Xenon,
>> which is quite large.
>> >>>
>> >>> On the profiling sessions, the worst culprit was "char[]" arrays,
>> mostly from the java_cup lexer. The Object[] and ArrayList[] were about
>> 2-3% each,
>> >>> for what the char[] was 90% (!)... Similarly, on profiling CPU, it
>> was the IO operations on zzRefill within the java_cup scanner that took the
>> largest chunk
>> >>> of the time (27%). Smart scanning (e.g., lookahead) has taken only
>> about 2-3%.
>> >>>
>> >>> I wonder... what was the profilling setup that you used to get to the
>> creation of Object[] ArrayList as the main problem?
>> >>> Although the change you refer to below shouldn't be relatively simple
>> to change, as Petra pointed out.
>> >>>
>> >>> I just want to get the right picture to tackle such performance
>> problem for larger specs.
>> >>>
>> >>> Best,
>> >>> Leo
>> >>>
>> >>> On 5 Jan 2012, at 23:51, Tim Miller wrote:
>> >>>
>> >>>> Hi everyone,
>> >>>>
>> >>>> Anthony Hall and I have been discussing some memory problems that CZT
>> >>>> has when parsing large specifications. Anthony has been trying to
>> >>>> typecheck the iFacts specification, but without much luck due to the
>> >>>> large memory resources.
>> >>>>
>> >>>> We've each been playing around with VisualVM, and Anthony pointed
>> out is
>> >>>> that the two largest memory hogs are Object[] and ArrayList, taking
>> up
>> >>>> around 23% and 10% of the heap respectively.
>> >>>> Most of the object arrays and ArrayLists contain exactly 10 items,
>> and
>> >>>> almost all items in these lists are null.
>> >>>>
>> >>>> After some poking around, I discovered that when ArrayList is created
>> >>>> using the default constructor, it allocates 10 items initially. This
>> >>>> appears to be where the 10 items come from in each case. I suspect
>> this
>> >>>> may also be contributing to some of the memory problems, considering
>> >>>> that CZT has so many empty annotation lists, etc. Creating ArrayLists
>> >>>> with an initial capacity of 0 or 1 (using the constructor
>> ArrayList(int
>> >>>> initialCapacity)) may give us some substantial space savings
>> >>>>
>> >>>> It appears that most ArrayLists are created in the gnast-generated
>> code.
>> >>>> Petra, how difficult would it be to create these lists using this
>> other
>> >>>> constructor?
>> >>>>
>> >>>> Regards,
>> >>>> Tim
>> >>>>
>> >>>>
>> ------------------------------------------------------------------------------
>> >>>> Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a
>> complex
>> >>>> infrastructure or vast IT resources to deliver seamless, secure
>> access to
>> >>>> virtual desktops. With this all-in-one solution, easily deploy
>> virtual
>> >>>> desktops for less than the cost of PCs and save 60% on VDI
>> infrastructure
>> >>>> costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox
>> >>>> _______________________________________________
>> >>>> CZT-Devel mailing list
>> >>>> CZT...@li...
>> >>>> https://lists.sourceforge.net/lists/listinfo/czt-devel
>> >>>
>> >>>
>> >>>
>> ------------------------------------------------------------------------------
>> >>> Try before you buy = See our experts in action!
>> >>> The most comprehensive online learning library for Microsoft
>> developers
>> >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
>> MVC3,
>> >>> Metro Style Apps, more. Free future releases when you subscribe now!
>> >>> http://p.sf.net/sfu/learndevnow-dev2
>> >>> _______________________________________________
>> >>> CZT-Devel mailing list
>> >>> CZT...@li...
>> >>> https://lists.sourceforge.net/lists/listinfo/czt-devel
>> >>
>> >>
>> >
>> >
>> >
>> ------------------------------------------------------------------------------
>> > Keep Your Developer Skills Current with LearnDevNow!
>> > The most comprehensive online learning library for Microsoft developers
>> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
>> > Metro Style Apps, more. Free future releases when you subscribe now!
>> > http://p.sf.net/sfu/learndevnow-d2d
>> > _______________________________________________
>> > CZT-Devel mailing list
>> > CZT...@li...
>> > https://lists.sourceforge.net/lists/listinfo/czt-devel
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Keep Your Developer Skills Current with LearnDevNow!
>> The most comprehensive online learning library for Microsoft developers
>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
>> Metro Style Apps, more. Free future releases when you subscribe now!
>> http://p.sf.net/sfu/learndevnow-d2d
>> _______________________________________________
>> CZT-Devel mailing list
>> CZT...@li...
>> https://lists.sourceforge.net/lists/listinfo/czt-devel
>>
>
>
>

Re: [CZT-Devel] Memory allocation with ArrayLists; arrays; etc.

[Leo F. <leo...@ne...>]

Hi Andrius,

I don't think so. The big integers are used for the line / column / buffer length numbers no?
So if you have  large files they will get large. Or am I mistaken? Where did the -16..16 came from?

Leo

On 12 Apr 2013, at 09:12, Andrius Velykis <and...@ne...<mailto:and...@ne...>> wrote:

Hi Leo,

(reviving and old post..)

As is, the only major problem with memory consumption is the duplication (e.g., ArrayList/Object[]).
The presence of various (4-6) BigIntegers in LocAnn is also an issue, but given most of them are shared
it's not such a big deal.

>From what I gather, BigIntegers are only shared when in the range of [-16, 16] -- looking at BigInteger.valueOf()? Or am I mistaken? If so, most of the BigIntegers would not be shared in CZT..

Andrius


On Wed, Jan 25, 2012 at 6:33 AM, Leo Freitas <leo...@ne...<mailto:leo...@ne...>> wrote:
Hi Tim,

Yes, after parsing. And yes, I found that rather odd too. There is something funny going on
in Garbage Collection somewhere. I guess the trouble is that there are certain structures
(e.g., TokenSequence iterators) that end up retaining the biggest part of the object memory.

Having said that, I was now trying to find the sources of the problem(s) using more intrusive
snapshot points (AKA memory dumps at particular execution points).

As is, the only major problem with memory consumption is the duplication (e.g., ArrayList/Object[]).
The presence of various (4-6) BigIntegers in LocAnn is also an issue, but given most of them are shared
it's not such a big deal.

I've tweaked the PerformanceSettings constant(s) a little bit more to keep arraylists/object[] capacity 0 at creation,
and that just about halved the memory used (!)... possibly at some speed expense given it will take some time to
increase capacity.

At the profiling sessions done, this wasn't a problem: I've put initial capacity at 0, 1, and 10 (e.g., only very few go
beyond that, mainly on type info), and the CPU increase was marginal (e,g., with 10 it was 300ms - 2.75% - quicker; with 1 100ms quicker),
yet  the memory gain was significant (e.g., with 10 and 1 it was it was twice the heap at 100MB instead of 50MB for 0).

One nice thing would be to have this as a configuration file or a SectionManager option.... once all key bottleneck points are identified.
By default I will keep it at 0, as the performance penalty in CPU time seems marginal/acceptable (e.g., 2-3% penalty for halving memory needs).

Suggestions / comments?

Best,
Leo

On 24 Jan 2012, at 23:11, Tim Miller wrote:

> Nice work!
>
> When you say you added "parser = null", I assume you mean after you do
> the parsing? This would mean that, prior to your change, a reference to
> the parser is being kept even after the variable scope of the variable
> "parser" ends? That seems odd.
>
> On 25/01/12 01:22, Leo Freitas wrote:
>> Tim,
>>
>> I've forgot to mention below that I was talking about parsing and typechecking only.
>>
>> I've just run the same profiling setup on the VCG for these larger examples and yes, the worst ones were
>> Object[] (24%), char[] (17.3%), ArrayList[] (9%) and BigInteger (2-3%).
>>
>> Surprising ones were int[] (6.5%) and short[] (6%) arrays, since they are not explicitly created anywhere in CZT.
>> LocalAnn took 3%; Iterators were also a surprise: both iterator() and listIterator() cover about 10%, where more
>> obvious ones like String (0.9%) and ZName (1.7%) were quite low.
>>
>> I've run the profiling sessions about 3-5 times on each example taking the average.
>>
>> ----
>>
>> Searching for potential sources of such unexpected types, I managed to find a few successful candidates:
>>
>> a) ParseUtils
>>
>> char[], short[], and int[] are mostly present in the Java CUP low-level classes to represent the internal parsing tables from the grammar.
>> I simply added an explicit "parser = null" to the main ParseUtils.parse method. These arrays combined account for about 30% of memory.
>>
>> after the change, these arrays now account for 1.1% (!!!) that's quite an improvement.
>>
>> b) ListIterator and Iterator
>>
>> These appear in various places, but places that should influence all runs are only on SmartScanner (and possibly PrettyPrinter).
>> After this change, they went from a combined (ListIterator + Iterator) 49% footprint to 17%; another good improvement.
>>
>> d) ArrayList and Object[]
>>
>> There are various places and identifying where are the ones of most significance is tedious/time-consuming.
>> Instead, I've done a thorough search through various projects and changed default constructors to more sensible values, when possible,
>> or to a parameterised default when not (e.g., PerformanceSettings interface in util project).
>>
>> This led to a decrease in the memory footprint of these objects of....
>>
>> e) BigInteger
>>
>> Why do we need big integers within LocAnn and other places? I guess because long would be potentially small?
>> But would there really be something bigger than 2^32 or 2^64 in number of lines of source in a file say? Hum...
>>
>> In terms of memory footprint, it varied between 0.7% to 3%, depending on the example. I won't change this one for now.
>>
>> f) Garbage collection time
>>
>> Firstly it was about 16-20% now it was about 46.1% of time taken. Don't know if that's related to changes, but looks like it.
>>
>> ======
>>
>> Changes a) and b) led to a drastic memory footprint decrease from 8MB to 0.91MB;
>> Other changes led to a minor improvement in comparison (i.e., 0.80MB).
>>
>> That's despite the fact that changes in d) were the most numerous - they didn't amount to much improvement, but some.
>>
>> Hopefully this will enable much better performance on larger specs. I am committing it now to see how it goes.
>>
>> Best,
>> Leo
>>
>> On 23 Jan 2012, at 14:00, Leo Freitas wrote:
>>
>>> Hi Tim,
>>>
>>> That's interesting. I';ve been doing similar (profiling) tests over specs of some size (e.g., Mondex, Tokeneer, Xenon, IEEE float point unit);
>>> although I guess they are smaller than iFACTS - apart from Xenon, which is quite large.
>>>
>>> On the profiling sessions, the worst culprit was "char[]" arrays, mostly from the java_cup lexer. The Object[] and ArrayList[] were about 2-3% each,
>>> for what the char[] was 90% (!)... Similarly, on profiling CPU, it was the IO operations on zzRefill within the java_cup scanner that took the largest chunk
>>> of the time (27%). Smart scanning (e.g., lookahead) has taken only about 2-3%.
>>>
>>> I wonder... what was the profilling setup that you used to get to the creation of Object[] ArrayList as the main problem?
>>> Although the change you refer to below shouldn't be relatively simple to change, as Petra pointed out.
>>>
>>> I just want to get the right picture to tackle such performance problem for larger specs.
>>>
>>> Best,
>>> Leo
>>>
>>> On 5 Jan 2012, at 23:51, Tim Miller wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> Anthony Hall and I have been discussing some memory problems that CZT
>>>> has when parsing large specifications. Anthony has been trying to
>>>> typecheck the iFacts specification, but without much luck due to the
>>>> large memory resources.
>>>>
>>>> We've each been playing around with VisualVM, and Anthony pointed out is
>>>> that the two largest memory hogs are Object[] and ArrayList, taking up
>>>> around 23% and 10% of the heap respectively.
>>>> Most of the object arrays and ArrayLists contain exactly 10 items, and
>>>> almost all items in these lists are null.
>>>>
>>>> After some poking around, I discovered that when ArrayList is created
>>>> using the default constructor, it allocates 10 items initially. This
>>>> appears to be where the 10 items come from in each case. I suspect this
>>>> may also be contributing to some of the memory problems, considering
>>>> that CZT has so many empty annotation lists, etc. Creating ArrayLists
>>>> with an initial capacity of 0 or 1 (using the constructor ArrayList(int
>>>> initialCapacity)) may give us some substantial space savings
>>>>
>>>> It appears that most ArrayLists are created in the gnast-generated code.
>>>> Petra, how difficult would it be to create these lists using this other
>>>> constructor?
>>>>
>>>> Regards,
>>>> Tim
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a complex
>>>> infrastructure or vast IT resources to deliver seamless, secure access to
>>>> virtual desktops. With this all-in-one solution, easily deploy virtual
>>>> desktops for less than the cost of PCs and save 60% on VDI infrastructure
>>>> costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox
>>>> _______________________________________________
>>>> CZT-Devel mailing list
>>>> CZT...@li...<mailto:CZT...@li...>
>>>> https://lists.sourceforge.net/lists/listinfo/czt-devel
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Try before you buy = See our experts in action!
>>> The most comprehensive online learning library for Microsoft developers
>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
>>> Metro Style Apps, more. Free future releases when you subscribe now!
>>> http://p.sf.net/sfu/learndevnow-dev2
>>> _______________________________________________
>>> CZT-Devel mailing list
>>> CZT...@li...<mailto:CZT...@li...>
>>> https://lists.sourceforge.net/lists/listinfo/czt-devel
>>
>>
>
>
> ------------------------------------------------------------------------------
> Keep Your Developer Skills Current with LearnDevNow!
> The most comprehensive online learning library for Microsoft developers
> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> Metro Style Apps, more. Free future releases when you subscribe now!
> http://p.sf.net/sfu/learndevnow-d2d
> _______________________________________________
> CZT-Devel mailing list
> CZT...@li...<mailto:CZT...@li...>
> https://lists.sourceforge.net/lists/listinfo/czt-devel


------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
CZT-Devel mailing list
CZT...@li...<mailto:CZT...@li...>
https://lists.sourceforge.net/lists/listinfo/czt-devel

[CZT-Devel] Reducing memory usage: Signatures are the biggest culprit?

[Andrius V. <and...@ne...>]

Hi,

Since I had a profiler on CZT running for catching some leftover debugging
code, I thought I would give a quick look into the CZT memory usage as well.

I have discovered that most of the large amount of ZNames that are created
are referenced from typechecker Signatures (via NameTypePairs). This is the
case because when creating type signatures during typechecking, the
Signatures are duplicated in a lot of places.

A prominent example of this is in
net.sourceforge.czt.typechecker.z.ExprChecker:94 (method visitRefExpr).
This bit of code calculates the type of a RefExpr. If a reference is to a
Schema, it duplicates the Schema signature (assigns new IDs to all its
names) and then uses the new signature within the power type.

As a quick test, I tried removing this duplication and instead reuse the
original schema signature (with original ZName ids): `Signature sig =
signature;`

This single change reduced the memory consumption of typechecked `spec.tex`
by about 60%!

This lead me to thinking - do signatures really need to be duplicated
everywhere? I imagine that the signature of schema A would be the same
everywhere? Can we reuse the signature, or is it important to duplicate and
assign new IDs to the name?

The massive space saving becomes quite obvious when you think about it. If
we have schema references, every RefExpr would duplicate the whole schema
definition when typechecked, hence creating a lot of new ZName instances
and consuming all this memory.

Leo advised that Tim would be the person to ask about the Signature
duplication in typechecker? Is it really necessary, or can we reuse the
objects?

Best regards,
Andrius

Re: [CZT-Devel] Memory allocation with ArrayLists; arrays; etc.

[Andrius V. <and...@ne...>]

Hi Leo,

(reviving and old post..)

As is, the only major problem with memory consumption is the duplication
> (e.g., ArrayList/Object[]).
> The presence of various (4-6) BigIntegers in LocAnn is also an issue, but
> given most of them are shared
> it's not such a big deal.
>

>From what I gather, BigIntegers are only shared when in the range of [-16,
16] -- looking at BigInteger.valueOf()? Or am I mistaken? If so, most of
the BigIntegers would not be shared in CZT..

Andrius


On Wed, Jan 25, 2012 at 6:33 AM, Leo Freitas <leo...@ne...>wrote:

> Hi Tim,
>
> Yes, after parsing. And yes, I found that rather odd too. There is
> something funny going on
> in Garbage Collection somewhere. I guess the trouble is that there are
> certain structures
> (e.g., TokenSequence iterators) that end up retaining the biggest part of
> the object memory.
>
> Having said that, I was now trying to find the sources of the problem(s)
> using more intrusive
> snapshot points (AKA memory dumps at particular execution points).
>
> As is, the only major problem with memory consumption is the duplication
> (e.g., ArrayList/Object[]).
> The presence of various (4-6) BigIntegers in LocAnn is also an issue, but
> given most of them are shared
> it's not such a big deal.
>
> I've tweaked the PerformanceSettings constant(s) a little bit more to keep
> arraylists/object[] capacity 0 at creation,
> and that just about halved the memory used (!)... possibly at some speed
> expense given it will take some time to
> increase capacity.
>
> At the profiling sessions done, this wasn't a problem: I've put initial
> capacity at 0, 1, and 10 (e.g., only very few go
> beyond that, mainly on type info), and the CPU increase was marginal
> (e,g., with 10 it was 300ms - 2.75% - quicker; with 1 100ms quicker),
> yet  the memory gain was significant (e.g., with 10 and 1 it was it was
> twice the heap at 100MB instead of 50MB for 0).
>
> One nice thing would be to have this as a configuration file or a
> SectionManager option.... once all key bottleneck points are identified.
> By default I will keep it at 0, as the performance penalty in CPU time
> seems marginal/acceptable (e.g., 2-3% penalty for halving memory needs).
>
> Suggestions / comments?
>
> Best,
> Leo
>
> On 24 Jan 2012, at 23:11, Tim Miller wrote:
>
> > Nice work!
> >
> > When you say you added "parser = null", I assume you mean after you do
> > the parsing? This would mean that, prior to your change, a reference to
> > the parser is being kept even after the variable scope of the variable
> > "parser" ends? That seems odd.
> >
> > On 25/01/12 01:22, Leo Freitas wrote:
> >> Tim,
> >>
> >> I've forgot to mention below that I was talking about parsing and
> typechecking only.
> >>
> >> I've just run the same profiling setup on the VCG for these larger
> examples and yes, the worst ones were
> >> Object[] (24%), char[] (17.3%), ArrayList[] (9%) and BigInteger (2-3%).
> >>
> >> Surprising ones were int[] (6.5%) and short[] (6%) arrays, since they
> are not explicitly created anywhere in CZT.
> >> LocalAnn took 3%; Iterators were also a surprise: both iterator() and
> listIterator() cover about 10%, where more
> >> obvious ones like String (0.9%) and ZName (1.7%) were quite low.
> >>
> >> I've run the profiling sessions about 3-5 times on each example taking
> the average.
> >>
> >> ----
> >>
> >> Searching for potential sources of such unexpected types, I managed to
> find a few successful candidates:
> >>
> >> a) ParseUtils
> >>
> >> char[], short[], and int[] are mostly present in the Java CUP low-level
> classes to represent the internal parsing tables from the grammar.
> >> I simply added an explicit "parser = null" to the main ParseUtils.parse
> method. These arrays combined account for about 30% of memory.
> >>
> >> after the change, these arrays now account for 1.1% (!!!) that's quite
> an improvement.
> >>
> >> b) ListIterator and Iterator
> >>
> >> These appear in various places, but places that should influence all
> runs are only on SmartScanner (and possibly PrettyPrinter).
> >> After this change, they went from a combined (ListIterator + Iterator)
> 49% footprint to 17%; another good improvement.
> >>
> >> d) ArrayList and Object[]
> >>
> >> There are various places and identifying where are the ones of most
> significance is tedious/time-consuming.
> >> Instead, I've done a thorough search through various projects and
> changed default constructors to more sensible values, when possible,
> >> or to a parameterised default when not (e.g., PerformanceSettings
> interface in util project).
> >>
> >> This led to a decrease in the memory footprint of these objects of....
> >>
> >> e) BigInteger
> >>
> >> Why do we need big integers within LocAnn and other places? I guess
> because long would be potentially small?
> >> But would there really be something bigger than 2^32 or 2^64 in number
> of lines of source in a file say? Hum...
> >>
> >> In terms of memory footprint, it varied between 0.7% to 3%, depending
> on the example. I won't change this one for now.
> >>
> >> f) Garbage collection time
> >>
> >> Firstly it was about 16-20% now it was about 46.1% of time taken. Don't
> know if that's related to changes, but looks like it.
> >>
> >> ======
> >>
> >> Changes a) and b) led to a drastic memory footprint decrease from 8MB
> to 0.91MB;
> >> Other changes led to a minor improvement in comparison (i.e., 0.80MB).
> >>
> >> That's despite the fact that changes in d) were the most numerous -
> they didn't amount to much improvement, but some.
> >>
> >> Hopefully this will enable much better performance on larger specs. I
> am committing it now to see how it goes.
> >>
> >> Best,
> >> Leo
> >>
> >> On 23 Jan 2012, at 14:00, Leo Freitas wrote:
> >>
> >>> Hi Tim,
> >>>
> >>> That's interesting. I';ve been doing similar (profiling) tests over
> specs of some size (e.g., Mondex, Tokeneer, Xenon, IEEE float point unit);
> >>> although I guess they are smaller than iFACTS - apart from Xenon,
> which is quite large.
> >>>
> >>> On the profiling sessions, the worst culprit was "char[]" arrays,
> mostly from the java_cup lexer. The Object[] and ArrayList[] were about
> 2-3% each,
> >>> for what the char[] was 90% (!)... Similarly, on profiling CPU, it was
> the IO operations on zzRefill within the java_cup scanner that took the
> largest chunk
> >>> of the time (27%). Smart scanning (e.g., lookahead) has taken only
> about 2-3%.
> >>>
> >>> I wonder... what was the profilling setup that you used to get to the
> creation of Object[] ArrayList as the main problem?
> >>> Although the change you refer to below shouldn't be relatively simple
> to change, as Petra pointed out.
> >>>
> >>> I just want to get the right picture to tackle such performance
> problem for larger specs.
> >>>
> >>> Best,
> >>> Leo
> >>>
> >>> On 5 Jan 2012, at 23:51, Tim Miller wrote:
> >>>
> >>>> Hi everyone,
> >>>>
> >>>> Anthony Hall and I have been discussing some memory problems that CZT
> >>>> has when parsing large specifications. Anthony has been trying to
> >>>> typecheck the iFacts specification, but without much luck due to the
> >>>> large memory resources.
> >>>>
> >>>> We've each been playing around with VisualVM, and Anthony pointed out
> is
> >>>> that the two largest memory hogs are Object[] and ArrayList, taking up
> >>>> around 23% and 10% of the heap respectively.
> >>>> Most of the object arrays and ArrayLists contain exactly 10 items, and
> >>>> almost all items in these lists are null.
> >>>>
> >>>> After some poking around, I discovered that when ArrayList is created
> >>>> using the default constructor, it allocates 10 items initially. This
> >>>> appears to be where the 10 items come from in each case. I suspect
> this
> >>>> may also be contributing to some of the memory problems, considering
> >>>> that CZT has so many empty annotation lists, etc. Creating ArrayLists
> >>>> with an initial capacity of 0 or 1 (using the constructor
> ArrayList(int
> >>>> initialCapacity)) may give us some substantial space savings
> >>>>
> >>>> It appears that most ArrayLists are created in the gnast-generated
> code.
> >>>> Petra, how difficult would it be to create these lists using this
> other
> >>>> constructor?
> >>>>
> >>>> Regards,
> >>>> Tim
> >>>>
> >>>>
> ------------------------------------------------------------------------------
> >>>> Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a
> complex
> >>>> infrastructure or vast IT resources to deliver seamless, secure
> access to
> >>>> virtual desktops. With this all-in-one solution, easily deploy virtual
> >>>> desktops for less than the cost of PCs and save 60% on VDI
> infrastructure
> >>>> costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox
> >>>> _______________________________________________
> >>>> CZT-Devel mailing list
> >>>> CZT...@li...
> >>>> https://lists.sourceforge.net/lists/listinfo/czt-devel
> >>>
> >>>
> >>>
> ------------------------------------------------------------------------------
> >>> Try before you buy = See our experts in action!
> >>> The most comprehensive online learning library for Microsoft developers
> >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
> MVC3,
> >>> Metro Style Apps, more. Free future releases when you subscribe now!
> >>> http://p.sf.net/sfu/learndevnow-dev2
> >>> _______________________________________________
> >>> CZT-Devel mailing list
> >>> CZT...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/czt-devel
> >>
> >>
> >
> >
> >
> ------------------------------------------------------------------------------
> > Keep Your Developer Skills Current with LearnDevNow!
> > The most comprehensive online learning library for Microsoft developers
> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> > Metro Style Apps, more. Free future releases when you subscribe now!
> > http://p.sf.net/sfu/learndevnow-d2d
> > _______________________________________________
> > CZT-Devel mailing list
> > CZT...@li...
> > https://lists.sourceforge.net/lists/listinfo/czt-devel
>
>
>
> ------------------------------------------------------------------------------
> Keep Your Developer Skills Current with LearnDevNow!
> The most comprehensive online learning library for Microsoft developers
> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> Metro Style Apps, more. Free future releases when you subscribe now!
> http://p.sf.net/sfu/learndevnow-d2d
> _______________________________________________
> CZT-Devel mailing list
> CZT...@li...
> https://lists.sourceforge.net/lists/listinfo/czt-devel
>

Re: [CZT-Devel] New CZT website and nightly builds

[Leo F. <leo...@ne...>]

Hi andrius,

This is brilliant! Many thanks for all your hard work on not only website and nightly builds, but also a bunch of other things like full eclipse integration, and many helpful discussions and shared coding on zeves integration, Isabelle integration , memory management, section transactions etc...

And all tat whilst doing your PhD!!! Which is well on its way for a great success.

Many congratulations, and thanks for the huge effort on Keeping CZT  ;-)

Best
Leo

On 10 Apr 2013, at 09:43, "Andrius Velykis" <and...@ne...<mailto:and...@ne...>> wrote:

Hi CZT developers,

I wanted to give some quick updates about the new CZT website and nightly CZT builds.

Yesterday night I have pushed the new CZT website live: http://czt.sourceforge.net
It is generated and updated automatically every night from the CZT source. Let me know what you think.

Also, the CZT repository is built every night and all the build results are automatically uploaded to the SourceForge release system (check out `/nightly` subdirectories in the downloads: http://sourceforge.net/projects/czt/files/). These include the classic `czt.jar` as well as new Eclipse-based CZT IDE, CZT Eclipse update site and CZT jEdit plugins. Furthermore, the new CZT release will eventually be available in Maven Central, and the nightly -SNAPSHOT builds are currently pushed to Sonatype OSS Maven repo.


## The why and how

We are still "pending" on the new release, but there are quite some updates and tools in CZT that users may want to try. To accommodate that and avoid sitting on the updates for years, I thought we should transition into a more agile deployment. To avoid long-out-of-date information, maybe its better to provide access to the latest goodies from CZT in the form of nightly builds and website updates.

We have a build server running here at Newcastle that performs nightly CZT builds (it is an internal one and I doubt it will be accessible from outside). The build process pushes nightly releases and website during the build as explained above.

To allow that, I have upgraded CZT Maven builds so that everything should go through with standard Maven commands without too much. This includes building Eclipse plugins/products via Maven and using Maven site for CZT website. I will try to add some page on "how to release" eventually..


## Contributing

One of the things holding up the new release is the lack of documentation. We still need to document things that have been added since the last release, and parts of the website are quite raw..

In addition to that, however, I would like to encourage contributing to CZT documentation and website. Some parts of the website have been updated, but others still have just the minimal information. You can update the main project pages by editing README.md files in each project directory (this becomes index.html in most cases during site generation). Otherwise the CZT website tries to follow standard Maven site conventions: edit/add the files in `src/site` subdirectory of each project. The *.md files are written in Markdown :)

When you push the changes to Git, they will become part of the website the next day!

---

By the way, Maven tells me that this year is the 10 year anniversary for CZT: <inceptionYear>2003</inceptionYear>? Congratulations everyone! It should be a good opportunity to get CZT 2.0 out of the door :)


Best regards,
Andrius
------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
CZT-Devel mailing list
CZT...@li...<mailto:CZT...@li...>
https://lists.sourceforge.net/lists/listinfo/czt-devel

Re: [CZT-Devel] New CZT website and nightly builds

[Tim M. <tm...@un...>]

Yes, seconded. The website looks great, and its nice to see some 
documentation coming back onto the website after we lost the stuff on 
the wiki a few years back. Nightly builds will be great too -- I'm a big 
fan of them in general.

Thanks for the excellent work.

Tim

On 10/04/13 19:26, Anthony Hall wrote:
> Andrius
>
> This is absolutely great – a huge step forward
>
> Thank you!
>
> Anthony
>
> *From:*Andrius Velykis [mailto:and...@ne...]
> *Sent:* 10 April 2013 09:44
> *To:* CZT-Devel
> *Subject:* [CZT-Devel] New CZT website and nightly builds
>
> Hi CZT developers,
>
> I wanted to give some quick updates about the new CZT website and
> nightly CZT builds.
>
> Yesterday night I have pushed the new CZT website live:
> http://czt.sourceforge.net
>
> It is generated and updated automatically every night from the CZT
> source. Let me know what you think.
>
> Also, the CZT repository is built every night and all the build results
> are automatically uploaded to the SourceForge release system (check out
> `/nightly` subdirectories in the downloads:
> http://sourceforge.net/projects/czt/files/). These include the classic
> `czt.jar` as well as new Eclipse-based CZT IDE, CZT Eclipse update site
> and CZT jEdit plugins. Furthermore, the new CZT release will eventually
> be available in Maven Central, and the nightly -SNAPSHOT builds are
> currently pushed to Sonatype OSS Maven repo.
>
> ## The why and how
>
> We are still "pending" on the new release, but there are quite some
> updates and tools in CZT that users may want to try. To accommodate that
> and avoid sitting on the updates for years, I thought we should
> transition into a more agile deployment. To avoid long-out-of-date
> information, maybe its better to provide access to the latest goodies
> from CZT in the form of nightly builds and website updates.
>
> We have a build server running here at Newcastle that performs nightly
> CZT builds (it is an internal one and I doubt it will be accessible from
> outside). The build process pushes nightly releases and website during
> the build as explained above.
>
> To allow that, I have upgraded CZT Maven builds so that everything
> should go through with standard Maven commands without too much. This
> includes building Eclipse plugins/products via Maven and using Maven
> site for CZT website. I will try to add some page on "how to release"
> eventually..
>
> ## Contributing
>
> One of the things holding up the new release is the lack of
> documentation. We still need to document things that have been added
> since the last release, and parts of the website are quite raw..
>
> In addition to that, however, I would like to encourage contributing to
> CZT documentation and website. Some parts of the website have been
> updated, but others still have just the minimal information. You can
> update the main project pages by editing README.md files in each project
> directory (this becomes index.html in most cases during site
> generation). Otherwise the CZT website tries to follow standard Maven
> site conventions: edit/add the files in `src/site` subdirectory of each
> project. The *.md files are written in Markdown :)
>
> When you push the changes to Git, they will become part of the website
> the next day!
>
> ---
>
> By the way, Maven tells me that this year is the 10 year anniversary for
> CZT: <inceptionYear>2003</inceptionYear>? Congratulations everyone! It
> should be a good opportunity to get CZT 2.0 out of the door :)
>
> Best regards,
>
> Andrius
>
>
>
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
>
>
>
> _______________________________________________
> CZT-Devel mailing list
> CZT...@li...
> https://lists.sourceforge.net/lists/listinfo/czt-devel
>

Re: [CZT-Devel] New CZT website and nightly builds

[Mark U. <ma...@cs...>]

Andrius,

Congratulations on getting these nightly beta versions pushed out.
Yes, we should definitely aim to get v2.0 out this year, to celebrate the
10th anniversary.  :-)

The new website looks really nice - much more modern!

I'm looking forward to trying out the new Eclipse features...

Mark



On 10 April 2013 18:43, Andrius Velykis <and...@ne...>wrote:

> Hi CZT developers,
>
> I wanted to give some quick updates about the new CZT website and nightly
> CZT builds.
>
> Yesterday night I have pushed the new CZT website live:
> http://czt.sourceforge.net
> It is generated and updated automatically every night from the CZT source.
> Let me know what you think.
>
> Also, the CZT repository is built every night and all the build results
> are automatically uploaded to the SourceForge release system (check out
> `/nightly` subdirectories in the downloads:
> http://sourceforge.net/projects/czt/files/). These include the classic
> `czt.jar` as well as new Eclipse-based CZT IDE, CZT Eclipse update site and
> CZT jEdit plugins. Furthermore, the new CZT release will eventually be
> available in Maven Central, and the nightly -SNAPSHOT builds are currently
> pushed to Sonatype OSS Maven repo.
>
>
> ## The why and how
>
> We are still "pending" on the new release, but there are quite some
> updates and tools in CZT that users may want to try. To accommodate that
> and avoid sitting on the updates for years, I thought we should transition
> into a more agile deployment. To avoid long-out-of-date information, maybe
> its better to provide access to the latest goodies from CZT in the form of
> nightly builds and website updates.
>
> We have a build server running here at Newcastle that performs nightly CZT
> builds (it is an internal one and I doubt it will be accessible from
> outside). The build process pushes nightly releases and website during the
> build as explained above.
>
> To allow that, I have upgraded CZT Maven builds so that everything should
> go through with standard Maven commands without too much. This includes
> building Eclipse plugins/products via Maven and using Maven site for CZT
> website. I will try to add some page on "how to release" eventually..
>
>
> ## Contributing
>
> One of the things holding up the new release is the lack of documentation.
> We still need to document things that have been added since the last
> release, and parts of the website are quite raw..
>
> In addition to that, however, I would like to encourage contributing to
> CZT documentation and website. Some parts of the website have been updated,
> but others still have just the minimal information. You can update the main
> project pages by editing README.md files in each project directory (this
> becomes index.html in most cases during site generation). Otherwise the CZT
> website tries to follow standard Maven site conventions: edit/add the files
> in `src/site` subdirectory of each project. The *.md files are written in
> Markdown :)
>
> When you push the changes to Git, they will become part of the website the
> next day!
>
> ---
>
> By the way, Maven tells me that this year is the 10 year anniversary for
> CZT: <inceptionYear>2003</inceptionYear>? Congratulations everyone! It
> should be a good opportunity to get CZT 2.0 out of the door :)
>
>
> Best regards,
> Andrius
>
>
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> CZT-Devel mailing list
> CZT...@li...
> https://lists.sourceforge.net/lists/listinfo/czt-devel
>
>

Re: [CZT-Devel] New CZT website and nightly builds

[Anthony H. <an...@an...>]

Andrius

 

This is absolutely great – a huge step forward

 

Thank you!

 

Anthony

 

From: Andrius Velykis [mailto:and...@ne...] 
Sent: 10 April 2013 09:44
To: CZT-Devel
Subject: [CZT-Devel] New CZT website and nightly builds

 

Hi CZT developers,

 

I wanted to give some quick updates about the new CZT website and nightly CZT builds.

 

Yesterday night I have pushed the new CZT website live: http://czt.sourceforge.net

It is generated and updated automatically every night from the CZT source. Let me know what you think.

 

Also, the CZT repository is built every night and all the build results are automatically uploaded to the SourceForge release system (check out `/nightly` subdirectories in the downloads: http://sourceforge.net/projects/czt/files/). These include the classic `czt.jar` as well as new Eclipse-based CZT IDE, CZT Eclipse update site and CZT jEdit plugins. Furthermore, the new CZT release will eventually be available in Maven Central, and the nightly -SNAPSHOT builds are currently pushed to Sonatype OSS Maven repo.

 

 

## The why and how

 

We are still "pending" on the new release, but there are quite some updates and tools in CZT that users may want to try. To accommodate that and avoid sitting on the updates for years, I thought we should transition into a more agile deployment. To avoid long-out-of-date information, maybe its better to provide access to the latest goodies from CZT in the form of nightly builds and website updates.

 

We have a build server running here at Newcastle that performs nightly CZT builds (it is an internal one and I doubt it will be accessible from outside). The build process pushes nightly releases and website during the build as explained above.

 

To allow that, I have upgraded CZT Maven builds so that everything should go through with standard Maven commands without too much. This includes building Eclipse plugins/products via Maven and using Maven site for CZT website. I will try to add some page on "how to release" eventually..

 

 

## Contributing

 

One of the things holding up the new release is the lack of documentation. We still need to document things that have been added since the last release, and parts of the website are quite raw..

 

In addition to that, however, I would like to encourage contributing to CZT documentation and website. Some parts of the website have been updated, but others still have just the minimal information. You can update the main project pages by editing README.md files in each project directory (this becomes index.html in most cases during site generation). Otherwise the CZT website tries to follow standard Maven site conventions: edit/add the files in `src/site` subdirectory of each project. The *.md files are written in Markdown :)

 

When you push the changes to Git, they will become part of the website the next day!

 

---

 

By the way, Maven tells me that this year is the 10 year anniversary for CZT: <inceptionYear>2003</inceptionYear>? Congratulations everyone! It should be a good opportunity to get CZT 2.0 out of the door :)

 

 

Best regards,

Andrius

[CZT-Devel] New CZT website and nightly builds

[Andrius V. <and...@ne...>]

Hi CZT developers,

I wanted to give some quick updates about the new CZT website and nightly
CZT builds.

Yesterday night I have pushed the new CZT website live:
http://czt.sourceforge.net
It is generated and updated automatically every night from the CZT source.
Let me know what you think.

Also, the CZT repository is built every night and all the build results are
automatically uploaded to the SourceForge release system (check out
`/nightly` subdirectories in the downloads:
http://sourceforge.net/projects/czt/files/). These include the classic
`czt.jar` as well as new Eclipse-based CZT IDE, CZT Eclipse update site and
CZT jEdit plugins. Furthermore, the new CZT release will eventually be
available in Maven Central, and the nightly -SNAPSHOT builds are currently
pushed to Sonatype OSS Maven repo.


## The why and how

We are still "pending" on the new release, but there are quite some updates
and tools in CZT that users may want to try. To accommodate that and avoid
sitting on the updates for years, I thought we should transition into a
more agile deployment. To avoid long-out-of-date information, maybe its
better to provide access to the latest goodies from CZT in the form of
nightly builds and website updates.

We have a build server running here at Newcastle that performs nightly CZT
builds (it is an internal one and I doubt it will be accessible from
outside). The build process pushes nightly releases and website during the
build as explained above.

To allow that, I have upgraded CZT Maven builds so that everything should
go through with standard Maven commands without too much. This includes
building Eclipse plugins/products via Maven and using Maven site for CZT
website. I will try to add some page on "how to release" eventually..


## Contributing

One of the things holding up the new release is the lack of documentation.
We still need to document things that have been added since the last
release, and parts of the website are quite raw..

In addition to that, however, I would like to encourage contributing to CZT
documentation and website. Some parts of the website have been updated, but
others still have just the minimal information. You can update the main
project pages by editing README.md files in each project directory (this
becomes index.html in most cases during site generation). Otherwise the CZT
website tries to follow standard Maven site conventions: edit/add the files
in `src/site` subdirectory of each project. The *.md files are written in
Markdown :)

When you push the changes to Git, they will become part of the website the
next day!

---

By the way, Maven tells me that this year is the 10 year anniversary for
CZT: <inceptionYear>2003</inceptionYear>? Congratulations everyone! It
should be a good opportunity to get CZT 2.0 out of the door :)


Best regards,
Andrius

Re: [CZT-Devel] Building from CZT snapshot

[Mark U. <ma...@cs...>]

Anthony and Andrius,



On 10 April 2013 03:18, Anthony Hall <an...@an...> wrote:

> Dear Andrius****
>
> * *
>
> That is an interesting observation! I am reading myself online now that
> JUnit does run tests in an arbitrary order.. I did have the same assumption
> as you.****
>
> Bizarre, isn’t it? I particularly like the doctrinaire ignorance of “of
> course your tests should be totally independent” – what world do these
> people live in?
>

Actually, it is probably caused by Java 1.7 -- since build 129 of Java SE
the reflection method getMethods() returns the methods in a
non-deterministic order.  (Before that, it returned them in source code
order).  So if you've upgraded to Java 1.7, JUnit has no way of knowing the
original order of the methods any more.

Sun says this is a feature, not a bug. And
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7023180  says:

Filed a request for JUnit to behave predictably irrespective of VM
order: https://github.com/KentBeck/junit/pull/293

2011-09-09




I don't know if this has been implemented in JUnit yet.  I doubt it.
 (Presumably they would have to parse the source code of your Java).

I think relying on the order of test methods is bad style anyway, because
it makes it hard to tools to run tests in isolation, means that test suite
minimisation algorithms do not work, you cannot automatically reorder tests
to run the quick ones first, etc.

Cheers
Mark


*Area:* HotSpot
*Synopsis:* Order of Methods returned by Class.get Methods can Vary
*Description:* In JDK 7, build 129, the following reflective operations in
java.lang.Class changed the fixed order in which they return the methods
and constructors of a class:

   - getMethods
   - getDeclaredMethods
   - getDeclaredConstructors

This may cause issues for code that assumes (contrary to the specification)
a particular order for methods or constructors.

Re: [CZT-Devel] Building from CZT snapshot

[Tim M. <tm...@un...>]

On 10/04/13 03:18, Anthony Hall wrote:
> That is an interesting observation! I am reading myself online now that
> JUnit does run tests in an arbitrary order.. I did have the same
> assumption as you.
>
I knew that junit reserved the right to run tests in any order, but I 
did not realise it would change from run to run. Very strange. I can see 
the value in have random sequences: if you have independent tests, they 
should run in any order, so trying different sequences each time give 
you different tests; but one would think that should be an option you 
have to turn *on*, rather than off. That makes it incredibly hard to 
replicate when tests fail.

> Bizarre, isn’t it? I particularly like the doctrinaire ignorance of “of
> course your tests should be totally independent” – what world do these
> people live in?
>
Yes, a rule of thumb that is a worthwhile goal, but some people are too 
dogmatic about it -- the Junit authors especially.

TestNG supports test dependencies well. I also recall reading a while 
ago that someone had contributed something to Junit to do this, mainly 
to stop tests running if other tests has failed to prevent cascading 
failures. I don't have the details handy.

It's good to see some activity on this mailing list! And that a lot of 
work is still happening on the project :)

Cheers,
Tim

Re: [CZT-Devel] Building from CZT snapshot

[Anthony H. <an...@an...>]

Dear Andrius

 

That is an interesting observation! I am reading myself online now that JUnit does run tests in an arbitrary order.. I did have the same assumption as you.

Bizarre, isn’t it? I particularly like the doctrinaire ignorance of “of course your tests should be totally independent” – what world do these people live in?

 

I am not sure if that is the case in VCG, but Leo's doing an overhaul of it at the moment so hopefully all such issues will be identified and resolved in the nearest future. Some of the flashing errors are corner cases for VCG which require some tuning (if I understand correctly), so they are ignored in the tests but are still producing verbose output at the moment. So that output can be ignored as long as the build does not fail.

Thanks – that’s good to know

 

How is the SectionManager running with the latest CZT version? Are you still getting transaction errors?

No, thank you. That was entirely the fault of maven. All seems well now. I only had to make the one change to the constructor call.

 

Many thanks for your help

 

Anthony

 

Best regards,

Andrius

 

 

On Tue, Apr 9, 2013 at 2:36 PM, Anthony Hall <an...@an...> wrote:

Thanks Andrius

 

I’ve changed the url 

 

When I rebuilt I again got VCG test errors. This time, however, when I tried again the build was successful, although a lot of errors flashed past during the VCG testing they seemed to be handled in some way that didn’t prevent the build working.

 

I’ve just discovered something horrible that may possibly throw light on that. I’m getting a similar problem with my own cztinterface build – tests are failing apparently randomly. The root cause in my case appears to be a feature of JUnit: within a suite it runs tests in whatever order it feels like (and its mood seems to change from run to run). I had assumed and relied on the fact that it ran them in the order I wrote them. I’m surprised that this hasn’t hit me before. Anyway, I wonder if that might be happening in the VCG case as well?

 

All the best

 

Anthony

 

 

From: Andrius Velykis [mailto:and...@ne...] 
Sent: 09 April 2013 13:57


To: Anthony Hall
Cc: CZT-Devel
Subject: Re: [CZT-Devel] Building from CZT snapshot

 

Dear Anthony,

 

On Tue, Apr 9, 2013 at 12:45 PM, Anthony Hall <an...@an...> wrote:

According to git config:

remote.origin.url=git://czt.git.sourceforge.net/gitroot/czt/czt

remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*

The url is what is given in setup.md 

 

 

Yes, sorry, this is the old CZT Git repository path. After the recent move to the new SourceForge platform, the Git repository addresses changed. The read-only access is now `git://git.code.sf.net/p/czt/code` <http://git.code.sf.net/p/czt/code%60> . For write access, check your personal link at http://sourceforge.net/p/czt/code/ when you are logged in to SourceForge.

 

Let me know if this gets the latest version for you. Here's some info on changing the Git remote URL: http://stackoverflow.com/questions/2432764/how-to-change-a-remote-repository-uri-using-git

 

We will update the instructions accordingly. Sorry the about misleading instructions in setup.md - I forgot to update the documentation after the repository change.

 

Best regards,

Andrius

 


------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
CZT-Devel mailing list
CZT...@li...
https://lists.sourceforge.net/lists/listinfo/czt-devel

 

Re: [CZT-Devel] Building from CZT snapshot

[Leo F. <leo...@ne...>]

Hi Anthony,

Thanks for sticking with git and its teething problems… we are getting there and hopefully with the new site it will be easy to document for us/myself as well,
then these problems will be rooted out earlier…

--- SectMngmt.

In a nutshell, review the documentation within the top of SectionManager class and in the get / put methods to see how that affects transactions.
We tried to keep the API as seemingness as possible, yet if you use commands you might need to worry about transactions . Details on the features eamil.

The key motivation for transactions came through the handling of large specs (and those memory issues we discussed a while ago).
The point was how / what to reparse-typecheck or not on the transitive dependency chain such that minimal work was done and consistency
was guarantee.

We took a pretty conservative approach (i.e. if inference of consistency isn't possible, assume section is dirty and redo work), yet even that (together
with memory management improvements) improved performance by an order of magnitude (!!!).

The price for that the API got a bit more complicated for certain kinds of commands. In particular those commands where the "scope" of transaction
need to be widened. For instance, LaTeX parsing leads to a plethora of intermediate stages (i.e. latex markup, operator table, newline table, keyword table, etc),
but in particular low-level Unicode scanning, which leads to transactions overlap that would potentially create inconsistency when a section depend
on similar parents (e.g. quite common with those depending on standard_toolkit).

The widening of transaction scope (to ensure consistency)  and at the same time guarantee a minimum amount of re-work possible (i.e. tokenising
and I/O during parsing is the lion's share of the time for type checking), at the price of some adjustment to section management code.

There are various examples throughout CZT on the use of transactions for the section manager you could draw inspiration from, but we ought / will
add this as a key part of its documentation.

--- VCGing

Some tests on VCG do fail and I know about the issue. They come mostly from weird / unusual schema calculus features for tests that there is
no actual implementation for them yet. I then explicitly say "EXCEPTION HANDLED DURING TESTING" so that I don't loose the test case by removing it.
Given they are from weird uses of Z rather than actual ones, I will only implement later on.

--- jUnit

jUnit is indeed unsorted so one cannot rely on declared order. Instead you could use the setup() and tearDown() methods for setting up things like
the section manager.

Best,
Leo


On 9 Apr 2013, at 14:36, Anthony Hall <an...@an...<mailto:an...@an...>>
 wrote:

Thanks Andrius

I’ve changed the url

When I rebuilt I again got VCG test errors. This time, however, when I tried again the build was successful, although a lot of errors flashed past during the VCG testing they seemed to be handled in some way that didn’t prevent the build working.

I’ve just discovered something horrible that may possibly throw light on that. I’m getting a similar problem with my own cztinterface build – tests are failing apparently randomly. The root cause in my case appears to be a feature of JUnit: within a suite it runs tests in whatever order it feels like (and its mood seems to change from run to run). I had assumed and relied on the fact that it ran them in the order I wrote them. I’m surprised that this hasn’t hit me before. Anyway, I wonder if that might be happening in the VCG case as well?

All the best

Anthony


From: Andrius Velykis [mailto:and...@ne...<http://newcastle.ac.uk>]
Sent: 09 April 2013 13:57
To: Anthony Hall
Cc: CZT-Devel
Subject: Re: [CZT-Devel] Building from CZT snapshot

Dear Anthony,

On Tue, Apr 9, 2013 at 12:45 PM, Anthony Hall <an...@an...<mailto:an...@an...>> wrote:
According to git config:
remote.origin.url=git://czt.git.sourceforge.net/gitroot/czt/czt<http://czt.git.sourceforge.net/gitroot/czt/czt>
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
The url is what is given in setup.md<http://setup.md>


Yes, sorry, this is the old CZT Git repository path. After the recent move to the new SourceForge platform, the Git repository addresses changed. The read-only access is now `git://git.code.sf.net/p/czt/code`<http://git.code.sf.net/p/czt/code%60>. For write access, check your personal link at http://sourceforge.net/p/czt/code/ when you are logged in to SourceForge.

Let me know if this gets the latest version for you. Here's some info on changing the Git remote URL: http://stackoverflow.com/questions/2432764/how-to-change-a-remote-repository-uri-using-git

We will update the instructions accordingly. Sorry the about misleading instructions in setup.md<http://setup.md> - I forgot to update the documentation after the repository change.

Best regards,
Andrius

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter_______________________________________________
CZT-Devel mailing list
CZT...@li...<mailto:CZT...@li...>
https://lists.sourceforge.net/lists/listinfo/czt-devel

Re: [CZT-Devel] Building from CZT snapshot

[Andrius V. <and...@ne...>]

Dear Anthony,

That is an interesting observation! I am reading myself online now that
JUnit does run tests in an arbitrary order.. I did have the same assumption
as you.

I am not sure if that is the case in VCG, but Leo's doing an overhaul of it
at the moment so hopefully all such issues will be identified and resolved
in the nearest future. Some of the flashing errors are corner cases for VCG
which require some tuning (if I understand correctly), so they are ignored
in the tests but are still producing verbose output at the moment. So that
output can be ignored as long as the build does not fail.

How is the SectionManager running with the latest CZT version? Are you
still getting transaction errors?

Best regards,
Andrius



On Tue, Apr 9, 2013 at 2:36 PM, Anthony Hall <an...@an...>wrote:

> Thanks Andrius****
>
> ** **
>
> I’ve changed the url ****
>
> ** **
>
> When I rebuilt I again got VCG test errors. This time, however, when I
> tried again the build was successful, although a lot of errors flashed past
> during the VCG testing they seemed to be handled in some way that didn’t
> prevent the build working.****
>
> ** **
>
> I’ve just discovered something horrible that may possibly throw light on
> that. I’m getting a similar problem with my own cztinterface build – tests
> are failing apparently randomly. The root cause in my case appears to be a
> feature of JUnit: within a suite it runs tests in whatever order it feels
> like (and its mood seems to change from run to run). I had assumed and
> relied on the fact that it ran them in the order I wrote them. I’m
> surprised that this hasn’t hit me before. Anyway, I wonder if that might be
> happening in the VCG case as well?****
>
> ** **
>
> All the best****
>
> ** **
>
> Anthony****
>
> ** **
>
> ** **
>
> *From:* Andrius Velykis [mailto:and...@ne...]
> *Sent:* 09 April 2013 13:57
>
> *To:* Anthony Hall
> *Cc:* CZT-Devel
> *Subject:* Re: [CZT-Devel] Building from CZT snapshot****
>
> ** **
>
> Dear Anthony,****
>
> ** **
>
> On Tue, Apr 9, 2013 at 12:45 PM, Anthony Hall <an...@an...>
> wrote:****
>
> According to git config:****
>
> remote.origin.url=git://czt.git.sourceforge.net/gitroot/czt/czt****
>
> remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*****
>
> The url is what is given in setup.md ****
>
> ** **
>
> ** **
>
> Yes, sorry, this is the old CZT Git repository path. After the recent move
> to the new SourceForge platform, the Git repository addresses changed. The
> read-only access is now `git://git.code.sf.net/p/czt/code`<http://git.code.sf.net/p/czt/code%60>.
> For write access, check your personal link at
> http://sourceforge.net/p/czt/code/ when you are logged in to SourceForge.*
> ***
>
> ** **
>
> Let me know if this gets the latest version for you. Here's some info on
> changing the Git remote URL:
> http://stackoverflow.com/questions/2432764/how-to-change-a-remote-repository-uri-using-git
> ****
>
> ** **
>
> We will update the instructions accordingly. Sorry the about misleading
> instructions in setup.md - I forgot to update the documentation after the
> repository change.****
>
> ** **
>
> Best regards,****
>
> Andrius****
>
> ** **
>
>
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> CZT-Devel mailing list
> CZT...@li...
> https://lists.sourceforge.net/lists/listinfo/czt-devel
>
>

Re: [CZT-Devel] Building from CZT snapshot

[Anthony H. <an...@an...>]

Thanks Andrius

 

I’ve changed the url 

 

When I rebuilt I again got VCG test errors. This time, however, when I tried again the build was successful, although a lot of errors flashed past during the VCG testing they seemed to be handled in some way that didn’t prevent the build working.

 

I’ve just discovered something horrible that may possibly throw light on that. I’m getting a similar problem with my own cztinterface build – tests are failing apparently randomly. The root cause in my case appears to be a feature of JUnit: within a suite it runs tests in whatever order it feels like (and its mood seems to change from run to run). I had assumed and relied on the fact that it ran them in the order I wrote them. I’m surprised that this hasn’t hit me before. Anyway, I wonder if that might be happening in the VCG case as well?

 

All the best

 

Anthony

 

 

From: Andrius Velykis [mailto:and...@ne...] 
Sent: 09 April 2013 13:57
To: Anthony Hall
Cc: CZT-Devel
Subject: Re: [CZT-Devel] Building from CZT snapshot

 

Dear Anthony,

 

On Tue, Apr 9, 2013 at 12:45 PM, Anthony Hall <an...@an...> wrote:

According to git config:

remote.origin.url=git://czt.git.sourceforge.net/gitroot/czt/czt

remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*

The url is what is given in setup.md 

 

 

Yes, sorry, this is the old CZT Git repository path. After the recent move to the new SourceForge platform, the Git repository addresses changed. The read-only access is now `git://git.code.sf.net/p/czt/code` <http://git.code.sf.net/p/czt/code%60> . For write access, check your personal link at http://sourceforge.net/p/czt/code/ when you are logged in to SourceForge.

 

Let me know if this gets the latest version for you. Here's some info on changing the Git remote URL: http://stackoverflow.com/questions/2432764/how-to-change-a-remote-repository-uri-using-git

 

We will update the instructions accordingly. Sorry the about misleading instructions in setup.md - I forgot to update the documentation after the repository change.

 

Best regards,

Andrius

 

Re: [CZT-Devel] Building from CZT snapshot

[Andrius V. <and...@ne...>]

Dear Anthony,

On Tue, Apr 9, 2013 at 12:45 PM, Anthony Hall <an...@an...>wrote:

> According to git config:
>
> remote.origin.url=git://czt.git.sourceforge.net/gitroot/czt/czt****
>
> remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*****
>
> The url is what is given in setup.md ****
>
>
>
Yes, sorry, this is the old CZT Git repository path. After the recent move
to the new SourceForge platform, the Git repository addresses changed. The
read-only access is now `git://git.code.sf.net/p/czt/code`. For write
access, check your personal link at http://sourceforge.net/p/czt/code/ when
you are logged in to SourceForge.

Let me know if this gets the latest version for you. Here's some info on
changing the Git remote URL:
http://stackoverflow.com/questions/2432764/how-to-change-a-remote-repository-uri-using-git

We will update the instructions accordingly. Sorry the about misleading
instructions in setup.md - I forgot to update the documentation after the
repository change.

Best regards,
Andrius

Re: [CZT-Devel] Building from CZT snapshot

[Anthony H. <an...@an...>]

Dear Andrius

 

However your question about which version I was using did prompt a couple of questions in return. The answer is that I was using the master cloned from the Git repository. I was slightly surprised that when I did a fetch after a couple of days there seemed to be no changes. When I looked at the log, it seems the last change to master was on Jan 10 at 17-10-07. Is that correct, or am I doing something wrong with git? What are the nightly builds made from?

 

Interesting issues - I am not really sure. Which address did you use to clone the Git repository?

According to git config:

remote.origin.url=git://czt.git.sourceforge.net/gitroot/czt/czt

remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*

The url is what is given in setup.md 

 

I have a hunch that our migration to the new SourceForge platform may have influenced this.. if you see http://sourceforge.net/p/czt/code/ , the code listing on their website has some "unknown" bits. After new commits, they get refreshed to the correct version.. So I am a bit puzzled, but hope it gets resolved over time by SourceForge.

 

Nightly builds are being made from the default read-only CZT git repository at `git://git.code.sf.net/p/czt/code` <http://git.code.sf.net/p/czt/code%60> .

That doesn’t seem to be the same. Which should I be using?

 

So have you managed to pull the latest updates now or are you still on 2007 version? If there are problems, I can investigate.

As I say the version I have is Jan 30th (2013 that is)

 

May I ask a related question, purely about git? When we were using subversion, I referred to the version of czt that I had used in building cztinterface by its svn revision number. What is the git equivalent that allows someone to go and get the exact revision? Is it the checksum, or timestamp, or what?

In Git, the commits are referenced by their hash, e.g. 967829cb33b9fba62c11bb577ca496fee0aa0525 is the current head of the `master` branch (though we will push a lot of changes soon).

The hash and info of the last commit can be obtained by running `git log -1`.

 

You can search for the commit info by this reference, e.g. use `git show 967829cb33b9fba62c11bb577ca496fee0aa0525`

Thanks for that

 

All the best

 

Anthony

Re: [CZT-Devel] Building from CZT snapshot

[Andrius V. <and...@ne...>]

Dear Anthony,

I just realised that you may have used the old CZT repository path `git://
czt.git.sourceforge.net/gitroot/czt/czt` if you were following our setup
instructions... After the recent move to the new SourceForge platform, the
Git repository addresses changed. The read-only access is now `git://
git.code.sf.net/p/czt/code`. For write access, check your personal link at
http://sourceforge.net/p/czt/code/ when you are logged in to SourceForge.

Let me know if this gets the latest version for you. Here's some info on
changing the Git remote URL:
http://stackoverflow.com/questions/2432764/how-to-change-a-remote-repository-uri-using-git

We will update the instructions accordingly.

Andrius


On Tue, Apr 9, 2013 at 12:20 PM, Andrius Velykis <
and...@ne...> wrote:

> Dear Anthony,
>
> Good to hear!
>
> On Tue, Apr 9, 2013 at 12:02 PM, Anthony Hall <an...@an...>wrote:
>
>>
>> So, I have now rebuilt czt and this time it did not give the errors in
>> the VCG I encountered before, and I could successfully build my
>> cztinterface. (Though I did have to change one thing – I see that there is
>> no longer a “typechecker”, but rather a “typechecker-z” – my pom had
>> referred to “typechecker” and so failed the first time.)
>>
>
> We have normalised all project names to indicate the dialect.
> `typechecker` is now a parent "umbrella" project for all typecheckers.
>
>
>> ****
>>
>> However your question about which version I was using did prompt a couple
>> of questions in return. The answer is that I was using the master cloned
>> from the Git repository. I was slightly surprised that when I did a fetch
>> after a couple of days there seemed to be no changes. When I looked at the
>> log, it seems the last change to master was on Jan 10 at 17-10-07. Is that
>> correct, or am I doing something wrong with git? What are the nightly
>> builds made from?
>>
>
> Interesting issues - I am not really sure. Which address did you use to
> clone the Git repository?
>
> I have a hunch that our migration to the new SourceForge platform may have
> influenced this.. if you see http://sourceforge.net/p/czt/code/ , the
> code listing on their website has some "unknown" bits. After new commits,
> they get refreshed to the correct version.. So I am a bit puzzled, but hope
> it gets resolved over time by SourceForge.
>
> Nightly builds are being made from the default read-only CZT git
> repository at `git://git.code.sf.net/p/czt/code`<http://git.code.sf.net/p/czt/code>
> .
>
> So have you managed to pull the latest updates now or are you still on
> 2007 version? If there are problems, I can investigate.
>
>
>> May I ask a related question, purely about git? When we were using
>> subversion, I referred to the version of czt that I had used in building
>> cztinterface by its svn revision number. What is the git equivalent that
>> allows someone to go and get the exact revision? Is it the checksum, or
>> timestamp, or what?
>>
> In Git, the commits are referenced by their hash, e.g.
> 967829cb33b9fba62c11bb577ca496fee0aa0525 is the current head of the
> `master` branch (though we will push a lot of changes soon).
> The hash and info of the last commit can be obtained by running `git log
> -1`.
>
> You can search for the commit info by this reference, e.g. use `git show
> 967829cb33b9fba62c11bb577ca496fee0aa0525`
>
> Thanks for all your help, and apologies for bothering you with the
>> maven-induced nonsense
>>
> No problem - builds are rarely trivial!
>
> Best regards,
> ~Andrius
>
>

Re: [CZT-Devel] Problem installing jEdit plugin

[Anthony H. <an...@an...>]

Dear Leo

 

Nice to hear from you --- and apologies for the delay. I was away in Brazil
on holiday in the middle of nowhere :-).

Thanks for this and several other messages - I'll consolidate replies into
one.

 

I've picked up on CZT again because of a possible need to use it. I'm also
very encouraged that there is going to be a new release and that prompts me
to try and find some time to look at the memory usage problem again. 

 

And of course trying to get back into it I've been hitting some problems
because of unfamiliarity and not having kept up with the changes - sorry to
have burdened the list with these troubles and I'm grateful for all the help
from Andrius.

 

 

--- git

 

I had a similar feeling when I started using it (through Andrius diligent
suggestion for my doing so). 

It is away better than SVN in various respects and not that difficult in
itself. A few new concepts like

push/pull/fetch, but they are natural (distributed) counterparts for
commit/update/check.

 

As I said, I don't object and haven't had any problem yet - just need to
learn it.

 

--- IDEs and CZT.jar

 

the jEdit development was meant as a testbed for how a CZT GUI would look
like / need --- if I recall correctly.

In itself jEdit came a long way since then and improved in various respects.
We keep it alive to make allow for

a light-weight IDE usage, but not all recent features have been implemented
for it.

 

Andrius is an Eclipse wizard, having worked in industry building
Eclipse-based plugins and systems. This made

the transition to Eclipse (and update over the original development by Mark
and his students) possible. 

 

Again, I insisted to keep the IDE separate (and a reflection of) the
CZT.jar, which does contain access to all

recent/new features. Eclipse has extra bells and whistles, though. They make
specification and cross referencing

much easier / smoother than in jEdit or other editor.

 

The Eclipse plugin seemed pretty nice - I'm still interested in the answer
to my question to Andrius about incremental checking and big specs in
multiple files.

 

---- release

 

Andrius recently added nightly builds, which means you can get newest
(stable) features without much fuss

(i.e. no need for long builds and/or GIT). 

 

We make a point about only pushing (new repository versions) stable versions
of features (i.e. those that passed

tests and that we have worked with in some model in practice). This way
nightly builds are reliable.

 

We plan to have a release over the summer, and Andrius is launching soon
(coming week) a new website / doc site too !

 

Yes - I think these are all really good developments and will make it a lot
better.

 

My two pence on the OZ discussion. On jEdit, you need to choose the ZCharmap
accordingly so that oz symbols are used.

You also need to change on the ZSidekick the default parser for .tex (or
.utf8) files (which is Z-Latex/Unicode) to OZ, unless

you add files with the right extension (I think .oz).  A similar story
occurs in Eclipse: you need to choose the "default compiler"

in the preferences for CZT as Z, OZ, Circus, ZEves. 

 

The only problem I had with Eclipse was the one about creating an oz8 file -
I didn't try oz I must admit as I wasn't keen on latex. I'm putting this one
on the back burner for now unless the need for it arises in which case I'll
look into it further. 

 

On the font issue, yes there was a problem with Visual Studio and the TTF
generated by font forge (which was the tool we used

to create the CZT font for Z, OZ and Circus). As Mark said I think this has
been resolved, but tests on Windows are due on our side.

In any case, installing/uninstalling it is just a matter of copying/deleting
a file from c:\windows\fonts, so shouldn't be too troublesome.

Let us know otherwise.

 

I've installed it and will let you know if I have any trouble

 

On OZ, I tried extending Andrius' Eclipse development for OZ and Circus and
got as far as having the type checker working

with the ZCharmap support under Eclipse. Missing features I know off are
syntax highlighting for example.

 

I may be interested in pursuing this but it depends on a potential customer
- I'll let you know if I really need to go forward with this.

 

On ZEves, I did try with Mark Saaltink to "revive" it, but the trouble was
change in the Canadian government civil service leading

to a "freeze" in "what to do". The bits owned by ORA canada are somewhat
available to tweak (i.e. the Z-TO-EVES and EVES-TO-Z)

and use, yet the underlying EVES prover isn't (and is unlikely to become).

 

Having said that, Mark and I started (on and off since 2008) to "redevelop"
EVES (i.e. Mark redoes it and I use/test/suggest), and we

got a version called "Z/P" (P=prover). Given EVES is close to ACL2 (with
Quantifiers), I also tried to link the Z translations back and 

forth to LISP as ACL2 but that's not as simple as I hoped.

 

I think the *key* to ZEves was its ability to turn LISP/ML-like Z
representation of the schema calculus back to a Z-LaTex/Unicode view.

I mean, it's relatively easy to go from Z to say Isabelle/HOL (i..e. see
Z/HOL, ISabelle/Z, Proof-Power-Z). The difficulty comes in bringing

back whatever the prover's result / intermediate proof is as a Z construct!
That part from ZEves is available and is what we are (Slowly)

working with. 

In practical terms, I think for now I should just assume it's dead. On
Proofpower, I am discussing with Rob Arthan the idea of a Z Word
Tools/Proofpower integration. As part of this, Rob is I think working on
making PP use Unicode - but there are as you say still issues with the fact
that you are really working in HOL all the time.

 

Many thanks for all your feedback.

 

On the memory usage question, I was thinking of starting by investigating
just what does get put on the AST. Is there an AST pretty printer or similar
that I could modify to do this?

 

Thanks

 

Anthony

 

Re: [CZT-Devel] Building from CZT snapshot

[Andrius V. <and...@ne...>]

Dear Anthony,

Good to hear!

On Tue, Apr 9, 2013 at 12:02 PM, Anthony Hall <an...@an...>wrote:

>
> So, I have now rebuilt czt and this time it did not give the errors in the
> VCG I encountered before, and I could successfully build my cztinterface.
> (Though I did have to change one thing – I see that there is no longer a
> “typechecker”, but rather a “typechecker-z” – my pom had referred to
> “typechecker” and so failed the first time.)
>

We have normalised all project names to indicate the dialect. `typechecker`
is now a parent "umbrella" project for all typecheckers.


> ****
>
> However your question about which version I was using did prompt a couple
> of questions in return. The answer is that I was using the master cloned
> from the Git repository. I was slightly surprised that when I did a fetch
> after a couple of days there seemed to be no changes. When I looked at the
> log, it seems the last change to master was on Jan 10 at 17-10-07. Is that
> correct, or am I doing something wrong with git? What are the nightly
> builds made from?
>

Interesting issues - I am not really sure. Which address did you use to
clone the Git repository?

I have a hunch that our migration to the new SourceForge platform may have
influenced this.. if you see http://sourceforge.net/p/czt/code/ , the code
listing on their website has some "unknown" bits. After new commits, they
get refreshed to the correct version.. So I am a bit puzzled, but hope it
gets resolved over time by SourceForge.

Nightly builds are being made from the default read-only CZT git repository
at `git://git.code.sf.net/p/czt/code`.

So have you managed to pull the latest updates now or are you still on 2007
version? If there are problems, I can investigate.


> May I ask a related question, purely about git? When we were using
> subversion, I referred to the version of czt that I had used in building
> cztinterface by its svn revision number. What is the git equivalent that
> allows someone to go and get the exact revision? Is it the checksum, or
> timestamp, or what?
>
In Git, the commits are referenced by their hash, e.g.
967829cb33b9fba62c11bb577ca496fee0aa0525 is the current head of the
`master` branch (though we will push a lot of changes soon).
The hash and info of the last commit can be obtained by running `git log
-1`.

You can search for the commit info by this reference, e.g. use `git show
967829cb33b9fba62c11bb577ca496fee0aa0525`

Thanks for all your help, and apologies for bothering you with the
> maven-induced nonsense
>
No problem - builds are rarely trivial!

Best regards,
~Andrius

Re: [CZT-Devel] Building from CZT snapshot

[Anthony H. <an...@an...>]

Dear Andrius

 

I apologise. This was not a CZT problem at all. 

 

It was, as I should have guessed, the diabolical Maven. (I don’t….no, I’d better not express my opinion of Maven – it isn’t fit for a polite mailing list).

 

So, I have now rebuilt czt and this time it did not give the errors in the VCG I encountered before, and I could successfully build my cztinterface. (Though I did have to change one thing – I see that there is no longer a “typechecker”, but rather a “typechecker-z” – my pom had referred to “typechecker” and so failed the first time.)

 

However your question about which version I was using did prompt a couple of questions in return. The answer is that I was using the master cloned from the Git repository. I was slightly surprised that when I did a fetch after a couple of days there seemed to be no changes. When I looked at the log, it seems the last change to master was on Jan 10 at 17-10-07. Is that correct, or am I doing something wrong with git? What are the nightly builds made from? 

 

May I ask a related question, purely about git? When we were using subversion, I referred to the version of czt that I had used in building cztinterface by its svn revision number. What is the git equivalent that allows someone to go and get the exact revision? Is it the checksum, or timestamp, or what?

 

Thanks for all your help, and apologies for bothering you with the maven-induced nonsense

 

Anthony

 

 

 

From: Andrius Velykis [mailto:and...@ne...] 
Sent: 09 April 2013 09:27
To: Anthony Hall
Cc: CZT-Devel
Subject: Re: [CZT-Devel] Building from CZT snapshot

 

Dear Anthony,

 

Thanks for the info. Which CZT sources are you using? I am looking at the latest CZT source code and have problems matching your stack trace to my code.. The calls that are made in your stack trace are non-existent in my code :( Are you working on the `master` branch of the CZT Git repository? Or are you using the nighly build czt.jar?

 

Best regards,

Andrius

 

 

 

On Mon, Apr 8, 2013 at 5:31 PM, Anthony Hall <an...@an...> wrote:

Dear Andrius

 

Thank you again for a very swift response

 

The SectionManager did change - mostly internally (we added a support for "transactions" as a step towards better incremental-checking of Z sections - so that you could re-check only downstream dependencies without throwing away all upstream ones). Note, however, that this change enforces a bit stricter way of how it should be called - so let us assist if you encounter problems regarding to it.

 

The change in SectionManager interface you encountered is that it now explicitly requires to provide a dialect to use (see net.sourceforge.czt.session.Dialect enum for appropriate values). So in your case it would be `new SectionManager(Dialect.Z)` for Z checking.

Thank you. I did that and it all compiled OK. However, now I seem to be running foul of the new transaction features, because I’m getting a lot of test failures of the form “SectionInfoException: There is an ongoing transaction for the given key”

Of course I’m not doing anything manually with transactions so I’ve really not got much clue as to how to fix this.

 

The tests work be repeatedly calling the main procedure of my class, and that does a new SectionManager(Dialect.Z) each time. So I’m not sure what else I need to do. 

 

I attach the main class (CZTCheck, the class that runs the test (CZTTestUtils), a failing test set and the surefire reports. If you could point me at what I need to do I’d be most grateful!

 

You can generate the JavaDocs yourself by running `mvn javadoc:javadoc` for some module. They will be put into `target/site/apidocs` of the project (I think).

That seems to work, yes – though I got a lot of warnings like

[WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, no dependency information available

[WARNING] Failed to retrieve plugin descriptor for org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0

but I don’t know whether that matters

 

And of course I have had a look at the SectionManager apidocs but it looks as if I shouldn’t have to do anything explicit unless I’m trying to do something clever with incremental compilation – or have I missed something there?

 

I know it is a bit annoying to manually have to generate all this. Since we now have nightly builds, I think I will just put a "nightly" version of the website as well - the current one has not been updated for 5 years! :(  Then all the changes that we add would be reflected immediately. This way all the up-to-date JavaDocs will also be online soon.

I think that would be really helpful – great!

 

Many thanks

 

Anthony

 

Let me know if there are any problems with the JavaDocs.

 

Best regards,

Andrius

 

On Mon, Apr 8, 2013 at 3:31 PM, Anthony Hall <an...@an...> wrote:

Part of Z Word tools is the cztinterface program which is built from the CZT snapshot. I last built it from CZT SVN Version 8214

 

Now I’ve downloaded and built the latest CZT version from Git and I can no longer build cztinterface. I get an error “No suitable constructor found for SectionManager()” at a line that says 

SectionManager manager = new SectionManager();

 

I seem to remember that the SectionManager was changed some time ago. Is there an up to date javadocs site that I can look at to see the documentation of the classes in the current version?

 

Thanks

 

Anthony

 


------------------------------------------------------------------------------
Minimize network downtime and maximize team effectiveness.
Reduce network management and security costs.Learn how to hire
the most talented Cisco Certified professionals. Visit the
Employer Resources Portal
http://www.cisco.com/web/learning/employer_resources/index.html
_______________________________________________
CZT-Devel mailing list
CZT...@li...
https://lists.sourceforge.net/lists/listinfo/czt-devel

 

Re: [CZT-Devel] Problem installing jEdit plugin

[Leo F. <leo...@ne...>]

Hi guys,

We will test here on Windows as much as we can, but Linux/Mac get more day-to-day use.

A note on "defaults", as part of new features, we closed some loops (left on the Iso Z Std) on how
file / project management is to take place. These are relatively obvious, yet to be documented like
default parsers-by-extension, directories, etc.

On the jEdit plugins, they are on the nightly build as well.

Best,
Leo

On 3 Apr 2013, at 13:42, Andrius Velykis <and...@ne...<mailto:and...@ne...>> wrote:

Dear Anthony,


Well I’d expected something like that,  but I can’t see any such option in the dropdown: it just seems to have docking options.

I see - I checked the source code - apparently it reacts to 'mode' change in jEdit. Use the dropdown in top-right corner to change modes, then the charmap should change accordingly..


Well as I said I’ve never used Eclipse at all, so I’m fumbling here – I must say it does look very nice though and indeed I can get an Object Z character map. The only thing I couldn’t see how to do was to get an Object-Z specification created. It has an option to create a new Z specification, but even though I’d set the compile to oz, it created a file with a zed8 extension rather than an oz8 extension. I tried Create file instead and gave the extension as oz8, but that just seemed to create an ordinary text file. I’m sure I’m doing something silly, though – as I say this is  my first ever experience of Eclipse!

Interesting point about extensions. What you encountered is what was supposed have happened. At the moment CZT Eclipse works with `zed`, `zed8` and `zed16` extensions - the Z Editor is opened by default for them. The dialect is selected in Preferences and has not been linked to the file extension, i.e. you select a dialect for all files, and the extension governs whether it is LaTeX or one of Unicode representations. Since it does not know about `oz8` extension, it will open it with text editor. You can try forcing Z Editor by right-clicking and selecting Open With > Z Editor.

We have discussed using dialect-specific extensions some time ago with Leo. Are there official ones, e.g. `oz` for Object-Z, etc, or at least some accepted practice?




The errors are unexpected. I know Leo's been doing some changes to VCG so maybe there are problems, though the last automatic build of `master` did pass correctly.. Are you getting this error on Windows? We will try to investigate.

Yes they are on Windows (W7 64 bit).

We have not tested much on Windows recently, so maybe it is worth checking it out. :)




I am afraid that CZT font still needs to be installed. I know loading the font automatically should not be hard to add, but at the moment it needs to be installed.

The need to install it wasn’t the problem – my concern was that the font itself used to be broken in some way that crashed windows pretty severely and I didn’t want to install it without knowing that had been fixed. (Yes, I know – who would write an OS that could be crashed by a defective font…? But it’s true – I managed to pull the same trick with my own font at one point)

I have committed several changes today - with the today's version of CZT Eclipse IDE you no longer need to install CZT font, it is loaded automatically. This has not been done for jEdit though yet.


I am not sure whether that is necessary. You need to get the czt.jar and the jEdit plugins - that should be enough?

Yes – but without building where would I get the jEdit plugins? Again I’m sure the answer is obvious to you but it wasn’t to me – sorry. Anyway, done it now.

You are right, the plugins needed to be built as well.

I have got around to finishing the `nightly` builds as well - if you go to the CZT SourceForge files system, each of possible download options now has a /nightly directory where you can download the latest version! We have a build server running here in Newcastle and the nightly results will be published to SourceForge now.

http://sourceforge.net/projects/czt/files/


Can you tell me a bit more about Z-Eves? I had thought that Z-Eves was dead: if that’s not true I’d be very interested in finding out more about it.

It is dead (as in not being developed anymore), though many institutions still have copies of it and it works quite well with this new GUI :) We had a buch of proofs in Z/EVES that we needed to work with so added the integration. Now here's to hoping that Z/EVES would be open-sourced some day.. but it is not clear whether that will happen.


Answers are certainly helpful. I will try to revive the memory consumption work, though I fear it may be deep inside CZT and beyond my competence.

The core developers are still around so I think help would come if needed :)

Thanks,
Andrius

------------------------------------------------------------------------------
Minimize network downtime and maximize team effectiveness.
Reduce network management and security costs.Learn how to hire
the most talented Cisco Certified professionals. Visit the
Employer Resources Portal
http://www.cisco.com/web/learning/employer_resources/index.html_______________________________________________
CZT-Devel mailing list
CZT...@li...
https://lists.sourceforge.net/lists/listinfo/czt-devel

Re: [CZT-Devel] Problem installing jEdit plugin

[Leo F. <leo...@ne...>]

Hi Anthony,

My two pence on the OZ discussion. On jEdit, you need to choose the ZCharmap accordingly so that oz symbols are used.
You also need to change on the ZSidekick the default parser for .tex (or .utf8) files (which is Z-Latex/Unicode) to OZ, unless
you add files with the right extension (I think .oz).  A similar story occurs in Eclipse: you need to choose the "default compiler"
in the preferences for CZT as Z, OZ, Circus, ZEves.

On the font issue, yes there was a problem with Visual Studio and the TTF generated by font forge (which was the tool we used
to create the CZT font for Z, OZ and Circus). As Mark said I think this has been resolved, but tests on Windows are due on our side.
In any case, installing/uninstalling it is just a matter of copying/deleting a file from c:\windows\fonts, so shouldn't be too troublesome.
Let us know otherwise.

Memory consumption + ZEVes on the features e-mail to follow.

Best,
Leo


On 1 Apr 2013, at 20:51, Anthony Hall <an...@an...<mailto:an...@an...>> wrote:

Dear Andrius
Good to hear about your progress and sorry for a bit of delay with my reply. My answers follow.

I really, really don’t think that two replies, one on Easter Sunday and the other on Easter Monday Bank Holiday, count as a delay!!
On the contrary, I’m very grateful to you for getting back so quickly.

First, my real question: when I did finally get jEdit and zcharmap working, I couldn’t find a way of inserting an object Z Class. Is there any way of doing this – can you edit object Z within CZT?

I am not too familiar with jEdit - there should be a drop-down on ZCharMap that allows you to choose a dialect? Then you should get Object-Z charmap that includes Class if I am not mistaken..
Well I’d expected something like that,  but I can’t see any such option in the dropdown: it just seems to have docking options.

I myself work on CZT-Eclipse integration, so you could try it there. If you use czt-ide standalone that I gave the link to previously: select Object Z dialect in Preferences >  CZT > Compiler. Then when you open the Characters view the Object-Z dialect should be selected there as well (or use drop-down). Note that I am typing these names from memory so they may actually be a bit different in the program :) Let me know if you get to Object-Z symbols either in jEdit or Eclipse.
Well as I said I’ve never used Eclipse at all, so I’m fumbling here – I must say it does look very nice though and indeed I can get an Object Z character map. The only thing I couldn’t see how to do was to get an Object-Z specification created. It has an option to create a new Z specification, but even though I’d set the compile to oz, it created a file with a zed8 extension rather than an oz8 extension. I tried Create file instead and gave the extension as oz8, but that just seemed to create an ordinary text file. I’m sure I’m doing something silly, though – as I say this is  my first ever experience of Eclipse!

The /src/site files are destined for the upcoming new CZT website. The website contents are generated automatically from these files. So as soon as we do get around to launching the new website and new release, these instructions should be accessible much easier (and rendered correctly). In the root of repository there is INSTALL.md that points to setup.md<http://setup.md> - but I do think a website will help with finding things :)
OK, well I was just groping around there so not to worry

The errors are unexpected. I know Leo's been doing some changes to VCG so maybe there are problems, though the last automatic build of `master` did pass correctly.. Are you getting this error on Windows? We will try to investigate.
Yes they are on Windows (W7 64 bit).

I am afraid that CZT font still needs to be installed. I know loading the font automatically should not be hard to add, but at the moment it needs to be installed.
The need to install it wasn’t the problem – my concern was that the font itself used to be broken in some way that crashed windows pretty severely and I didn’t want to install it without knowing that had been fixed. (Yes, I know – who would write an OS that could be crashed by a defective font…? But it’s true – I managed to pull the same trick with my own font at one point)


Good catch on encoding advice. Yea, I guess the instructions should say that. We do lack testing in Windows and UTF8, since Leo and myself are working on Macs and Z-LaTeX mostly :) Furthermore, jEdit plugins have not seen much updates except from the compatibility with jEdit 5. More work has gone towards the Eclipse-based integration from myself, since I am more familiar with developing for Eclipse. So any updates, suggestions and problem reports are most welcome!
OK, I’m happy to try to work with the stand alone ide – that seems nice.

(Answers to your previous e-mail below)
I suppose I had better bite the bullet and get going with git :(.
I would recommend Git wholeheartedly, especially if there is at least some collaboration going..
Well it seemed to work fine. I was just reluctant to learn yet another tool but that’s old age for you :). It was really more that I couldn’t face all that maven nonsense again – but this build worked much better than previous ones which used to have perplexing maven errors all the time.

As I have worked on CZT for Eclipse, I would recommend trying the Eclipse way :) Just don't forget to set Object-Z dialect in Preferences.
OK, happy to do that, modulo query about files above.

I am not sure whether that is necessary. You need to get the czt.jar and the jEdit plugins - that should be enough?
Yes – but without building where would I get the jEdit plugins? Again I’m sure the answer is obvious to you but it wasn’t to me – sorry. Anyway, done it now.

Yes, there is a plan.
That’s very good news

We have quite some new content, just need to finish off some documentation and website updates.. and these take time and willpower :)
I do know that! Certainly the main difficulties I’ve had have all been really down to not having enough documentation so I do thoroughly applaud your efforts in that direction. I know from experience how hard it is to write for users totally unfamiliar with a project.

I would agree on limited activity in mailing lists. Leo and me are working together at Newcastle University, so we are discussing face-to-face. Otherwise it has been a bit more silent. I can only comment on project activity that I know of. Leo and myself have new content that we have added and would like to release: quite big organisational changes in builds, updates to CZT for Eclipse, verification condition generator, integration with Z/EVES theorem prover, and more changes, cleanup and fixes all over the board. There's also some new work on Circus-Time coming from University of York. So the project is not bustling, but is still going forward :)
That’s very good to hear.

Can you tell me a bit more about Z-Eves? I had thought that Z-Eves was dead: if that’s not true I’d be very interested in finding out more about it.

Thank you for the reports - hope the answers help. And all contributions are welcome of course - whether it is the memory consumption, or other upgrades :)
Answers are certainly helpful. I will try to revive the memory consumption work, though I fear it may be deep inside CZT and beyond my competence.

Very many thanks for your help

Anthony


------------------------------------------------------------------------------
Own the Future-Intel&reg; Level Up Game Demo Contest 2013
Rise to greatness in Intel's independent game demo contest.
Compete for recognition, cash, and the chance to get your game
on Steam. $5K grand prize plus 10 genre and skill prizes.
Submit your demo by 6/6/13. http://p.sf.net/sfu/intel_levelupd2d_______________________________________________
CZT-Devel mailing list
CZT...@li...<mailto:CZT...@li...>
https://lists.sourceforge.net/lists/listinfo/czt-devel