1) I have just d/l'ed everything from the CVS and tried to
compile them.
It does not compile because it can not find SettingsTypes.pas
Is it because SF's CVS is outdated or perhaps an oversight.
2) I have scanned all the pas files I have for files larger than 500 K,
I have found
some 46 files. I was going to test them with JCF to see if anything
would break,
but since I can not compile JCF I thought you might like to do that.
I keep the sourceforge CVS up to date with the latest code that passes all the regression tests. The last commit was on Sunday night.
The lack of that file in CVS is probably an oversight on my part. I will look into it tonight. Despite the initial config for CVS being a bit painfull, it's good that you are doing this, it's really the best way to make sure that the CVS code is complete.
If you are desperate to get on with compiling, I suggest that you find the file in source given in the Beta 4 zip file. Hopefully it will not have changed since that point.
Anyway aside from the fact that there's just more stuff to go wrong in large files than in small files, there should not be any reason why large files are inherently more prone to failing. I was contemplating running a sample of the smaller source files that ship with Delphi through the formatter to see if I could pick up any breakages.
As before, conditional compilation with $IFDEF remains a known issue that will not be adressed right away. However it is becoming clear that when the current ongoing batch of parse fails are all fixed, it should be next up.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Here are a few observations based on formatting of my
large test units.
First, it seems there is a great deal of memory leaks
especially after it throws an exception (i.e. on finding
something it does not like).
Personally, I dont consider memory leaks a show stopper;
these can be traced out once every other thing is handled.
But, FYI, it is not uncommon that the memory usage reaches
more than 300 MB, this is way more than any other application
I have seen.
Then again, we all know that this is not a true measure of
memory footprint since it is a value displayed by Windows
Task Manager.
Secondly, and again, this is not meant to be a show stopper
either but is something to take care of later: JCF uses
an incredible amount of CPU power. When I run it, it is
not uncommon that it consumes 95-100% CPU for extended
periods of time; during which it does not even paint
itself --seen by the OS as 'Not Responding'.
Finally, when I tried to format the units, I have found
a lot off odd cases which needs to be taken care of.
I have copied of those cases, not as test cases, and list
them below.
Even though I know of your dislike of large units, I believe
it would be appropriate to use them at least at this stage
to see what breaks and when --especially since it is more
likley that obscure constructs and tricks are used in these
large units. There is also the fact that one might best be
able to tune JCF for speed and against memory leaks by using
larger units.
{cDataStructs.pas}
Function ASimpleType.GetParameterDefinitions (const MethodName :
String) : TParameterAttributesArray;
var M : TSimpleTypeMethod;
I, L : Integer;
Begin
M := SimpleTypeMethodFromName (MethodName);
if M = stmNone then
Result := inherited GetParameterDefinitions (MethodName) else
begin
L := SimpleTypeHardCodedMethodParams [M];
SetLength (Result, L);
For I := 0 to L - 1 do
Result [I] := [paRequired];
end;
End;
{XDOM.pas}
function IsXmlChar(const S: WideChar): boolean;
begin
Case Word(S) of
$0009,$000A,$000D,$0020..$D7FF,$E000..$FFFD, // Unicode below $FFFF
$D800..$DBFF, // High surrogate of Unicode character
[$10000..$10FFFF]
$DC00..$DFFF: // Low surrogate of Unicode character
[$10000..$10FFFF]
result:= true;
else
result:= false;
end;
end;
{ffsreng.pas}
function TffSrBaseCursor.CanClose(const Mark : Boolean) : Boolean;
begin
{ Cursor can be closed if it is not in a transaction or if the table
is
temporary. }
Result := (bcDatabase.Transaction = nil) or
(fffaTemporary in bcTable.Files[0].fiAttributes);
if Result then
Result := inherited CanClose(Mark)
{!!.05 - Start}
else if (bcDatabase.Transaction <> nil) then
bcCloseWTrans := True;
{!!.05 - End}
end;
{IpHtml.pas}
function GetFloatProperty : string;
const
Precisions : array[TFloatType] of Integer = (7, 15, 18, 18, 19);
begin
Result := FloatToStrF(GetFloatProp(AObject, PI), ffGeneral,
Precisions[GetTypeData(GetPropType)^.FloatType], 0);
end;
{Unicoder.pas}
Procedure UCS4CodeToUTF8String(ACode: LongWord; Var AOutIndex:
LongWord);
Const
MaxCode: Array[0..5] Of LongWord = ($7F, $7FF, $FFFF, $1FFFFF,
$3FFFFFF, $7FFFFFFF);
FirstByte: Array[0..5] Of Byte = (0, $C0, $E0, $F0, $F8, $FC);
Var
Mo, Index1, StartIndex1: LongWord;
Begin
Mo := 0; // get number of bytes
While ((ACode > MaxCode[Mo]) And (Mo < 5)) Do Inc(Mo);
StartIndex1 := AOutIndex;
AOutIndex := StartIndex1 + Mo;
If AOutIndex >= FConvertBufferSize Then IncBufSize(AOutIndex);
For Index1 := AOutIndex Downto StartIndex1 + 1 Do Begin {fill bytes
from rear end}
PChar(FConvertBuffer)[Index1] := Char($80 Or (ACode And $3F));
ACode := ACode Shr 6;
End;
PChar(FConvertBuffer)[StartIndex1] := Char(FirstByte[Mo] Or ACode);
// fill first byte
End;
{WABD_Objects.pas}
function TWABD_HTML.Object_To_HTML:string;
begin
Result:=inherited Object_To_HTML;
end;
====================END TEXT===================
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
thanks for that. It seems there's a lot to do. I have started testing with some of Delphi's .pas files and have encountered some of these same issues – specifically "property … default 0;" "Result := inherited …" and "function contains".
This code is not yet in CVS as the test cases are not yet complete and a regression has not yet been run (Every bug found gets a test case that exhibits it. All tests cases must pass before I check in. know I'm being strict on this, it's worth it).
My current plan is
– check in remaining outstanding CVS files: tonight.
– Release a beta that can parse most or all pas files that ship with Delphi: 2-3 weeks
– Fix memory leaks and look at memory size issues
– Look at run time issues
- conditional compilation issues
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
> thanks for that. It seems there's a lot to do. I have started testing
> with some of Delphi's .pas files and have encountered some of these
> same issues – specifically "property … default 0;"
> "Result := inherited …" and "function contains".
Indeed, there does seem to be a lot to do -however , definitely not
as much as when there was no JCF:-)
> This code is not yet in CVS as the test cases are not yet complete
> and a regression has not yet been run (Every bug found gets a test
> case that exhibits it. All tests cases must pass before I check in.
> know I'm being strict on this, it's worth it).
It definitely is worth it. It takes a lot of stamina to do that though.
> My current plan is
> – check in remaining outstanding CVS files: tonight.
> – Release a beta that can parse most or all pas files that ship
> with Delphi: 2-3 weeks
>
> – Fix memory leaks and look at memory size issues
> – Look at run time issues
> - conditional compilation issues
I am wondering if the code could not be tweaked somewhat so that
during these tests it outputs a text log and not stop at the first
error.
I bid you good luck, patience, persistence and perseverence :-))
Cheers,
Adem
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
" Release a beta that can parse most or all pas files that ship with Delphi: 2-3 weeks"
Well, I have put beta 5 out because of the nubmer of fixes (see the changelog) I haven't worked with all the delphi source files (in fact I got as far as files starting with 'F', thats around 250 files out of 1000). Note that this excludes files larger than 100b in size, and files that cannot be parsed due to preprocessor/ conditional compilation.
But given that the same bugs occur several times, and will be fixed the first time that they are found (also empirically, the bug rate was highest in the earlier files), I think that more than half of the parse bugs lurking there have been fixed.
Next intermediate goal is to complete this process, and release another beta in 2-3 weeks time again with all resulting fixes, and fixes to anything that turns up in the sourceforge bug tracker.
Also possibly memory issues - if anyone can pinpoint a memory leak I'd be interested...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
> But given that the same bugs occur several times, and will be fixed
> the first time that they are found (also empirically, the bug rate
> was highest in the earlier files), I think that more than half of the
> parse bugs lurking there have been fixed.
This is very good news.
> Next intermediate goal is to complete this process, and release
> another beta in 2-3 weeks time again with all resulting fixes, and
> fixes to anything that turns up in the sourceforge bug tracker.
>
> Also possibly memory issues - if anyone can pinpoint a memory leak
> I'd be interested...
You probably know of this, but in case you didnt, here is a cute
little piece of code that does usually better than commercial
alternatives. It is quite unintrusive too.
Anthony,
Two things:
1) I have just d/l'ed everything from the CVS and tried to
compile them.
It does not compile because it can not find SettingsTypes.pas
Is it because SF's CVS is outdated or perhaps an oversight.
2) I have scanned all the pas files I have for files larger than 500 K,
I have found
some 46 files. I was going to test them with JCF to see if anything
would break,
but since I can not compile JCF I thought you might like to do that.
Archive sizes (same files, different compression)
ACE:3.8 MB (WinACE v2.2)
RAR: 4 MB (WinRAR v3.x)
ZIP: 5 MB (WinZip v8.1SR1)
Please drop me an email if you'd like me to send you these test files
(and what archive
format you'd prefer).
adembaba at excite dot com
Cheers,
Adem
I keep the sourceforge CVS up to date with the latest code that passes all the regression tests. The last commit was on Sunday night.
The lack of that file in CVS is probably an oversight on my part. I will look into it tonight. Despite the initial config for CVS being a bit painfull, it's good that you are doing this, it's really the best way to make sure that the CVS code is complete.
If you are desperate to get on with compiling, I suggest that you find the file in source given in the Beta 4 zip file. Hopefully it will not have changed since that point.
Anyway aside from the fact that there's just more stuff to go wrong in large files than in small files, there should not be any reason why large files are inherently more prone to failing. I was contemplating running a sample of the smaller source files that ship with Delphi through the formatter to see if I could pick up any breakages.
As before, conditional compilation with $IFDEF remains a known issue that will not be adressed right away. However it is becoming clear that when the current ongoing batch of parse fails are all fixed, it should be next up.
I managed to compile them by taking units from beta4.
Apparently, these are the missing units from the CVS.
Settings\SettingsTypes.pas
Process\Capitalisation\UnitNameCaps.pas
ReadWrite\EditorConverter.pas
ReadWrite\EditorReader.pas
ReadWrite\EditorWriter.pas
Here are a few observations based on formatting of my
large test units.
First, it seems there is a great deal of memory leaks
especially after it throws an exception (i.e. on finding
something it does not like).
Personally, I dont consider memory leaks a show stopper;
these can be traced out once every other thing is handled.
But, FYI, it is not uncommon that the memory usage reaches
more than 300 MB, this is way more than any other application
I have seen.
Then again, we all know that this is not a true measure of
memory footprint since it is a value displayed by Windows
Task Manager.
Secondly, and again, this is not meant to be a show stopper
either but is something to take care of later: JCF uses
an incredible amount of CPU power. When I run it, it is
not uncommon that it consumes 95-100% CPU for extended
periods of time; during which it does not even paint
itself --seen by the OS as 'Not Responding'.
Finally, when I tried to format the units, I have found
a lot off odd cases which needs to be taken care of.
I have copied of those cases, not as test cases, and list
them below.
Even though I know of your dislike of large units, I believe
it would be appropriate to use them at least at this stage
to see what breaks and when --especially since it is more
likley that obscure constructs and tricks are used in these
large units. There is also the fact that one might best be
able to tune JCF for speed and against memory leaks by using
larger units.
Just my 0.02
Cheers,
Adem
====================BEGIN TEXT===================
Const
{Advgrid.pas}
Numeric_Characters = [$30..$39, $8, $9, $D, ord('-'), ord(#27)];
Positive_Numeric_Characters = [$30..$39, $8, $9, $D, ord(#27)];
Float_Characters = [$30..$39, $8, $9, $D, ord('-'), ord(','),
ord('.'), ord(#27)];
CSVSeparators: Array[1..10] Of char = (',', ';', '#', #9, '|', '@',
'*', '-', '+', '&');
Type
{DirectX.pas}
TDIEnumEffectsInFileCallback = function(const lpDiFileEf:
TDIFileEffect; pvRef: Pointer): BOOL; far pascal;
LPDIENUMEFFECTSINFILECALLBACK = TDIEnumEffectsInFileCallback;
{ComCtrls.pas}
property TabHeight: Smallint read FTabSize.Y write SetTabHeight
default 0;
{OpWrd2K.pas}
property StatusBar: WideString writeonly dispid 97;
{DLL96V1.pas}
{DLL Callbacks.These Callbacks are dispatched to the event handler.}
TCallBackTags = function (itag : Word;
ilen : SmallInt;
lvalue : LongInt;
pStrData : PChar;
d_object : longint): SmallInt cdecl;
type
TCurImageList = {$IFDEF
DELPHI4}TCustomImageList{$ELSE}TImageList{$ENDIF};
{dxBar.pas}
{dxMasterView.pas}
TCurDesigner =
{$IFDEF DELPHI6}IUnknown{$ELSE}{$IFDEF
DELPHI4}IDesigner{$ELSE}TDesigner{$ENDIF}{$ENDIF};
{dxTL.pas}
function GetDragImages: {$IFDEF
DELPHI4}TDragImageList{$ELSE}TCustomImageList{$ENDIF}; override;
{ElTree.pas}
type
TElHeaderSection = ElHeader.TElHeaderSection;
{JwaWinBase.pas}
_SYSTEMTIME = Windows._SYSTEMTIME;
{JMySSL.pas}
function f_sk_value(arg0: PSTACK; arg1: Integer):PChar; cdecl
{MSHTML.pas}
function contains(const pChild: IHTMLElement): WordBool; safecall;
{Excel8G2.pas}
ShapeNodesDisp = dispinterface
['{000C0319-0000-0000-C000-000000000046}']
function Item(Index: OleVariant): ShapeNode; dispid 0;
property _NewEnum: IUnknown readonly dispid -4;
procedure Delete(Index: SYSINT); dispid 11;
end;
{FUNCkyLib_TLB.pas}
IFUNCky = interface(IDispatch)
['{F1600000-3D1E-11D1-AC88-0000C085C327}']
function At(const SearchStr: WideString; const Str: WideString):
OleVariant; safecall;
end;
{SyntaxEd.pas}
TEditableEntry = Record
EditName: String[_MaxEditableLen];
End;
{Qt.pas}
function QWidget_mouseGrabber(): QWidgetH; cdecl;
{dbisamen.pas}
implementation
{$IFDEF D6ORHIGHER}
uses DB, FMTBcd, DBISAMTb, Forms, DBISAMSq;
{$ELSE}
uses DB, DBISAMTb, Forms, DBISAMSq;
{$ENDIF}
{cDataStructs.pas}
Function ASimpleType.GetParameterDefinitions (const MethodName :
String) : TParameterAttributesArray;
var M : TSimpleTypeMethod;
I, L : Integer;
Begin
M := SimpleTypeMethodFromName (MethodName);
if M = stmNone then
Result := inherited GetParameterDefinitions (MethodName) else
begin
L := SimpleTypeHardCodedMethodParams [M];
SetLength (Result, L);
For I := 0 to L - 1 do
Result [I] := [paRequired];
end;
End;
{XDOM.pas}
function IsXmlChar(const S: WideChar): boolean;
begin
Case Word(S) of
$0009,$000A,$000D,$0020..$D7FF,$E000..$FFFD, // Unicode below $FFFF
$D800..$DBFF, // High surrogate of Unicode character
[$10000..$10FFFF]
$DC00..$DFFF: // Low surrogate of Unicode character
[$10000..$10FFFF]
result:= true;
else
result:= false;
end;
end;
{ffsreng.pas}
function TffSrBaseCursor.CanClose(const Mark : Boolean) : Boolean;
begin
{ Cursor can be closed if it is not in a transaction or if the table
is
temporary. }
Result := (bcDatabase.Transaction = nil) or
(fffaTemporary in bcTable.Files[0].fiAttributes);
if Result then
Result := inherited CanClose(Mark)
{!!.05 - Start}
else if (bcDatabase.Transaction <> nil) then
bcCloseWTrans := True;
{!!.05 - End}
end;
{IpHtml.pas}
function GetFloatProperty : string;
const
Precisions : array[TFloatType] of Integer = (7, 15, 18, 18, 19);
begin
Result := FloatToStrF(GetFloatProp(AObject, PI), ffGeneral,
Precisions[GetTypeData(GetPropType)^.FloatType], 0);
end;
{Unicoder.pas}
Procedure UCS4CodeToUTF8String(ACode: LongWord; Var AOutIndex:
LongWord);
Const
MaxCode: Array[0..5] Of LongWord = ($7F, $7FF, $FFFF, $1FFFFF,
$3FFFFFF, $7FFFFFFF);
FirstByte: Array[0..5] Of Byte = (0, $C0, $E0, $F0, $F8, $FC);
Var
Mo, Index1, StartIndex1: LongWord;
Begin
Mo := 0; // get number of bytes
While ((ACode > MaxCode[Mo]) And (Mo < 5)) Do Inc(Mo);
StartIndex1 := AOutIndex;
AOutIndex := StartIndex1 + Mo;
If AOutIndex >= FConvertBufferSize Then IncBufSize(AOutIndex);
For Index1 := AOutIndex Downto StartIndex1 + 1 Do Begin {fill bytes
from rear end}
PChar(FConvertBuffer)[Index1] := Char($80 Or (ACode And $3F));
ACode := ACode Shr 6;
End;
PChar(FConvertBuffer)[StartIndex1] := Char(FirstByte[Mo] Or ACode);
// fill first byte
End;
{WABD_Objects.pas}
function TWABD_HTML.Object_To_HTML:string;
begin
Result:=inherited Object_To_HTML;
end;
====================END TEXT===================
SettingsTypes.pas is now in CVS.
thanks for that. It seems there's a lot to do. I have started testing with some of Delphi's .pas files and have encountered some of these same issues – specifically "property … default 0;" "Result := inherited …" and "function contains".
This code is not yet in CVS as the test cases are not yet complete and a regression has not yet been run (Every bug found gets a test case that exhibits it. All tests cases must pass before I check in. know I'm being strict on this, it's worth it).
My current plan is
– check in remaining outstanding CVS files: tonight.
– Release a beta that can parse most or all pas files that ship with Delphi: 2-3 weeks
– Fix memory leaks and look at memory size issues
– Look at run time issues
- conditional compilation issues
Hi Anthony,
> thanks for that. It seems there's a lot to do. I have started testing
> with some of Delphi's .pas files and have encountered some of these
> same issues – specifically "property … default 0;"
> "Result := inherited …" and "function contains".
Indeed, there does seem to be a lot to do -however , definitely not
as much as when there was no JCF:-)
> This code is not yet in CVS as the test cases are not yet complete
> and a regression has not yet been run (Every bug found gets a test
> case that exhibits it. All tests cases must pass before I check in.
> know I'm being strict on this, it's worth it).
It definitely is worth it. It takes a lot of stamina to do that though.
> My current plan is
> – check in remaining outstanding CVS files: tonight.
> – Release a beta that can parse most or all pas files that ship
> with Delphi: 2-3 weeks
>
> – Fix memory leaks and look at memory size issues
> – Look at run time issues
> - conditional compilation issues
I am wondering if the code could not be tweaked somewhat so that
during these tests it outputs a text log and not stop at the first
error.
I bid you good luck, patience, persistence and perseverence :-))
Cheers,
Adem
" Release a beta that can parse most or all pas files that ship with Delphi: 2-3 weeks"
Well, I have put beta 5 out because of the nubmer of fixes (see the changelog) I haven't worked with all the delphi source files (in fact I got as far as files starting with 'F', thats around 250 files out of 1000). Note that this excludes files larger than 100b in size, and files that cannot be parsed due to preprocessor/ conditional compilation.
But given that the same bugs occur several times, and will be fixed the first time that they are found (also empirically, the bug rate was highest in the earlier files), I think that more than half of the parse bugs lurking there have been fixed.
Next intermediate goal is to complete this process, and release another beta in 2-3 weeks time again with all resulting fixes, and fixes to anything that turns up in the sourceforge bug tracker.
Also possibly memory issues - if anyone can pinpoint a memory leak I'd be interested...
Anthony Steele wrote:
> But given that the same bugs occur several times, and will be fixed
> the first time that they are found (also empirically, the bug rate
> was highest in the earlier files), I think that more than half of the
> parse bugs lurking there have been fixed.
This is very good news.
> Next intermediate goal is to complete this process, and release
> another beta in 2-3 weeks time again with all resulting fixes, and
> fixes to anything that turns up in the sourceforge bug tracker.
>
> Also possibly memory issues - if anyone can pinpoint a memory leak
> I'd be interested...
You probably know of this, but in case you didnt, here is a cute
little piece of code that does usually better than commercial
alternatives. It is quite unintrusive too.
http://v.mahon.free.fr/pro/freeware/memcheck/
Did you give it a try?
BTW, is the new beta same as that in the current CVS?
Cheers,
Adem
> here is a cute little piece of code
> http://v.mahon.free.fr/pro/freeware/memcheck/
I have used memcheck before, it is great. I shall use it on JCF sooner or later.
> BTW, is the new beta same as that in the current CVS?
Nope, the CVS already has another parse error fix. More to come :)