#455 Number lexing allows much bad input

[TeeEmCee]

FB version tested with: FreeBASIC Compiler - Version 0.21.0 (11-10-2008) for win32 (target:win32)

I'm writing a lexer (and eventually parser) for FB and working on literals. Rather than read the compiler source, I did extensive testing to figure out what FB does and doesn't allow, and what type it actually interprets various literals as. I didn't find any errors in type determination, but there were some bizarre literals allowed.

Here is the actual grammar for FreeBASIC integer and float literals that I have determined (partially based on what AGS posted on the forum). I believe this to be 100% accurate:
NumLiteral = [0-9]+ "." [0-9] FloatExp? Fsuffix?
| "." [0-9]+ FloatExp? Fsuffix?
| [0-9]+ I_Fsuffix?
| "&" Hex_Oct_Bin ("u" Fsuffix | I_Fsuffix)?
FloatExp = [de] [-+]? [0-9]
Fsuffix = [!#df]
I_Fsuffix = [!#df%&ul] | "ul" | "ll" | "ull"
Hex_Oct_Bin = "h" [0-9a-f]
| "o" [0-7]
| "b" [01]*

The chief offender is hexidecimal/octal/binary. Here are some examples of weird results. QB4.5 doesn't allow any of them, so I don't see why FB should
1d3! is single precision 1000.0
1e3# is double precision 1000.0
&h is integer 0; the numerals are optional.
&h# is double precision 0.0.
&of is single precision 0.0
&buf is double precistion 0.0; you can specify unsigned floating point literals!

Also, QB45 defaults to octal on seeing a &, reads &100 as &o100, but FB doesn't allow it

[TeeEmCee]

Some more:
3e
0e-

Also, I obviously forgot negatives in the above grammar.

[TeeEmCee]

Opps, I was wrong. There's also a bug in determining the type of a floating literal. Using d or D instead of e or E should indicate a double precision literal, but FB ignores that.

In -lang qb "__sizeof(1d3)" returns 4 when it should return 8.
The following code:

a# = 4.523131241D+16
b# = 2.312415153D+16
c# = a# * b#
d# = 4.523131241D+16 * 2.312415153D+16
PRINT c#
PRINT d#

produces the correct double precision answer,
1.045935722069609D+33
1.045935722069609D+33
in QB45, but the incorrect single precision values,
1.045935747160737e+033
1.045935693934723e+033
With --lang qb

In other dialects using 'e' as exponent doesn't force single precision, but that is almost certainly the desired behaviour (though incompatible with QB)

[Matthew]

OK...

1e3#/1e3! - I think this is fine. The 'e' doesn't force a float type. It just uses the default, the same you'd get with sticking a ".0" on the end of an integer. (This happens to be Double when not in lang qb.)

1d3! - Ordinarily the D would force Double dtype, but is overruled by the ! symbol. I'm OK with this too, because it would be weird to allow 1e3! and not 1d3!

&h / &o / &b - I will see about adding an error message for when no digits are present. QBASIC's error message, for some reason, is "overflow". I'd add a dedicated one though - "Expected digit" or something.

&h...# - I don't see a reason to disallow this. It's a lot less cumbersome than cdbl(&h...)

&of - I don't know about this one. Thtat it's a floating-point suffix, is OK. But I guess it's inconsistent to allow an 'f' suffix on bin/oct numbers but not (for obvious reasons) on hex numbers, so I might disable that suffix there.

...uf - no, that doesn't make any sense. Float suffixes should be disallowed after 'u'.

&123 octal - Not sure on this one. It means added complication to the val/valint/... routines, but I guess it would be necessary to do that anyway for better QB conversion compatibility.
Some would say the '&' symbol already has too many meanings. But I guess &1 doesn't open up any new ambiguities over &o1... I'll think about it.

3e - I'll think about this. It's badly formatted as an exponent, but the current docs allow its use as a suffix.

3e- - No, that probably shouldn't be allowed. But depending on the previous point, I might make the trailing '-' become a separate '-' operator. Or I might just make it show the "Expected digit error".

'-' issues - there shouldn't be any. '-' isn't part of the numeric literal grammer, but just an operator. I think as a special case it "signs" unsigned literals, but that would be intentional.

lang qb issues - Extensive work has been done on lang qb literal detection since the last release. Do please give a recent SVN build a try for an up-to-date look at lang qb literal handling.

[fxm (freebasic.net)]

• Referring to Matthew's remark:

  &o...f - I don't know about this one. Thtat it's a floating-point suffix, is OK. But I guess it's inconsistent to allow an 'f' suffix on bin/oct numbers but not (for obvious reasons) on hex numbers, so I might disable that suffix there.

I don't understand why '&o...f' or '&b...f' would be forbidden and not '&h...f'.
All represent an integer value expressed in different bases, similarly to '123' in decimal base by default.
Then adding a suffix allows to modify the default type (integer numeric).

• About:

  In -lang qb "__sizeof(1d3)" returns 4 when it should return 8.

the present release of fb (-lang qb) outputs the right value

• About the last point of TeeEmCee:
  The present release of fb (-lang qb) outputs the right values in double precision as QB45:

 1.045935722069609e+033
 1.045935722069609e+033

[fxm (freebasic.net)]

I think that the only point which would remain to fix in the parser is presently:

'&h' / '&o' / '&b' (without any digit) to disallow (or not?).

('...uf' is presently disallowed)

The other allowed syntaxes are properly covered now by the current documentation ('Literals' documentation page).

[Jeff Marshall]

Changed/Fixed in fbc 1.10.0

• '&nnn' octal literals are allowed in fbc source
• &b | &o | &h will give a warning if no digits follow

commit: [0e692197a742bc25ea53c5aeaac7c57177fdfb1e]