#1166 EDIFACT Lexer

Committed
closed
5
2016-12-05
2016-11-22
Iain Clarke
No

I've had a SCLEX_CONTAINER lexer for Scintilla for a while now, and thought it would be nice to give back, as I've got a lot of benefit from Scintilla over the years. It's fairly simple, but it gives a different style to segment headers, element and composite punctuation, and to special segment headers. Lastly, if the edifact file is not closed properly (with a ') it will show an error style.

Attached is a diff of the necessary files (the sce_edi defines, the calalogue addition, and a new LexEDIFSCT.cxx file).

I hope this benefits others!

Iain.

1 Attachments

Discussion

  • Neil Hodgson

    Neil Hodgson - 2016-11-23
    • labels: lexer --> lexer, edifact, scintilla
    • assigned_to: Neil Hodgson
     
  • Neil Hodgson

    Neil Hodgson - 2016-11-23

    Thanks for the contribution. There is a good chance the lexer will be useful to some people in its current form.

    There are a few warnings and other problems with the code currently which means it can't be included in the standard distribution.

    There should be a reference to the license as is present in other lexers. Without any indication of a license, it is dangerous to include code this in a project.

    Scintilla has to build with old compilers across several perating systems so C++11 features are not available, including auto, nullptr and override. This should change in 2017.

    Version is the version of the ILexer interface supported which should be lvOriginal. See LexerBasic.

    ../lexers/LexEDIFACT.cxx: In member function 'virtual void LexerEDIFACT::Release()':
    ../lexers/LexEDIFACT.cxx:34:10: warning: deleting object of polymorphic class type 'LexerEDIFACT' which has non-virtual destructor might cause undefined behaviour [-Wdelete-non-virtual-dtor]
       delete this;
              ^
    

    Add a virtual destructor.

    ../lexers/LexEDIFACT.cxx: In member function 'virtual void LexerEDIFACT::Lex(Sci_PositionU, Sci_Position, int, IDocument*)':
    ../lexers/LexEDIFACT.cxx:103:40: warning: overflow in implicit constant conversion [-Woverflow]
      pAccess->StartStyling(posCurrent, 0xFF);
                                            ^
    

    Use '\377' which is a char.

    ../lexers/LexEDIFACT.cxx: In member function 'Sci_Position LexerEDIFACT::InitialiseFromUNA(IDocument*, Sci_PositionU)':
    ../lexers/LexEDIFACT.cxx:180:16: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       if (startPos >= MaxLength)
                    ^
    
    ..\lexers\LexEDIFACT.cxx(40): warning C4100: 'name': unreferenced formal parameter
    ..\lexers\LexEDIFACT.cxx(44): warning C4100: 'name': unreferenced formal parameter
    ..\lexers\LexEDIFACT.cxx(48): warning C4100: 'val': unreferenced formal parameter
    ..\lexers\LexEDIFACT.cxx(48): warning C4100: 'key': unreferenced formal parameter
    ..\lexers\LexEDIFACT.cxx(56): warning C4100: 'wl': unreferenced formal parameter
    ..\lexers\LexEDIFACT.cxx(56): warning C4100: 'n': unreferenced formal parameter
    ..\lexers\LexEDIFACT.cxx(62): warning C4100: 'pointer': unreferenced formal parameter
    ..\lexers\LexEDIFACT.cxx(62): warning C4100: 'operation': unreferenced formal parameter
    ..\lexers\LexEDIFACT.cxx(103): warning C4309: 'argument': truncation of constant value
    ..\lexers\LexEDIFACT.cxx(96): warning C4100: 'initStyle': unreferenced formal parameter
    ..\lexers\LexEDIFACT.cxx(168): warning C4100: 'pAccess': unreferenced formal parameter
    ..\lexers\LexEDIFACT.cxx(168): warning C4100: 'initStyle': unreferenced formal parameter
    ..\lexers\LexEDIFACT.cxx(168): warning C4100: 'lengthDoc': unreferenced formal parameter
    ..\lexers\LexEDIFACT.cxx(168): warning C4100: 'startPos': unreferenced formal parameter
    ..\lexers\LexEDIFACT.cxx(180): warning C4018: '>=': signed/unsigned mismatch
    

    VS2008 is particularly fun:

    ..\lexers\LexEDIFACT.cxx(29) : warning C4481: nonstandard extension used: override specifier 'override'
    ..\lexers\LexEDIFACT.cxx(33) : warning C4481: nonstandard extension used: override specifier 'override'
    ..\lexers\LexEDIFACT.cxx(37) : warning C4481: nonstandard extension used: override specifier 'override'
    ..\lexers\LexEDIFACT.cxx(41) : warning C4481: nonstandard extension used: override specifier 'override'
    ..\lexers\LexEDIFACT.cxx(45) : warning C4481: nonstandard extension used: override specifier 'override'
    ..\lexers\LexEDIFACT.cxx(49) : warning C4481: nonstandard extension used: override specifier 'override'
    ..\lexers\LexEDIFACT.cxx(53) : warning C4481: nonstandard extension used: override specifier 'override'
    ..\lexers\LexEDIFACT.cxx(57) : warning C4481: nonstandard extension used: override specifier 'override'
    ..\lexers\LexEDIFACT.cxx(60) : warning C4481: nonstandard extension used: override specifier 'override'
    ..\lexers\LexEDIFACT.cxx(61) : warning C4481: nonstandard extension used: override specifier 'override'
    ..\lexers\LexEDIFACT.cxx(63) : warning C4481: nonstandard extension used: override specifier 'override'
    ..\lexers\LexEDIFACT.cxx(38) : error C2065: 'nullptr' : undeclared identifier
    ..\lexers\LexEDIFACT.cxx(46) : error C2065: 'nullptr' : undeclared identifier
    ..\lexers\LexEDIFACT.cxx(54) : error C2065: 'nullptr' : undeclared identifier
    ..\lexers\LexEDIFACT.cxx(64) : error C2065: 'nullptr' : undeclared identifier
    ..\lexers\LexEDIFACT.cxx(103) : warning C4309: 'argument' : truncation of constant value
    ..\lexers\LexEDIFACT.cxx(124) : error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
    ..\lexers\LexEDIFACT.cxx(180) : warning C4018: '>=' : signed/unsigned mismatch
    ..\lexers\LexEDIFACT.cxx(199) : warning C4127: conditional expression is constant
    ..\lexers\LexEDIFACT.cxx(213) : error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
    ..\lexers\LexEDIFACT.cxx(213) : error C2440: 'initializing' : cannot convert from 'const char *' to 'int'
            There is no context in which this conversion is possible
    ..\lexers\LexEDIFACT.cxx(215) : error C2109: subscript requires array or pointer type
    ..\lexers\LexEDIFACT.cxx(215) : fatal error C1903: unable to recover from previous error(s); stopping compilation
    

    Calling pAccess->BufferPointer() can be expensive on large files as it copies (on average) half the document with each call to make all the text contiguous when there has been a change. So, if the user is typing, the copy occurs after each character is typed. Most lexer code should use LexAccessor::operator[] or LexAccessor::SafeGetCharAt which don't collapse the gap.

     
  • Iain Clarke

    Iain Clarke - 2016-11-23

    I've sorted out the above issues. I don't have access to GCC etc, but the new cpp file compiles cleanly in vs2015 and 2005. I think it's safe to assume it compiles cleanly in inbetween versions too!

    Feel free to edit the comments at the start or suggest changes. I am happy for licensing to bewide open, though vanity makes me want my name to stay up there.

     
  • Iain Clarke

    Iain Clarke - 2016-11-23

    And here's one with folding operational too. It folds at UNH segments, headers for messages inside the overall package.

     
  • Neil Hodgson

    Neil Hodgson - 2016-11-23

    OK, that's better.

    However there is a warning from VS 2015:

    ..\lexers\LexEDIFACT.cxx(244) : warning C4127: conditional expression is constant
    

    The code is a little weird, using a single iteration loop to be able to break instead of using ifs but it also appears wrong. The call to memcpy will always succeed since memcpy returns its destination argument. Perhaps you want memcmp?

    FindPreviousEnd can be marked const.

    LexerModule.h should be at the end of the list of #includes. Scintilla has a canonical header order defined in scripts/HeaderOrder.txt and a checking script scripts/HeaderCheck.py to ensure there aren't header ordering bugs and precompiled headers work reasonably.

     
  • Iain Clarke

    Iain Clarke - 2016-11-24

    The memcpy was a stupid typo - I had already fixed it, but had to go back in time to an earlier version when I messed up folding, and missed that memcpy.

    I quite like the do { } while (0); with breaks when you have a lot of failure conditions to jump out from, as it prevents a lot of if nesting. but as there's not much of an endurance test here, I've changed to a couple of if's instead.

    All my helper functions have been const'd, except for InitialiseFromUNA (). I am a big fan of const, and the new override modifiers - anything that makes it harder to do silly mistakes must be good.

    The include order has been fixed, and a comment added to remind future Iain not to mess it up later.

     
  • Neil Hodgson

    Neil Hodgson - 2016-11-25
    • Group: Completed --> Committed
     
  • Neil Hodgson

    Neil Hodgson - 2016-11-27

    Additional changes committed to enable building on macOS: [94f0d4].

     

    Related

    Commit: [94f0d4]

  • Neil Hodgson

    Neil Hodgson - 2016-12-05
    • status: open --> closed
     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks