Re: [Flex-help] Including Newline and Carriage Return in yytext

[Chris v. <che...@gm...>]

I tried that out on my machine (OSX, flex 2.5.35) and it works fine:

% perl -e 'open($fh, q{>}, "t.in"); print {$fh} "MY STRING=asdf\r\n";
close($fh);'
% ./a.out < t.in
We got the string <asdf
>
Total match length is <16> long
The characters in yytext are:
    <4d>
    <59>
    <20>
    <53>
    <54>
    <52>
    <49>
    <4e>
    <47>
    <3d>
    <61>
    <73>
    <64>
    <66>
    <d>
    <a>
%

My guess is that your input isn't what you think it is, maybe.  Perhaps the
\r and \n are swapped, or there is no \r at all?  If you created the file
on linux and imported it to Windows, it would likely be missing the extra
\r, for example.  I think I heard even later versions of Windows have
stopped with the \r char.

I did change two things in the code, though neither should be relevant:
1. added "%option noyywrap"
2. changed the printf of yyleng from %d to %zu.

-Chris



On Fri, Dec 12, 2014 at 6:29 AM, Allen Blaylock <Al...@ep...>
wrote:
>
> Here is an example parser:
>
>
> %option never-interactive
> %option reentrant
>
> %{
> char my_buf[1024];
> %}
>
> %%
>
> MY[ ]STRING=.*\r?\n {
>         int i;
>         printf("We got the string <%s>\n",&(yytext[10]));
>         printf("Total match length is <%d> long\n",yyleng);
>         printf("The characters in yytext are:\n");
>         for(i = 0; i < strlen(yytext); ++i)
>         {
>                 printf("\t<%x>\n",(unsigned int)yytext[i]);
>         }
>         memcpy(my_buf,yytext,yyleng);
> }
>
> . {
>         // Do nothing, just consume the text
> }
>
> %%
>
> int main(int argc, char ** argv)
> {
>         yyscan_t scanner;
>         yylex_init( &scanner );
>         yyset_in(stdin, scanner);
>     yylex(scanner);
>         yylex_destroy(scanner);
>     return 0;
> }
>
>
> I then take this file and use it against the following text in a file:
>
> Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin ornare orci
> id
> neque imperdiet tristique. Nulla faucibus, lacus ut aliquet mollis, mi
> tortor
> fermentum tellus, suscipit MY STRING=THE QUICK BROWN FOX
> bibendum lorem enim vitae massa. Nulla ultrices mi
> vel lorem tempor tincidunt. Sed urna tortor, commodo at maximus eget,
> pretium ac dolor. Nam auctor
>
>
> ## END OF FILE
>
> It is important to note that the line endings for the above text should be
> (carriage return)(line feed).
>
> What I would like to see in yytext is "MY STRING=THE QUICK BROWN
> FOX(\0x0D\0x0A)" but what I get is: "MY STRING=THE QUICK BROWN FOX(\0x0A)"
> so there is no carriage return.
> It is important for my parser to be able to parse certain strings byte for
> byte as they appear in the input file.
>
>
> Allen Blaylock
>
> -----Original Message-----
> From: Harjot kaur [mailto:har...@gm...]
> Sent: Thursday, December 11, 2014 10:29 PM
> To: Allen Blaylock
> Subject: Re: [Flex-help] Including Newline and Carriage Return in yytext
>
> On Fri, Dec 12, 2014 at 2:51 AM, Allen Blaylock <Al...@ep...>
> wrote:
> > I am struggling to get the newline and carriage return characters to
> appear in the yytext "buffer."
> >
> I think it shows your token is not declared in yytext or yylval.
> > For instance let's say I was trying to match with the rule:
> >
> > MY[ ]STRING=.*\r?\n {
> >               memcpy(my_buf,yytext,yyleng); }
> Are u using this function in Flex file or Bison file?
> >
> > Which you can see copies the buffer exactly to my_buf. Unfortunatly I
> only get the portion of the string matched by MY[ ]STRING=.* but not the
> optional carriage return or newline character.
>
> Is it solved out or not? Please show your code.
>
>
>
>
> --
> Harjot Kaur
> Blog: harjotpandher93.wordpress.com
> github: https://github.com/jotpandher
> " ਕੰਮ ਨਾਲ ਪਿਆਰ ਕਰੀਏ ਫਿਰ ਕੰਮ ਹੀ ਸ਼ੋਂਕ ਬਣ ਜਾਏਗਾ "
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
>
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> --
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help
>