dev-cpp-users

[Dev-C++] Tokenize function

[Alexsandro M. <mei...@gm...>]

Dear all,

I have build a function to tokenize a sentence and to return the words in my
main function. It seems ok, but it's not working the way I intend.

The code is:

main()
{
  char str[50];
  char *array[50];
  printf("Token 0.1\nDeveloped by Alexsandro Meireles\nType");
  printf(" the sentence: ");
  gets(str);
  tokenizar(str);
  printf("\n");
  printf("Word #%d is %s\n",0, array[0]);   //PROBLEM IS HERE!!!!!
  system("pause");
}


I need to have access to the words returned by the function tokenizar, but
what I get as result, for example for "a casa pegou fogo." is:

Word #0 is 'a'
Word #1 is 'casa'
Word #2 is 'pegou'
Word #3 is 'fogo'

AND THEN A LOOP

Word #0 is Word #0 is Word #0 is Word #0 is ...

IN OTHER WORDS, I CAN'T HAVE ACCESS TO THE ARRAY OF WORDS CREATED INSIDE
'tokenizar'.

What am I doing wrong?

Thanks in advance!

-- 
Prof. Dr. Alexsandro Meireles, linguist
Federal University of Espírito Santo
Departamento de Línguas e Letras
Av. Fernando Ferrari, 514. Campus Universitário
Goiabeiras. 29075-910
Vitória-ES. Brazil
mei...@gm...
+55-27-41021734

Re: [Dev-C++] Tokenize function

[Kononov A. <cl...@ya...>]

It seems like you don't actually dealing with "array[]" inside tokenizar,<br />so, you try to use uninitialised "array[0]" vaule ...<br /><br />05.03.09, 14:36, "Alexsandro Meireles" &lt;mei...@gm...&gt;:

<BLOCKQUOTE mce_style="border-left:1px solid #CCCCCC;margin:0pt 0pt 0pt 0.8ex;padding-left:1em;" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1em;" >Dear all,<br /><br />I have build a function to tokenize a sentence and to
return the words in my main function. It seems ok, but it's not working
the way I intend.<br /><br />The code is:<br /><br />main()<br />{<br />  char str[50];   <br />
  char *array[50];<br />  printf("Token 0.1\nDeveloped by Alexsandro Meireles\nType");<br />  printf(" the sentence: ");<br />  gets(str);<br />  tokenizar(str);<br />  printf("\n");<br />  printf("Word #%d is %s\n",0, array[0]);   //PROBLEM IS HERE!!!!!<br />

  system("pause");<br />}<br /><br /><br />I
need to have access to the words returned by the function tokenizar,
but what I get as result, for example for "a casa pegou fogo." is:<br /><br />Word #0 is 'a'<br />
Word #1 is 'casa'<br />Word #2 is 'pegou'<br />Word #3 is 'fogo'<br /><br />AND THEN A LOOP<br /><br />Word #0 is Word #0 is Word #0 is Word #0 is ...<br /><br />IN OTHER WORDS, I CAN'T HAVE ACCESS TO THE ARRAY OF WORDS CREATED INSIDE 'tokenizar'.<br />
<br />What am I doing wrong?<br /><br />Thanks in advance!<br /><br />-- <br />Prof. Dr. Alexsandro Meireles, linguist<br />Federal University of Espírito Santo<br />Departamento de Línguas e Letras<br />Av. Fernando Ferrari, 514. Campus Universitário<br />

Goiabeiras. 29075-910<br />Vitória-ES. Brazil<br /><A target="_blank" mce_href="mailto:mei...@gm..." href="mailto:mei...@gm..." >mei...@gm...</A><br />+55-27-41021734
</BLOCKQUOTE><br /><br />-- <br />С уважением, Алексей.

[Dev-C++] Fwd: Tokenize function

[Derek C. <de...@ci...>]

Forgot to reply all to include the group!


---------- Forwarded message ----------
From: Derek Clarke <de...@ci...>
Date: Thu, Mar 5, 2009 at 10:45 PM
Subject: Re: [Dev-C++] Tokenize function
To: Alexsandro Meireles <mei...@gm...>


Can you tell us what's inside the tokenizar function?

Also the code you have published doesn't match the results you report
- you are obviously printing the words in a loop, so where's the loop
in main?

On Thu, Mar 5, 2009 at 11:36 AM, Alexsandro Meireles
<mei...@gm...> wrote:
> Dear all,
>
> I have build a function to tokenize a sentence and to return the words in my
> main function. It seems ok, but it's not working the way I intend.
>
> The code is:
>
> main()
> {
>   char str[50];
>   char *array[50];
>   printf("Token 0.1\nDeveloped by Alexsandro Meireles\nType");
>   printf(" the sentence: ");
>   gets(str);
>   tokenizar(str);
>   printf("\n");
>   printf("Word #%d is %s\n",0, array[0]);   //PROBLEM IS HERE!!!!!
>   system("pause");
> }
>
>
> I need to have access to the words returned by the function tokenizar, but
> what I get as result, for example for "a casa pegou fogo." is:
>
> Word #0 is 'a'
> Word #1 is 'casa'
> Word #2 is 'pegou'
> Word #3 is 'fogo'
>
> AND THEN A LOOP
>
> Word #0 is Word #0 is Word #0 is Word #0 is ...
>
> IN OTHER WORDS, I CAN'T HAVE ACCESS TO THE ARRAY OF WORDS CREATED INSIDE
> 'tokenizar'.
>
> What am I doing wrong?
>
> Thanks in advance!
>
> --
> Prof. Dr. Alexsandro Meireles, linguist
> Federal University of Espírito Santo
> Departamento de Línguas e Letras
> Av. Fernando Ferrari, 514. Campus Universitário
> Goiabeiras. 29075-910
> Vitória-ES. Brazil
> mei...@gm...
> +55-27-41021734
> ------------------------------------------------------------------------------
> Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
> -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
> -Strategies to boost innovation and cut costs with open source participation
> -Receive a $600 discount off the registration fee with the source code: SFAD
> http://p.sf.net/sfu/XcvMzF8H
> _______________________________________________
> Dev-cpp-users mailing list
> Dev...@li...
> TO UNSUBSCRIBE: http://www23.brinkster.com/noicys/devcpp/ub.htm
> https://lists.sourceforge.net/lists/listinfo/dev-cpp-users
>
>

Re: [Dev-C++] Tokenize function

[Derek C. <de...@ci...>]

You've just mixed in too much test code with the production stuff!

As given, the tokenizar function works fine.

However combine it with the main program given and the problem is that
 array[] in the main program is not initialised. Trying to dereference
uninitialised pointers (in the printf) doesn't usually mean that good
things happen!

You have a general problem with tokenizar in that there is no way to
get the tokens back to the main program. You might want to add another
parameter or two, i.e. make the signature void tokenizar(char
*palavra, char* array[], unsigned int size_of_array)

Now you can pass in the token array and get results back.

On Thu, Mar 5, 2009 at 10:49 PM, Alexsandro Meireles
<mei...@gm...> wrote:
> Here it is.
>
> #include<string.h>  //É DEMAIS ESTE SCRIPT. DÁ PARA REFERENCIAR CADA SÍLABA
> E CADA PALAVRA
> #include<stdio.h>  //Se a sílaba 1 for X e sílaba 2 for Y. Se a palavra 1
> for X e a palavra 2 for Y.
>
> void tokenizar(char *palavra) //void acentua(char *palavra)
> {
>   //char search_string[]="Woody Norm Cliff";
>  // char str[50];   //Que tamanho devo botar aqui
>   char *array[50];
>   int loop;
>
>  // gets(str);
>
>
>   array[0]=strtok(palavra," ,");   //Analisa para ver se tem algo escrito. O
> primeiro string no caso tem que ser separado por um espaço vazio.
>   if(array[0]==NULL)
>    {
>     printf("No test to search.\n");
>     exit(0);
>    }
>
>    for(loop=1;loop<50;loop++)
>    {
>     array[loop]=strtok(NULL," ,.:!?");    //Separadores usados " " , : .
>     if(array[loop]==NULL)
>       break;
>    }
>
>   for(loop=0;loop<50;loop++)
>    {
>     if(array[loop]==NULL)
>       break;
>     printf("Item #%d is %s\n",loop,array[loop]);   //fazer um loop for
> (;loop;) Sei lá.
>    }
>    system("pause");
>   return; //tava return 0
> }
> //fazer array[tok] = silaba A cada loop incrementa a silaba
>
>
> //IMPORTANTE!!!!!!! TRANSFORMAR ISSO EM FUNÇÃO!!!!
>
>
>
>
> On Thu, Mar 5, 2009 at 7:45 PM, Derek Clarke <de...@ci...> wrote:
>>
>> Can you tell us what's inside the tokenizar function?
>>
>> Also the code you have published doesn't match the results you report
>> - you are obviously printing the words in a loop, so where's the loop
>> in main?
>>
>> On Thu, Mar 5, 2009 at 11:36 AM, Alexsandro Meireles
>> <mei...@gm...> wrote:
>> > Dear all,
>> >
>> > I have build a function to tokenize a sentence and to return the words
>> > in my
>> > main function. It seems ok, but it's not working the way I intend.
>> >
>> > The code is:
>> >
>> > main()
>> > {
>> >   char str[50];
>> >   char *array[50];
>> >   printf("Token 0.1\nDeveloped by Alexsandro Meireles\nType");
>> >   printf(" the sentence: ");
>> >   gets(str);
>> >   tokenizar(str);
>> >   printf("\n");
>> >   printf("Word #%d is %s\n",0, array[0]);   //PROBLEM IS HERE!!!!!
>> >   system("pause");
>> > }
>> >
>> >
>> > I need to have access to the words returned by the function tokenizar,
>> > but
>> > what I get as result, for example for "a casa pegou fogo." is:
>> >
>> > Word #0 is 'a'
>> > Word #1 is 'casa'
>> > Word #2 is 'pegou'
>> > Word #3 is 'fogo'
>> >
>> > AND THEN A LOOP
>> >
>> > Word #0 is Word #0 is Word #0 is Word #0 is ...
>> >
>> > IN OTHER WORDS, I CAN'T HAVE ACCESS TO THE ARRAY OF WORDS CREATED INSIDE
>> > 'tokenizar'.
>> >
>> > What am I doing wrong?
>> >
>> > Thanks in advance!
>> >
>> > --
>> > Prof. Dr. Alexsandro Meireles, linguist
>> > Federal University of Espírito Santo
>> > Departamento de Línguas e Letras
>> > Av. Fernando Ferrari, 514. Campus Universitário
>> > Goiabeiras. 29075-910
>> > Vitória-ES. Brazil
>> > mei...@gm...
>> > +55-27-41021734
>> >
>> > ------------------------------------------------------------------------------
>> > Open Source Business Conference (OSBC), March 24-25, 2009, San
>> > Francisco, CA
>> > -OSBC tackles the biggest issue in open source: Open Sourcing the
>> > Enterprise
>> > -Strategies to boost innovation and cut costs with open source
>> > participation
>> > -Receive a $600 discount off the registration fee with the source code:
>> > SFAD
>> > http://p.sf.net/sfu/XcvMzF8H
>> > _______________________________________________
>> > Dev-cpp-users mailing list
>> > Dev...@li...
>> > TO UNSUBSCRIBE: http://www23.brinkster.com/noicys/devcpp/ub.htm
>> > https://lists.sourceforge.net/lists/listinfo/dev-cpp-users
>> >
>> >
>
>
>
> --
> Prof. Dr. Alexsandro Meireles, linguist
> Federal University of Espírito Santo
> Departamento de Línguas e Letras
> Av. Fernando Ferrari, 514. Campus Universitário
> Goiabeiras. 29075-910
> Vitória-ES. Brazil
> mei...@gm...
> +55-27-41021734
>

Re: [Dev-C++] Tokenize function

[Per W. <pw...@ia...>]

1) array is a private variable inside main(). If you want this array of
poinster to be initialized by a function call to tokenizar, then you must
send the array as a parameter, together with the information that it can
hold at most 50 words.

And you must also let the function tokenizar return the number of words it
found, so you know how many words to print.

But another thing. Do not ever (!) use gets() to read a string. Always use
fgets(). The reason is that gets() does not take the maximum size of the
input buffer as parameter, so if the input sentence is larger than 49
characters, you will get a buffer overrun.

If you use fgets(), it will end the input when the buffer is full. You can
then check if there is a '\n' (newline character) in the string, or if the
input string was too long.

/pwm

On Thu, 5 Mar 2009, Alexsandro Meireles wrote:

> Dear all,
>
> I have build a function to tokenize a sentence and to return the words in my
> main function. It seems ok, but it's not working the way I intend.
>
> The code is:
>
> main()
> {
>   char str[50];
>   char *array[50];
>   printf("Token 0.1\nDeveloped by Alexsandro Meireles\nType");
>   printf(" the sentence: ");
>   gets(str);
>   tokenizar(str);
>   printf("\n");
>   printf("Word #%d is %s\n",0, array[0]);   //PROBLEM IS HERE!!!!!
>   system("pause");
> }
>
>
> I need to have access to the words returned by the function tokenizar, but
> what I get as result, for example for "a casa pegou fogo." is:
>
> Word #0 is 'a'
> Word #1 is 'casa'
> Word #2 is 'pegou'
> Word #3 is 'fogo'
>
> AND THEN A LOOP
>
> Word #0 is Word #0 is Word #0 is Word #0 is ...
>
> IN OTHER WORDS, I CAN'T HAVE ACCESS TO THE ARRAY OF WORDS CREATED INSIDE
> 'tokenizar'.
>
> What am I doing wrong?
>
> Thanks in advance!
>
> --
> Prof. Dr. Alexsandro Meireles, linguist
> Federal University of Espírito Santo
> Departamento de Línguas e Letras
> Av. Fernando Ferrari, 514. Campus Universitário
> Goiabeiras. 29075-910
> Vitória-ES. Brazil
> mei...@gm...
> +55-27-41021734
>

Re: [Dev-C++] Tokenize function

[Alexsandro M. <mei...@gm...>]

I corrected what you guys suggested to me. fgets intead of gets, input array
as parameter, and so on. Thank you very much for your comments. Now the
program is working a way better.

On Fri, Mar 6, 2009 at 6:42 AM, Per Westermark <pw...@ia...> wrote:

> 1) array is a private variable inside main(). If you want this array of
> poinster to be initialized by a function call to tokenizar, then you must
> send the array as a parameter, together with the information that it can
> hold at most 50 words.
>
> And you must also let the function tokenizar return the number of words it
> found, so you know how many words to print.
>
> But another thing. Do not ever (!) use gets() to read a string. Always use
> fgets(). The reason is that gets() does not take the maximum size of the
> input buffer as parameter, so if the input sentence is larger than 49
> characters, you will get a buffer overrun.
>
> If you use fgets(), it will end the input when the buffer is full. You can
> then check if there is a '\n' (newline character) in the string, or if the
> input string was too long.
>
> /pwm
>
> On Thu, 5 Mar 2009, Alexsandro Meireles wrote:
>
> > Dear all,
> >
> > I have build a function to tokenize a sentence and to return the words in
> my
> > main function. It seems ok, but it's not working the way I intend.
> >
> > The code is:
> >
> > main()
> > {
> >   char str[50];
> >   char *array[50];
> >   printf("Token 0.1\nDeveloped by Alexsandro Meireles\nType");
> >   printf(" the sentence: ");
> >   gets(str);
> >   tokenizar(str);
> >   printf("\n");
> >   printf("Word #%d is %s\n",0, array[0]);   //PROBLEM IS HERE!!!!!
> >   system("pause");
> > }
> >
> >
> > I need to have access to the words returned by the function tokenizar,
> but
> > what I get as result, for example for "a casa pegou fogo." is:
> >
> > Word #0 is 'a'
> > Word #1 is 'casa'
> > Word #2 is 'pegou'
> > Word #3 is 'fogo'
> >
> > AND THEN A LOOP
> >
> > Word #0 is Word #0 is Word #0 is Word #0 is ...
> >
> > IN OTHER WORDS, I CAN'T HAVE ACCESS TO THE ARRAY OF WORDS CREATED INSIDE
> > 'tokenizar'.
> >
> > What am I doing wrong?
> >
> > Thanks in advance!
> >
> > --
> > Prof. Dr. Alexsandro Meireles, linguist
> > Federal University of Espírito Santo
> > Departamento de Línguas e Letras
> > Av. Fernando Ferrari, 514. Campus Universitário
> > Goiabeiras. 29075-910
> > Vitória-ES. Brazil
> > mei...@gm...
> > +55-27-41021734
> >
>
>


-- 
Prof. Dr. Alexsandro Meireles, linguist
Federal University of Espírito Santo
Departamento de Línguas e Letras
Av. Fernando Ferrari, 514. Campus Universitário
Goiabeiras. 29075-910
Vitória-ES. Brazil
mei...@gm...
+55-27-41021734