Thread: [Gauche-devel] string-split vs. string-tokenize

The Gauche's reference manual states as follows:

	Note: When splitter is a char-set, string-split is equivalent
	to string-tokenize except the meaning of the char-set is
	reversed.

It is not true in some boundary cases.

(string-split "" (char-set-complement char-set:graphic))
=> ("")
(string-tokenize "")
=> ()

(string-split " a " (char-set-complement char-set:graphic))
=> ("" "a" "")
(string-tokenize "")
=> ("a")

This is the same behavior difference between python's split() and
perl/ruby's.

Needs another vote? I prefer the string-tokenize's behavior, just
because it fits in the code I'm writing now. ;)

-- fuyuki

From: Kimura Fuyuki <fu...@ni...>
Subject: [Gauche-devel] string-split vs. string-tokenize
Date: Mon, 06 Oct 2003 12:25:53 +0900

> It is not true in some boundary cases.
> 
> (string-split "" (char-set-complement char-set:graphic))
> => ("")
> (string-tokenize "")
> => ()
> 
> (string-split " a " (char-set-complement char-set:graphic))
> => ("" "a" "")
> (string-tokenize "")
> => ("a")

I see.  I withdraw the description in the manual.

Conceptually, string-split returns what's remaining in
the string, after removing delimiter sequences.  
So I feel (string-split "" char-set:full) should return "",
since nothing is taken from the given string, "".

On the other hand, string-tokenize extracts tokens that
match given charset, so (string-tokenize "") => () is
natural, since no token is matched in the string "".

--shiro