[moses-smt/mosesdecoder] 38e145: Use util::TokenIter to tokenize n-best lists.

[GitHub <no...@gi...>]

  Branch: refs/heads/master
  Home:   https://github.com/moses-smt/mosesdecoder
  Commit: 38e145e556274a199e270cc9ede833491b0f1a86
      https://github.com/moses-smt/mosesdecoder/commit/38e145e556274a199e270cc9ede833491b0f1a86
  Author: Tetsuo Kiso <tet...@is...>
  Date:   2012-12-06 (Thu, 06 Dec 2012)

  Changed paths:
    M mert/Data.cpp
    M mert/Data.h
    M mert/DataTest.cpp
    M mert/FeatureArray.cpp
    M mert/FeatureArray.h
    M mert/FeatureData.cpp
    M mert/FeatureData.h
    M mert/InterpolatedScorer.cpp
    M mert/ScoreArray.cpp
    M mert/ScoreArray.h
    M mert/ScoreData.cpp
    M mert/ScoreData.h
    M mert/Types.h
    M mert/evaluator.cpp

  Log Message:
  -----------
  Use util::TokenIter to tokenize n-best lists.

Reduce creating std::string objects, too. In both ScoreArray
and FeatureArray classes, the private members to track sentence
indices (namely, "m_index") were unnecessarily declared as
std::string, but it's better to directly declare them as 'int'.