[moses-smt/mosesdecoder] 38e145: Use util::TokenIter to tokenize n-best lists.

[GitHub <no...@gi...>]

  Branch: refs/heads/weight-new
  Home:   https://github.com/moses-smt/mosesdecoder
  Commit: 38e145e556274a199e270cc9ede833491b0f1a86
      https://github.com/moses-smt/mosesdecoder/commit/38e145e556274a199e270cc9ede833491b0f1a86
  Author: Tetsuo Kiso <tet...@is...>
  Date:   2012-12-06 (Thu, 06 Dec 2012)

  Changed paths:
    M mert/Data.cpp
    M mert/Data.h
    M mert/DataTest.cpp
    M mert/FeatureArray.cpp
    M mert/FeatureArray.h
    M mert/FeatureData.cpp
    M mert/FeatureData.h
    M mert/InterpolatedScorer.cpp
    M mert/ScoreArray.cpp
    M mert/ScoreArray.h
    M mert/ScoreData.cpp
    M mert/ScoreData.h
    M mert/Types.h
    M mert/evaluator.cpp

  Log Message:
  -----------
  Use util::TokenIter to tokenize n-best lists.

Reduce creating std::string objects, too. In both ScoreArray
and FeatureArray classes, the private members to track sentence
indices (namely, "m_index") were unnecessarily declared as
std::string, but it's better to directly declare them as 'int'.


  Commit: c7f6e3832615e92bc1321c2fb868064271c68d7e
      https://github.com/moses-smt/mosesdecoder/commit/c7f6e3832615e92bc1321c2fb868064271c68d7e
  Author: Tetsuo Kiso <tet...@is...>
  Date:   2012-12-06 (Thu, 06 Dec 2012)

  Changed paths:
    M mert/Data.cpp

  Log Message:
  -----------
  Use FilePiece to load N-best lists.

Since FilePiece is friendly with StringPiece.


  Commit: 6c04c4ad9c1fa9b184a3ad95c8e7b75bcb5c92c3
      https://github.com/moses-smt/mosesdecoder/commit/6c04c4ad9c1fa9b184a3ad95c8e7b75bcb5c92c3
  Author: Tetsuo Kiso <tet...@is...>
  Date:   2012-12-06 (Thu, 06 Dec 2012)

  Changed paths:
    M mert/DataTest.cpp

  Log Message:
  -----------
  Add more tests to the Data class.


  Commit: 8fdec9bf3059e388b267da06b4d5a0ca67615df7
      https://github.com/moses-smt/mosesdecoder/commit/8fdec9bf3059e388b267da06b4d5a0ca67615df7
  Author: Tetsuo Kiso <tet...@is...>
  Date:   2012-12-06 (Thu, 06 Dec 2012)

  Changed paths:
    M mert/BleuScorer.cpp
    M mert/BleuScorer.h
    M mert/Scorer.cpp
    M mert/Scorer.h
    M mert/Vocabulary.cpp
    M mert/Vocabulary.h

  Log Message:
  -----------
  Use boost::unordered_map instead of std::map.

For storing the word vocabulary used in computation of
BLEU scores. This change will reduce the running time
of extractor about 2-3 seconds (9% reduction).


  Commit: 5a783f166e2bf40bd0f7b2ac4ee669e1e037453e
      https://github.com/moses-smt/mosesdecoder/commit/5a783f166e2bf40bd0f7b2ac4ee669e1e037453e
  Author: Hieu Hoang <fis...@gm...>
  Date:   2012-12-06 (Thu, 06 Dec 2012)

  Changed paths:
    M mert/BleuScorer.cpp
    M mert/BleuScorer.h
    M mert/Data.cpp
    M mert/Data.h
    M mert/DataTest.cpp
    M mert/FeatureArray.cpp
    M mert/FeatureArray.h
    M mert/FeatureData.cpp
    M mert/FeatureData.h
    M mert/InterpolatedScorer.cpp
    M mert/ScoreArray.cpp
    M mert/ScoreArray.h
    M mert/ScoreData.cpp
    M mert/ScoreData.h
    M mert/Scorer.cpp
    M mert/Scorer.h
    M mert/Types.h
    M mert/Vocabulary.cpp
    M mert/Vocabulary.h
    M mert/evaluator.cpp

  Log Message:
  -----------
  Merge branch 'master' into weight-new


Compare: https://github.com/moses-smt/mosesdecoder/compare/3d6d53bf4905...5a783f166e2b