Many evaluation metrics over several RankLib LTR models shows either very good or very bad results that are seemingly affected by shuffling the input data. This is happening with a significantly large enough training data although feature counts are small.
The suspicion is that the problem may be related to scoring class function doing sorting before scoring.
See https://sourceforge.net/p/lemur/discussion/ranklib/thread/15d0a4e8/?limit=25#0b0d for more details on the problem.
I don't believe input data ordering should effect evaluation metric results.