#294 RankLib LambdaMART Fails With Very Large Data

[Lemur Project]

Possible bug in RankLib LambdaMART algorithm implementation.

LambdaMART generates OutOfMemoryError exceptions for some processing threads when run large training data, in particular, one query of data set containing 500,000 documents.

Some debugging checks involved the following:
o Plenty of RAM on running host -- 768G: asking for 500G via Java -Xmx argument
o No system memory limits for processes
o Adding Java -Xms value similar to -Xmx still failed
o Tried LamdaMART using NDCG instead of ERR still failed.
o No problem using different algorithm (MART with ERR or NDCG)
o No problem using reduced dataset with ERR or NDCG

So problem appears to be with LambdaMART implementation given problem shows up only with high number of document (almost 500k for a specific query) dataset.

Note: it is possible this is not actually a bug, but an implementation error where algorithm speed was optimized over space usage. The algorithm simply might not be scalable.

[Diego]

The bug manifest itself also with dataset containing 68491 documents for one query, but it's not showing with 45661 documents.

[Surabhi Amit Chembra]

Thanks Diego, so if the number of documents for each query is less than 45661, then the bug is not triggered even if there are thousands of queries? And is this fixed in new versions of RankLib?

[Diego]

Ehm...I have no memories of the problem other than the mail I've sent.

Il mar 8 set 2020, 20:51 Surabhi Amit Chembra schembra@users.sourceforge.net ha scritto:

Thanks Diego, so if the number of documents for each query is less than
45661, then the bug is not triggered even if there are thousands of queries?

---

• [bugs:#294] https://sourceforge.net/p/lemur/bugs/294/ RankLib
  LambdaMART Fails With Very Large Data*

Status: open
Group: v1.x
Labels: RankLib LambdaMART memory
Created: Thu Jan 26, 2017 03:43 PM UTC by Lemur Project
Last Updated: Mon Jan 30, 2017 09:35 AM UTC
Owner: Lemur Project

Possible bug in RankLib LambdaMART algorithm implementation.

LambdaMART generates OutOfMemoryError exceptions for some processing
threads when run large training data, in particular, one query of data set
containing 500,000 documents.

Some debugging checks involved the following:
o Plenty of RAM on running host -- 768G: asking for 500G via Java -Xmx
argument
o No system memory limits for processes
o Adding Java -Xms value similar to -Xmx still failed
o Tried LamdaMART using NDCG instead of ERR still failed.
o No problem using different algorithm (MART with ERR or NDCG)
o No problem using reduced dataset with ERR or NDCG

So problem appears to be with LambdaMART implementation given problem
shows up only with high number of document (almost 500k for a specific
query) dataset.

Note: it is possible this is not actually a bug, but an implementation
error where algorithm speed was optimized over space usage. The algorithm
simply might not be scalable.

---

Sent from sourceforge.net because you indicated interest in
https://sourceforge.net/p/lemur/bugs/294/

To unsubscribe from further messages, please visit
https://sourceforge.net/auth/subscriptions/

[Surabhi Amit Chembra]

Thanks Diego. Appreciate the response.