EvalAI

EvalAI is an open-source platform for evaluating and comparing machine learning (ML) and artificial intelligence (AI) algorithms at scale. We allow the creation of an arbitrary number of evaluation phases and dataset splits, compatibility using any programming language, and organizing results in both public and private leaderboards. Certain large-scale challenges need special computing capabilities for evaluation. If the challenge needs extra computational power, challenge organizers can easily add their own cluster of worker nodes to process participant submissions while we take care of hosting the challenge, handling user submissions, and maintaining the leaderboard. EvalAI lets participants submit code for their agent in the form of docker images which are evaluated against test environments on the evaluation server. During the evaluation, the worker fetches the image, test environment, and model snapshot and spins up a new container to perform the evaluation.