"Memory exhausted" ! ! is this a bug ?

  • Atidel

    Hi all,

    I try to create a matrix (235 x 1200000) but I get the following error message : "Matrix -- Memory exhausted". I can create matrices until the size (100 x 1200000) only. I have 2Go of RAM. On another machine (which is not mine) the matrix is created normally although the RAM is only 1 Go.

    What do you think the problem is due to ?

    Thanks ---- Atidel

    • Yann LeCun
      Yann LeCun

      It's not a bug, and it has nothing to do with Lush:
      On a 32-bit machine, you can't have arrays larger than 1GB or 2GB (depending
      on how Linux was compiled). If you need bigger arrays, get a 64 bit machine
      and install a 64-bit version of Linux.

      -- Yann

    • Atidel

      Thank you Yann, but I am not very convinced by the answer.

      Does the machine architecture (in particular the processors bits numbers) constraint the choice of dataset size that I choose for NN training ? knowing that NNs have been effectively used many decades ago when computers were not as performant as nowdays.

      -- Atidel

      • cmejia

        Is not the processor bits number but the capacity of an integer to address memory. Pointers to memory are interger or long type data, both 32 bits wide in a 32 bits linux. Thus you can address a maximum of 2^(32-1) memory words (normally bytes). That is 2147483648 bytes or 2Gb ... this limits several things in a system: theoretical program memory size, array size, file size, etc. and in older linux, partition sizes. With a linux 64 bits the limit is 2^(64 -1), a very very big number. Try it !