Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

MemoryError Message performing sync

dsbcpas
2008-07-14
2013-02-19
  • dsbcpas
    dsbcpas
    2008-07-14

    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
        An unexpected error has occurred.
      Please report the following lines to:
      s3tools-general@lists.sourceforge.net
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    S3cmd:  0.9.8.2
    Python: 2.5.1 (r251:54863, Feb 10 2008, 15:36:48)  [GCC 3.4.6 20060404 (Red Hat 3.4.6-9)]

    Traceback (most recent call last):
      File "/usr/bin/s3cmd", line 1041, in <module>
        main()
      File "/usr/bin/s3cmd", line 1021, in main
        cmd_func(args)
      File "/usr/bin/s3cmd", line 628, in cmd_sync
        return cmd_sync_local2remote(src, dst)
      File "/usr/bin/s3cmd", line 571, in cmd_sync_local2remote
        _compare_filelists(loc_list, rem_list, True)
      File "/usr/bin/s3cmd", line 390, in _compare_filelists
        src_md5 = Utils.hash_file_md5(src_list[file]['full_name'])
      File "/usr/bin/S3/Utils.py", line 142, in hash_file_md5
        h.update(f.read())
    MemoryError

    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
        An unexpected error has occurred.
        Please report the above lines to:
      s3tools-general@lists.sourceforge.net
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

     
    • Michal Ludvig
      Michal Ludvig
      2008-07-14

      Looks like you're syncing a very big file? Or perhaps huge number of files?

       
    • dsbcpas
      dsbcpas
      2008-07-14

      Yes, it was a 1.3 gig movie. It very well could have hit the 5 gig prefix limit problem.

       
      • Michal Ludvig
        Michal Ludvig
        2008-07-14

        It's more likely it ran out of memory on your machine.

        The statement "h.update(f.read())" tries to read the complete input file into a memory and then compute a checksum. That could easily run OOM with big files. I'll rework that part to compute the checksum in a loop from small blocks. Not today though, I go to sleep now ;-)