Test case failure.

Help
Zhai, Ke
2012-11-29
2012-12-07
  • Zhai, Ke
    Zhai, Ke
    2012-11-29

    Hi, I am new to python and hadoop streaming. I am trying to play with pydoop-0.7.0. I think I setup pydoop successfully on a public cluster. I have access to a Hadoop cluster. I don't know whether these errors will cause any problems.

    ======================================================================
    ERROR: connect (test_hdfs_fs.TestConnection)


    Traceback (most recent call last):
      File "/fs/clip-lsbi/Workspace/pydoop-0.7.0-rc1/test/test_hdfs_fs.py", line 48, in connect
        fs = hdfs.hdfs(host, port, user=user)
      File "/cliphomes/zhaike/.local/lib/python2.7/site-packages/pydoop/hdfs/fs.py", line 124, in __init__
        h, p, u, fs = _get_connection_info(host, port, user)
      File "/cliphomes/zhaike/.local/lib/python2.7/site-packages/pydoop/hdfs/fs.py", line 60, in _get_connection_info
        fs = hdfs_ext.hdfs_fs(host, port, user)
    IOError: Cannot connect to localhost

    ======================================================================
    ERROR: cache (test_hdfs_fs.TestConnection)


    Traceback (most recent call last):
      File "/fs/clip-lsbi/Workspace/pydoop-0.7.0-rc1/test/test_hdfs_fs.py", line 55, in cache
        fs = hdfs.hdfs(host, port)
      File "/cliphomes/zhaike/.local/lib/python2.7/site-packages/pydoop/hdfs/fs.py", line 124, in __init__
        h, p, u, fs = _get_connection_info(host, port, user)
      File "/cliphomes/zhaike/.local/lib/python2.7/site-packages/pydoop/hdfs/fs.py", line 60, in _get_connection_info
        fs = hdfs_ext.hdfs_fs(host, port, user)
    IOError: Cannot connect to localhost

    ======================================================================
    ERROR: chown (test_hdfs_fs.TestHDFS)


    Traceback (most recent call last):
      File "/fs/clip-lsbi/Workspace/pydoop-0.7.0-rc1/test/test_hdfs_fs.py", line 86, in chown
        self.fs.chown(path, user=new_owner)
      File "/cliphomes/zhaike/.local/lib/python2.7/site-packages/pydoop/hdfs/fs.py", line 440, in chown
        return self.fs.chown(path, user, group)
    IOError: Cannot chown hdfs://bespin00.umiacs.umd.edu/user/zhaike/pydoop_test_7e8cf51d3ae243e6b7e84d9d51ec409a/8406603da2104bec9544c03b6d135fdb to nobody,


    Ran 130 tests in 34.443s

    FAILED (errors=3)

     
  • Simone Leo
    Simone Leo
    2012-11-30

    Hello, and welcome to Pydoop!

    The chown error is normal if you don't have superuser privileges - see the following section of the docs for details:

    http://pydoop.sourceforge.net/docs/installation.html#testing-your-installation

    The other two errors can show up when the test module tries to connect to HDFS with explicitly set host and port. Again, this is explained in greater detail at the above URL.

    Simone

     
  • Hi,

    I was going to open a new thread, but as kzhai opened this one, my question fits here. I am also new into pydoop. I am installing it in our hadoop cluster as a requirement of the short read alignment and analysis toolkit called Seal. I've read the notes on the section you linked before, and everything is fine in that sense. Still, I'm failing the cache test:

    ======================================================================
    FAIL: cache (test_hdfs_fs.TestConnection)


    Traceback (most recent call last):
      File "/home/vagrant/pydoop/test/test_hdfs_fs.py", line 56, in cache
        self.assertTrue(fs.fs is orig_fs.fs)
    AssertionError: False is not true


    Ran 132 tests in 18.228s

    FAILED (failures=1)

    Do you have any idea of why it is failing? I don't know exactly what this test does. Here a bit more information about my cluster:

        - It is a virtualized cluster with 1 namenode and 3 datanodes.
        - All the virtual machines are running Ubuntu 12.04 64 bits, 512MB of RAM assigned.
        - I'm using CDH4 cloudera distribution
        - Replication factor of 2.

    If I can provide some more information, please tell me.

    Thank you very much for your help!

     
  • Zhai, Ke
    Zhai, Ke
    2012-11-30

    Thanks very much.

     
  • Simone Leo
    Simone Leo
    2012-12-03

    Hello guillemcat,

    the test checks that HDFS instance caching is working correctly. It's a feature we introduced to avoid possible JVM crashes when multiple handles to the same HDFS connection are opened. Basically, the hdfs class stores HDFS connection objects after resolving aliases for host (e.g., "default" vs real host name), port and user, and reuses them when you try to create a new HDFS instance that references an existing open connection. If you're interested in the details, see the source code for the hdfs.fs module:

    http://pydoop.git.sourceforge.net/git/gitweb.cgi?p=pydoop/pydoop;a=blob_plain;f=pydoop/hdfs/fs.py;hb=9876d9fef9d28504830ed185b0a1ff576f636cea

    More specifically, the test suite creates an instance for the default (i.e., the one specified in the Hadoop configuration files) HDFS, using ("default", 0) for the host name and port number. The "cache" test creates more instances using (<host name>, <port number>) pairs that should all refer to the same (default) instance. One of these attempts uses explicitly set host and port that, in a CDH environment, are set to "localhost" and 8020. Since users can point to an arbitrary HDFS when running the test, the above case is more prone to failure.

    In conclusion:

    1. check that you have installed the latest Pydoop release
    2. if you are not using the default (host, port) setup, set the HDFS_HOST and HDFS_PORT environment variables accordingly before running the test

    To see what's happening (and check whether the case that's failing is actually the one I expect), you can add some dump code to the test:

    def cache(self):
      orig_fs = hdfs.hdfs(*self.hp_cases[0])
      print
      print "ORIG. HOST: %r, ORIG. PORT: %r" % self.hp_cases[0]
      for host, port in self.hp_cases[1:]:
        print "HOST: %r, PORT: %r" % (host, port)
        fs = hdfs.hdfs(host, port)
        self.assertTrue(fs.fs is orig_fs.fs)
        fs.close()
        self.assertFalse(orig_fs.closed)
      orig_fs.close()
      self.assertTrue(orig_fs.closed)
    

    Here is the output on my machine:

    cache (__main__.TestConnection) ... 
    ORIG. HOST: 'default', ORIG. PORT: 0
    HOST: 'localhost', PORT: 8020
    HOST: '127.0.0.1', PORT: 8020
    ok
    

    Hope this helps,

    Simone

     
  • Hi Simone,

    Thank you for your reply, it has been very helpful. I've realized that what is happening is that I'm having problems with my DNS system. The connections to "localhost" are being refused, so If I type:

    telnet localhost 8020
    

    It is refused…

    Thanks again, I'll try to solve this.

    Best,

    Guillermo