Menu
▾
▴
[Myhadoop-users] Fwd: myHadoop how to start pbs-example.sh
|
From: Sriram K. <sri...@gm...> - 2011-03-25 00:48:58
|
---------- Forwarded message ---------- From: <so...@us...> Date: Thu, Mar 24, 2011 at 2:33 PM Subject: myHadoop how to start pbs-example.sh To: sri...@us... hi! i have downloaded myHadoop to use it in a grid cluster using torque... we have a node where we log in and submit jobs with qsub (submit node --> from where i will run pbs-example.sh)... qsub then allocates other nodes from the cluster (the worker nodes..the initial submit node not included)... i have the following questions: 1) i must copy the hadoop-0.20.2 on all worker nodes right? 2) mHadoop-0.2a must be on the submit node only or on the sudmit node and on all of the worker nodes?? 3) how to i start a job?? i run pbs-example.sh or qsub pbs-example.sh from the submit node? if i run qsub pbs-example it will run stat-all.sh script for hadoop on all nodes which is not correct.. it must be done only in the master node (or at least i think so)... also the master node will not be on the submit node where i run pbs-example.sh.... maybe i do something really silly... here is the output i get (not all cause i kill the job with ctrl-c): out: Resources : cput=196:00:00 neednodes=wn001.grid.tuc.gr+wn002.grid.tuc.gr+wn003.grid.tuc.gr+ wn004.grid.tuc.gr nodes=4:ppn=1 walltime=218:00:00 Walltime : 218:00:00 Node_list : wn001.grid.tuc.gr+wn002.grid.tuc.gr+wn003.grid.tuc.gr+wn004.grid.tuc.gr ,nodes=4:ppn=1,walltime=218:00:00 Select Result: DBI::st=HASH(0x1865bbe0) Set up the configurations for myHadoop Number of Hadoop nodes requested: 4 Generation Hadoop configuration in directory: /storage/tuclocal/asouris/configuration_directory Not persisting HDFS state Received 4 nodes from PBS Master is: wn001.grid.tuc.gr Configuring node: wn001.grid.tuc.gr rm -rf /tmp/hadoop-test-dir/log-dir; mkdir -p /tmp/hadoop-test-dir/log-dir rm -rf /tmp/hadoop-test-dir/data-dir; mkdir -p /tmp/hadoop-test-dir/data-dir Configuring node: wn002.grid.tuc.gr rm -rf /tmp/hadoop-test-dir/log-dir; mkdir -p /tmp/hadoop-test-dir/log-dir rm -rf /tmp/hadoop-test-dir/data-dir; mkdir -p /tmp/hadoop-test-dir/data-dir Configuring node: wn003.grid.tuc.gr rm -rf /tmp/hadoop-test-dir/log-dir; mkdir -p /tmp/hadoop-test-dir/log-dir rm -rf /tmp/hadoop-test-dir/data-dir; mkdir -p /tmp/hadoop-test-dir/data-dir Configuring node: wn004.grid.tuc.gr rm -rf /tmp/hadoop-test-dir/log-dir; mkdir -p /tmp/hadoop-test-dir/log-dir rm -rf /tmp/hadoop-test-dir/data-dir; mkdir -p /tmp/hadoop-test-dir/data-dir Format HDFS Start all Hadoop daemons starting namenode, logging to /tmp/hadoop-test-dir/log-dir/hadoop-asouris-namenode-wn001.grid.tuc.gr.out wn002.grid.tuc.gr: Permission denied, please try again. wn002.grid.tuc.gr: Permission denied, please try again. wn002.grid.tuc.gr: Permission denied (publickey,gssapi-with-mic,password). wn004.grid.tuc.gr: Permission denied, please try again. wn004.grid.tuc.gr: Permission denied, please try again. wn004.grid.tuc.gr: Permission denied (publickey,gssapi-with-mic,password). wn003.grid.tuc.gr: Permission denied, please try again. wn003.grid.tuc.gr: Permission denied, please try again. wn003.grid.tuc.gr: Permission denied (publickey,gssapi-with-mic,password). wn001.grid.tuc.gr: Permission denied, please try again. wn001.grid.tuc.gr: Permission denied, please try again. wn001.grid.tuc.gr: Permission denied (publickey,gssapi-with-mic,password). wn001.grid.tuc.gr: Permission denied, please try again. wn001.grid.tuc.gr: Permission denied, please try again. wn001.grid.tuc.gr: Permission denied (publickey,gssapi-with-mic,password). starting jobtracker, logging to /tmp/hadoop-test-dir/log-dir/hadoop-asouris-jobtracker-wn001.grid.tuc.gr.out wn001.grid.tuc.gr: Permission denied, please try again. wn001.grid.tuc.gr: Permission denied, please try again. wn001.grid.tuc.gr: Permission denied (publickey,gssapi-with-mic,password). wn003.grid.tuc.gr: Permission denied, please try again. wn003.grid.tuc.gr: Permission denied, please try again. wn003.grid.tuc.gr: Permission denied (publickey,gssapi-with-mic,password). wn002.grid.tuc.gr: Permission denied, please try again. wn002.grid.tuc.gr: Permission denied, please try again. wn002.grid.tuc.gr: Permission denied (publickey,gssapi-with-mic,password). wn004.grid.tuc.gr: Permission denied, please try again. wn004.grid.tuc.gr: Permission denied, please try again. wn004.grid.tuc.gr: Permission denied (publickey,gssapi-with-mic,password). Run some test Hadoop jobs Job ID: 61257.se01.grid.tuc.gr User ID: asouris Group ID: vsam Job Name: myHadoop Session ID: 13994 Resource List: cput=196:00:00,neednodes=4:ppn=1,nodes=4:ppn=1,walltime=218:00:00 Resources Used: cput=00:00:08,mem=154032kb,vmem=4341576kb,walltime=00:13:23 Queue Name: tuc Account String: (END) err: DBD::mysqlPP::st execute failed: #08S01Bad handshake at /storage/exp_soft/tuc/mypbs/sbin/mypbs_pr line 777. Can't call method "each" on an undefined value at /usr/lib/perl5/site_perl/5.8.8/DBD/mysqlPP.pm line 392. Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-with-mic,password). Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-with-mic,password). Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-with-mic,password). Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-with-mic,password). Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-with-mic,password). Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-with-mic,password). Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-with-mic,password). Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-with-mic,password). 11/03/24 23:15:08 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = wn001.grid.tuc.gr/147.27.48.101 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 0.20.2 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010 ************************************************************/ Re-format filesystem in /tmp/hadoop-test-dir/data-dir/dfs/name ? (Y or N) Format aborted in /tmp/hadoop-test-dir/data-dir/dfs/name 11/03/24 23:15:09 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at wn001.grid.tuc.gr/147.27.48.101 ************************************************************/ mkdir: cannot create directory Data: File exists copyFromLocal: /tmp/.java_pid20214 (Permission denied) -- This message has been sent to you, a registered SourceForge.net user, by another site user, through the SourceForge.net site. This message has been delivered to your SourceForge.net mail alias. You may reply to this message using the "Reply" feature of your email client, or using the messaging facility of SourceForge.net at: https://sourceforge.net/sendmessage.php?touser=3307261 |
