Menu
▾
▴
[Smartfrog-checkins] [Smartfrog.org-JIRA] Commented: (SFOS-1211) JobSubmissionTest failing with "Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES"
|
From: Steve L. (JIRA) <ji...@sm...> - 2009-06-01 14:15:05
|
[ http://jira.smartfrog.org/jira/browse/SFOS-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=11689#action_11689 ]
Steve Loughran commented on SFOS-1211:
--------------------------------------
trace
[sf-startdaemon-debug] 09/06/01 14:21:03 [TaskTracker] INFO mapred.TaskTracker : LaunchTaskAction (registerTask): attempt_200906011420_0001_m_000000_0 task's state:UNASSIGNED
[sf-startdaemon-debug] 09/06/01 14:21:03 [TaskLauncher for task] INFO mapred.TaskTracker : Trying to launch : attempt_200906011420_0001_m_000000_0
[sf-startdaemon-debug] 09/06/01 14:21:03 [TaskLauncher for task] INFO mapred.TaskTracker : In TaskLauncher, current free slots : 2 and trying to launch attempt_200906011420_0001_m_000000_0
[sf-startdaemon-debug] 09/06/01 14:21:03 [Thread-169] INFO mapred.JvmManager : In JvmRunner constructed JVM ID: jvm_200906011420_0001_m_-1849792162
[sf-startdaemon-debug] 09/06/01 14:21:03 [Thread-169] INFO mapred.JvmManager : JVM Runner jvm_200906011420_0001_m_-1849792162 spawned.
[sf-startdaemon-debug] 09/06/01 14:21:03 [ExtDataNode] INFO datanode.DataNode : BlockReport of 6 blocks got processed in 7 msecs
[sf-startdaemon-debug] 09/06/01 14:21:04 [IPC Server handler 1 on 34641] INFO mapred.TaskTracker : JVM with ID: jvm_200906011420_0001_m_-1849792162 given task: attempt_200906011420_0001_m_000000_0
[sf-startdaemon-debug] 09/06/01 14:21:05 [IPC Server handler 2 on 34641] INFO mapred.TaskTracker : attempt_200906011420_0001_m_000000_0 0.0%
[sf-startdaemon-debug] 09/06/01 14:21:05 [org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@89c535d] INFO datanode.DataBlockScanner : Verification succeeded for blk_-251036468853122663_1001
[sf-startdaemon-debug] 09/06/01 14:21:05 [IPC Server handler 3 on 34641] INFO mapred.TaskTracker : attempt_200906011420_0001_m_000000_0 0.667% map > sort
[sf-startdaemon-debug] 09/06/01 14:21:05 [IPC Server handler 0 on 34641] INFO mapred.TaskTracker : Task attempt_200906011420_0001_m_000000_0 is done.
[sf-startdaemon-debug] 09/06/01 14:21:05 [IPC Server handler 0 on 34641] INFO mapred.TaskTracker : reported output size for attempt_200906011420_0001_m_000000_0 was 0
[sf-startdaemon-debug] 09/06/01 14:21:05 [Thread-169] INFO mapred.TaskTracker : Error cleaning up task runner: java.lang.NullPointerException
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.cleanup(TaskTracker.java:2561)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.taskFinished(TaskTracker.java:2329)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.reportTaskFinished(TaskTracker.java:2161)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:453)
[sf-startdaemon-debug] 09/06/01 14:21:05 [Thread-169] INFO mapred.TaskTracker : addFreeSlot : current free slots : 2
[sf-startdaemon-debug] 09/06/01 14:21:06 [IPC Server handler 3 on 8012] INFO mapred.JobInProgress : Task 'attempt_200906011420_0001_m_000000_0' has completed task_200906011420_0001_m_000000 successfully.
[sf-startdaemon-debug] 09/06/01 14:21:06 [IPC Server handler 3 on 8012] INFO mapred.ResourceEstimator : completedMapsUpdates:1 completedMapsInputSize:76 completedMapsOutputSize:91
[sf-startdaemon-debug] 09/06/01 14:21:06 [IPC Server handler 3 on 8012] INFO mapred.JobTracker : Adding task 'attempt_200906011420_0001_r_000000_0' to tip task_200906011420_0001_r_000000, for tracker 'tracker_morzine.hpl.hp.com:localhost/127.0.0.1:34641'
[sf-startdaemon-debug] 09/06/01 14:21:06 [JVM Runner jvm_200906011420_0001_m_-1849792162 spawned.] WARN util.ProcessTree : Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process
[sf-startdaemon-debug] 09/06/01 14:21:06 [JVM Runner jvm_200906011420_0001_m_-1849792162 spawned.] INFO util.ProcessTree : Killing all processes in the process group 14298 with SIGTERM. Exit code 1
[sf-startdaemon-debug] 09/06/01 14:21:06 [TaskTracker] INFO mapred.TaskTracker : LaunchTaskAction (registerTask): attempt_200906011420_0001_r_000000_0 task's state:UNASSIGNED
[sf-startdaemon-debug] 09/06/01 14:21:06 [TaskLauncher for task] INFO mapred.TaskTracker : Trying to launch : attempt_200906011420_0001_r_000000_0
[sf-startdaemon-debug] 09/06/01 14:21:06 [TaskLauncher for task] INFO mapred.TaskTracker : In TaskLauncher, current free slots : 2 and trying to launch attempt_200906011420_0001_r_000000_0
[sf-startdaemon-debug] 09/06/01 14:21:06 [Thread-179] INFO mapred.JvmManager : In JvmRunner constructed JVM ID: jvm_200906011420_0001_r_-1307341032
[sf-startdaemon-debug] 09/06/01 14:21:06 [Thread-179] INFO mapred.JvmManager : JVM Runner jvm_200906011420_0001_r_-1307341032 spawned.
[sf-startdaemon-debug] 09/06/01 14:21:07 [IPC Server handler 2 on 34641] INFO mapred.TaskTracker : JVM with ID: jvm_200906011420_0001_r_-1307341032 given task: attempt_200906011420_0001_r_000000_0
[sf-startdaemon-debug] 09/06/01 14:21:07 [JVM Runner jvm_200906011420_0001_m_-1307341032 spawned.] WARN util.ProcessTree : Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process
[sf-startdaemon-debug] 09/06/01 14:21:07 [JVM Runner jvm_200906011420_0001_m_-1307341032 spawned.] INFO util.ProcessTree : Killing process group14250 with SIGKILL. Exit code 1
[sf-startdaemon-debug] 09/06/01 14:21:07 [JVM Runner jvm_200906011420_0001_m_-1307341032 spawned.] INFO mapred.DefaultTaskController : Process exited with exit code:0
[sf-startdaemon-debug] 09/06/01 14:21:07 [JVM Runner jvm_200906011420_0001_m_-1307341032 spawned.] INFO mapred.JvmManager : JVM : jvm_200906011420_0001_m_-1307341032 exited. Number of tasks it ran: 1
[sf-startdaemon-debug] 09/06/01 14:21:11 [JVM Runner jvm_200906011420_0001_m_-1849792162 spawned.] WARN util.ProcessTree : Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process
[sf-startdaemon-debug] 09/06/01 14:21:11 [JVM Runner jvm_200906011420_0001_m_-1849792162 spawned.] INFO util.ProcessTree : Killing process group14298 with SIGKILL. Exit code 1
[sf-startdaemon-debug] 09/06/01 14:21:11 [JVM Runner jvm_200906011420_0001_m_-1849792162 spawned.] INFO mapred.DefaultTaskController : Process exited with exit code:0
[sf-startdaemon-debug] 09/06/01 14:21:11 [JVM Runner jvm_200906011420_0001_m_-1849792162 spawned.] INFO mapred.JvmManager : JVM : jvm_200906011420_0001_m_-1849792162 exited. Number of tasks it ran: 1
[sf-startdaemon-debug] 09/06/01 14:21:13 [IPC Server handler 2 on 34641] INFO mapred.TaskTracker : attempt_200906011420_0001_r_000000_0 0.0% reduce > copy >
[sf-startdaemon-debug] 09/06/01 14:21:13 [ExtDataNode] INFO datanode.DataNode : BlockReport of 6 blocks got processed in 1 msecs
[sf-startdaemon-debug] 2009/06/01 14:21:14:196 BST [INFO ][LivenessSender_HOST morzine.hpl.hp.com:rootProcess:testJobSubmission] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - Task Id : attempt_200906011420_0001_m_000002_0, Status : SUCCEEDED
[sf-startdaemon-debug] 2009/06/01 14:21:14:197 BST [INFO ][LivenessSender_HOST morzine.hpl.hp.com:rootProcess:testJobSubmission] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - Task Id : attempt_200906011420_0001_m_000000_0, Status : SUCCEEDED
[sf-startdaemon-debug] 2009/06/01 14:21:14:198 BST [INFO ][LivenessSender_HOST morzine.hpl.hp.com:rootProcess:testJobSubmission] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - Task Id : attempt_200906011420_0001_m_000002_0, Status : SUCCEEDED
[sf-startdaemon-debug] 2009/06/01 14:21:14:198 BST [INFO ][LivenessSender_HOST morzine.hpl.hp.com:rootProcess:testJobSubmission] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - Task Id : attempt_200906011420_0001_m_000000_0, Status : SUCCEEDED
[sf-startdaemon-debug] 09/06/01 14:21:16 [IPC Server handler 2 on 34641] INFO mapred.TaskTracker : attempt_200906011420_0001_r_000000_0 0.0% reduce > copy >
[sf-startdaemon-debug] 09/06/01 14:21:17 [IPC Server handler 1 on 34641] FATAL mapred.TaskTracker : Task: attempt_200906011420_0001_r_000000_0 - Killed due to Shuffle Failure: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
[sf-startdaemon-debug] 09/06/01 14:21:17 [IPC Server handler 1 on 34641] INFO mapred.TaskTracker : About to purge task: attempt_200906011420_0001_r_000000_0
[sf-startdaemon-debug] 09/06/01 14:21:17 [IPC Server handler 1 on 34641] INFO util.ProcessTree : Killing all processes in the process group 14348 with SIGTERM. Exit code 0
[sf-startdaemon-debug] 09/06/01 14:21:22 [IPC Server handler 1 on 34641] WARN util.ProcessTree : Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process
[sf-startdaemon-debug] 09/06/01 14:21:22 [IPC Server handler 1 on 34641] INFO util.ProcessTree : Killing process group14348 with SIGKILL. Exit code 1
[sf-startdaemon-debug] 09/06/01 14:21:22 [IPC Server handler 1 on 34641] INFO mapred.DefaultTaskController : Process exited with exit code:143
[sf-startdaemon-debug] 09/06/01 14:21:22 [IPC Server handler 1 on 34641] INFO mapred.TaskTracker : addFreeSlot : current free slots : 2
[sf-startdaemon-debug] 09/06/01 14:21:22 [IPC Server handler 1 on 34641] INFO mapred.TaskRunner : attempt_200906011420_0001_r_000000_0 done; removing files.
[sf-startdaemon-debug] 09/06/01 14:21:22 [IPC Server handler 1 on 34641] INFO mapred.TaskTracker : Error cleaning up task runner: java.lang.NullPointerException
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.cleanup(TaskTracker.java:2549)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.jobHasFinished(TaskTracker.java:2433)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker.purgeTask(TaskTracker.java:1506)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker.shuffleError(TaskTracker.java:2716)
[sf-startdaemon-debug] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[sf-startdaemon-debug] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[sf-startdaemon-debug] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[sf-startdaemon-debug] at java.lang.reflect.Method.invoke(Method.java:597)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:960)
[sf-startdaemon-debug] at javax.security.auth.Subject.doAs(Subject.java:396)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.Server$Handler.run(Server.java:958)
[sf-startdaemon-debug] 09/06/01 14:21:22 [IPC Server handler 1 on 34641] WARN ipc.Server : IPC Server Responder, call shuffleError(attempt_200906011420_0001_r_000000_0, Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.) from 127.0.0.1:55925: output error
[sf-startdaemon-debug] 09/06/01 14:21:22 [IPC Server handler 1 on 34641] INFO ipc.Server : IPC Server handler 1 on 34641 caught: java.nio.channels.ClosedChannelException
[sf-startdaemon-debug] at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
[sf-startdaemon-debug] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.Server.channelWrite(Server.java:1200)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.Server.access$1900(Server.java:77)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:616)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:680)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.Server$Handler.run(Server.java:986)
[sf-startdaemon-debug] 09/06/01 14:21:22 [IPC Server handler 1 on 8012] INFO mapred.TaskInProgress : Error from attempt_200906011420_0001_r_000000_0: Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
[sf-startdaemon-debug] 09/06/01 14:21:22 [JVM Runner jvm_200906011420_0001_r_-1307341032 spawned.] INFO mapred.JvmManager : JVM : jvm_200906011420_0001_r_-1307341032 exited. Number of tasks it ran: 0
[sf-startdaemon-debug] 09/06/01 14:21:22 [IPC Server handler 1 on 8012] INFO mapred.JobTracker : Adding task (cleanup)'attempt_200906011420_0001_r_000000_0' to tip task_200906011420_0001_r_000000, for tracker 'tracker_morzine.hpl.hp.com:localhost/127.0.0.1:34641'
[sf-startdaemon-debug] 09/06/01 14:21:22 [TaskTracker] INFO mapred.TaskTracker : LaunchTaskAction (registerTask): attempt_200906011420_0001_r_000000_0 task's state:FAILED_UNCLEAN
[sf-startdaemon-debug] 09/06/01 14:21:22 [TaskLauncher for task] INFO mapred.TaskTracker : Trying to launch : attempt_200906011420_0001_r_000000_0
[sf-startdaemon-debug] 09/06/01 14:21:22 [TaskLauncher for task] INFO mapred.TaskTracker : In TaskLauncher, current free slots : 2 and trying to launch attempt_200906011420_0001_r_000000_0
[sf-startdaemon-debug] 09/06/01 14:21:22 [Thread-204] WARN mapred.TaskRunner : mkdirs failed. Ignoring
[sf-startdaemon-debug] 09/06/01 14:21:22 [Thread-204] INFO mapred.JvmManager : In JvmRunner constructed JVM ID: jvm_200906011420_0001_r_-1849792162
[sf-startdaemon-debug] 09/06/01 14:21:22 [Thread-204] INFO mapred.JvmManager : JVM Runner jvm_200906011420_0001_r_-1849792162 spawned.
[sf-startdaemon-debug] 09/06/01 14:21:23 [ExtDataNode] INFO datanode.DataNode : BlockReport of 6 blocks got processed in 1 msecs
[sf-startdaemon-debug] 09/06/01 14:21:24 [IPC Server handler 3 on 34641] INFO mapred.TaskTracker : JVM with ID: jvm_200906011420_0001_r_-1849792162 given task: attempt_200906011420_0001_r_000000_0
[sf-startdaemon-debug] 09/06/01 14:21:24 [IPC Server handler 0 on 34641] INFO mapred.TaskTracker : attempt_200906011420_0001_r_000000_0 0.0%
[sf-startdaemon-debug] 09/06/01 14:21:24 [IPC Server handler 1 on 34641] INFO mapred.TaskTracker : attempt_200906011420_0001_r_000000_0 0.0% cleanup
[sf-startdaemon-debug] 09/06/01 14:21:24 [IPC Server handler 2 on 34641] INFO mapred.TaskTracker : Task attempt_200906011420_0001_r_000000_0 is done.
[sf-startdaemon-debug] 09/06/01 14:21:24 [IPC Server handler 2 on 34641] INFO mapred.TaskTracker : reported output size for attempt_200906011420_0001_r_000000_0 was 0
[sf-startdaemon-debug] 09/06/01 14:21:24 [Thread-204] INFO mapred.TaskRunner : attempt_200906011420_0001_r_000000_0 done; removing files.
[sf-startdaemon-debug] 09/06/01 14:21:24 [Thread-204] INFO mapred.TaskTracker : Error cleaning up task runner: java.lang.NullPointerException
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.cleanup(TaskTracker.java:2549)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.taskFinished(TaskTracker.java:2329)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.reportTaskFinished(TaskTracker.java:2161)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:453)
[sf-startdaemon-debug] 09/06/01 14:21:24 [Thread-204] INFO mapred.TaskTracker : addFreeSlot : current free slots : 2
[sf-startdaemon-debug] 09/06/01 14:21:25 [JVM Runner jvm_200906011420_0001_r_-1849792162 spawned.] WARN util.ProcessTree : Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process
[sf-startdaemon-debug] 09/06/01 14:21:25 [JVM Runner jvm_200906011420_0001_r_-1849792162 spawned.] INFO util.ProcessTree : Killing all processes in the process group 14426 with SIGTERM. Exit code 1
[sf-startdaemon-debug] 09/06/01 14:21:25 [Thread-179] INFO mapred.TaskRunner : attempt_200906011420_0001_r_000000_0 done; removing files.
[sf-startdaemon-debug] 09/06/01 14:21:25 [Thread-179] INFO mapred.TaskTracker : Error cleaning up task runner: java.lang.NullPointerException
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.cleanup(TaskTracker.java:2549)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.taskFinished(TaskTracker.java:2329)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.reportTaskFinished(TaskTracker.java:2161)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:453)
[sf-startdaemon-debug] 09/06/01 14:21:25 [IPC Server handler 2 on 8012] INFO mapred.JobTracker : Adding task 'attempt_200906011420_0001_r_000000_1' to tip task_200906011420_0001_r_000000, for tracker 'tracker_morzine.hpl.hp.com:localhost/127.0.0.1:34641'
[sf-startdaemon-debug] 09/06/01 14:21:25 [IPC Server handler 2 on 8012] INFO mapred.JobTracker : Removed completed task 'attempt_200906011420_0001_r_000000_0' from 'tracker_morzine.hpl.hp.com:localhost/127.0.0.1:34641'
[sf-startdaemon-debug] 09/06/01 14:21:25 [TaskTracker] INFO mapred.TaskTracker : LaunchTaskAction (registerTask): attempt_200906011420_0001_r_000000_1 task's state:UNASSIGNED
[sf-startdaemon-debug] 09/06/01 14:21:25 [TaskLauncher for task] INFO mapred.TaskTracker : Trying to launch : attempt_200906011420_0001_r_000000_1
[sf-startdaemon-debug] 09/06/01 14:21:25 [TaskLauncher for task] INFO mapred.TaskTracker : In TaskLauncher, current free slots : 2 and trying to launch attempt_200906011420_0001_r_000000_1
[sf-startdaemon-debug] 09/06/01 14:21:25 [Thread-213] INFO mapred.JvmManager : In JvmRunner constructed JVM ID: jvm_200906011420_0001_r_-1068610141
[sf-startdaemon-debug] 09/06/01 14:21:25 [Thread-213] INFO mapred.JvmManager : JVM Runner jvm_200906011420_0001_r_-1068610141 spawned.
[sf-startdaemon-debug] 09/06/01 14:21:27 [IPC Server handler 0 on 34641] INFO mapred.TaskTracker : JVM with ID: jvm_200906011420_0001_r_-1068610141 given task: attempt_200906011420_0001_r_000000_1
[sf-startdaemon-debug] 2009/06/01 14:21:29:200 BST [INFO ][LivenessSender_HOST morzine.hpl.hp.com:rootProcess:testJobSubmission] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - Task Id : attempt_200906011420_0001_r_000000_0, Status : FAILED
[sf-startdaemon-debug] 2009/06/01 14:21:29:201 BST [INFO ][LivenessSender_HOST morzine.hpl.hp.com:rootProcess:testJobSubmission] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - Task Id : attempt_200906011420_0001_r_000000_0, Status : FAILED
[sf-startdaemon-debug] 09/06/01 14:21:30 [JVM Runner jvm_200906011420_0001_r_-1849792162 spawned.] WARN util.ProcessTree : Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process
[sf-startdaemon-debug] 09/06/01 14:21:30 [JVM Runner jvm_200906011420_0001_r_-1849792162 spawned.] INFO util.ProcessTree : Killing process group14426 with SIGKILL. Exit code 1
[sf-startdaemon-debug] 09/06/01 14:21:30 [JVM Runner jvm_200906011420_0001_r_-1849792162 spawned.] INFO mapred.DefaultTaskController : Process exited with exit code:0
[sf-startdaemon-debug] 09/06/01 14:21:30 [JVM Runner jvm_200906011420_0001_r_-1849792162 spawned.] INFO mapred.JvmManager : JVM : jvm_200906011420_0001_r_-1849792162 exited. Number of tasks it ran: 1
[sf-startdaemon-debug] 09/06/01 14:21:33 [IPC Server handler 0 on 34641] INFO mapred.TaskTracker : attempt_200906011420_0001_r_000000_1 0.0% reduce > copy >
[sf-startdaemon-debug] 09/06/01 14:21:34 [ExtDataNode] INFO datanode.DataNode : BlockReport of 6 blocks got processed in 11 msecs
[sf-startdaemon-debug] 09/06/01 14:21:36 [IPC Server handler 0 on 34641] INFO mapred.TaskTracker : attempt_200906011420_0001_r_000000_1 0.0% reduce > copy >
[sf-startdaemon-debug] 09/06/01 14:21:37 [IPC Server handler 3 on 34641] FATAL mapred.TaskTracker : Task: attempt_200906011420_0001_r_000000_1 - Killed due to Shuffle Failure: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
[sf-startdaemon-debug] 09/06/01 14:21:37 [IPC Server handler 3 on 34641] INFO mapred.TaskTracker : About to purge task: attempt_200906011420_0001_r_000000_1
[sf-startdaemon-debug] 09/06/01 14:21:37 [IPC Server handler 3 on 34641] INFO util.ProcessTree : Killing all processes in the process group 14471 with SIGTERM. Exit code 0
[sf-startdaemon-debug] 09/06/01 14:21:42 [IPC Server handler 3 on 34641] WARN util.ProcessTree : Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process
[sf-startdaemon-debug] 09/06/01 14:21:42 [IPC Server handler 3 on 34641] INFO util.ProcessTree : Killing process group14471 with SIGKILL. Exit code 1
[sf-startdaemon-debug] 09/06/01 14:21:42 [IPC Server handler 3 on 34641] INFO mapred.DefaultTaskController : Process exited with exit code:143
[sf-startdaemon-debug] 09/06/01 14:21:42 [IPC Server handler 3 on 34641] INFO mapred.TaskTracker : addFreeSlot : current free slots : 2
[sf-startdaemon-debug] 09/06/01 14:21:42 [JVM Runner jvm_200906011420_0001_r_-1068610141 spawned.] INFO mapred.JvmManager : JVM : jvm_200906011420_0001_r_-1068610141 exited. Number of tasks it ran: 0
[sf-startdaemon-debug] 09/06/01 14:21:42 [IPC Server handler 3 on 34641] INFO mapred.TaskRunner : attempt_200906011420_0001_r_000000_1 done; removing files.
[sf-startdaemon-debug] 09/06/01 14:21:42 [IPC Server handler 3 on 34641] INFO mapred.TaskTracker : Error cleaning up task runner: java.lang.NullPointerException
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.cleanup(TaskTracker.java:2549)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.jobHasFinished(TaskTracker.java:2433)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker.purgeTask(TaskTracker.java:1506)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker.shuffleError(TaskTracker.java:2716)
[sf-startdaemon-debug] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[sf-startdaemon-debug] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[sf-startdaemon-debug] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[sf-startdaemon-debug] at java.lang.reflect.Method.invoke(Method.java:597)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:960)
[sf-startdaemon-debug] at javax.security.auth.Subject.doAs(Subject.java:396)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.Server$Handler.run(Server.java:958)
[sf-startdaemon-debug] 09/06/01 14:21:42 [IPC Server handler 3 on 34641] WARN ipc.Server : IPC Server Responder, call shuffleError(attempt_200906011420_0001_r_000000_1, Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.) from 127.0.0.1:35456: output error
[sf-startdaemon-debug] 09/06/01 14:21:42 [IPC Server handler 3 on 34641] INFO ipc.Server : IPC Server handler 3 on 34641 caught: java.nio.channels.ClosedChannelException
[sf-startdaemon-debug] at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
[sf-startdaemon-debug] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.Server.channelWrite(Server.java:1200)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.Server.access$1900(Server.java:77)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:616)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:680)
[sf-startdaemon-debug] at org.apache.hadoop.ipc.Server$Handler.run(Server.java:986)
[sf-startdaemon-debug] 09/06/01 14:21:42 [IPC Server handler 0 on 8012] INFO mapred.TaskInProgress : Error from attempt_200906011420_0001_r_000000_1: Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
[sf-startdaemon-debug] 09/06/01 14:21:42 [IPC Server handler 0 on 8012] INFO mapred.JobTracker : Adding task (cleanup)'attempt_200906011420_0001_r_000000_1' to tip task_200906011420_0001_r_000000, for tracker 'tracker_morzine.hpl.hp.com:localhost/127.0.0.1:34641'
[sf-startdaemon-debug] 09/06/01 14:21:42 [TaskTracker] INFO mapred.TaskTracker : LaunchTaskAction (registerTask): attempt_200906011420_0001_r_000000_1 task's state:FAILED_UNCLEAN
[sf-startdaemon-debug] 09/06/01 14:21:42 [TaskLauncher for task] INFO mapred.TaskTracker : Trying to launch : attempt_200906011420_0001_r_000000_1
[sf-startdaemon-debug] 09/06/01 14:21:42 [TaskLauncher for task] INFO mapred.TaskTracker : In TaskLauncher, current free slots : 2 and trying to launch attempt_200906011420_0001_r_000000_1
[sf-startdaemon-debug] 09/06/01 14:21:42 [Thread-241] WARN mapred.TaskRunner : mkdirs failed. Ignoring
[sf-startdaemon-debug] 09/06/01 14:21:42 [Thread-241] INFO mapred.JvmManager : In JvmRunner constructed JVM ID: jvm_200906011420_0001_r_547561319
[sf-startdaemon-debug] 09/06/01 14:21:42 [Thread-241] INFO mapred.JvmManager : JVM Runner jvm_200906011420_0001_r_547561319 spawned.
[sf-startdaemon-debug] 09/06/01 14:21:43 [IPC Server handler 1 on 34641] INFO mapred.TaskTracker : JVM with ID: jvm_200906011420_0001_r_547561319 given task: attempt_200906011420_0001_r_000000_1
[sf-startdaemon-debug] 09/06/01 14:21:44 [IPC Server handler 2 on 34641] INFO mapred.TaskTracker : attempt_200906011420_0001_r_000000_1 0.0%
[sf-startdaemon-debug] 09/06/01 14:21:44 [ExtDataNode] INFO datanode.DataNode : BlockReport of 6 blocks got processed in 1 msecs
[sf-startdaemon-debug] 09/06/01 14:21:44 [IPC Server handler 3 on 34641] INFO mapred.TaskTracker : attempt_200906011420_0001_r_000000_1 0.0% cleanup
[sf-startdaemon-debug] 09/06/01 14:21:44 [IPC Server handler 0 on 34641] INFO mapred.TaskTracker : Task attempt_200906011420_0001_r_000000_1 is done.
[sf-startdaemon-debug] 09/06/01 14:21:44 [IPC Server handler 0 on 34641] INFO mapred.TaskTracker : reported output size for attempt_200906011420_0001_r_000000_1 was 0
[sf-startdaemon-debug] 09/06/01 14:21:44 [Thread-241] INFO mapred.TaskRunner : attempt_200906011420_0001_r_000000_1 done; removing files.
[sf-startdaemon-debug] 09/06/01 14:21:44 [Thread-241] INFO mapred.TaskTracker : Error cleaning up task runner: java.lang.NullPointerException
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.cleanup(TaskTracker.java:2549)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.taskFinished(TaskTracker.java:2329)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.reportTaskFinished(TaskTracker.java:2161)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:453)
[sf-startdaemon-debug] 09/06/01 14:21:44 [Thread-241] INFO mapred.TaskTracker : addFreeSlot : current free slots : 2
[sf-startdaemon-debug] 09/06/01 14:21:44 [JVM Runner jvm_200906011420_0001_r_547561319 spawned.] WARN util.ProcessTree : Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process
[sf-startdaemon-debug] 09/06/01 14:21:44 [JVM Runner jvm_200906011420_0001_r_547561319 spawned.] INFO util.ProcessTree : Killing all processes in the process group 14550 with SIGTERM. Exit code 1
[sf-startdaemon-debug] 09/06/01 14:21:45 [Thread-213] INFO mapred.TaskRunner : attempt_200906011420_0001_r_000000_1 done; removing files.
[sf-startdaemon-debug] 09/06/01 14:21:45 [Thread-213] INFO mapred.TaskTracker : Error cleaning up task runner: java.lang.NullPointerException
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.cleanup(TaskTracker.java:2549)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.taskFinished(TaskTracker.java:2329)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.reportTaskFinished(TaskTracker.java:2161)
[sf-startdaemon-debug] at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:453)
[sf-startdaemon-debug] 09/06/01 14:21:45 [IPC Server handler 3 on 8012] INFO mapred.JobTracker : Adding task 'attempt_200906011420_0001_r_000000_2' to tip task_200906011420_0001_r_000000, for tracker 'tracker_morzine.hpl.hp.com:localhost/127.0.0.1:34641'
[sf-startdaemon-debug] 09/06/01 14:21:45 [IPC Server handler 3 on 8012] INFO mapred.JobTracker : Removed completed task 'attempt_200906011420_0001_r_000000_1' from 'tracker_morzine.hpl.hp.com:localhost/127.0.0.1:34641'
[sf-startdaemon-debug] 09/06/01 14:21:45 [TaskTracker] INFO mapred.TaskTracker : LaunchTaskAction (registerTask): attempt_200906011420_0001_r_000000_2 task's state:UNASSIGNED
[sf-startdaemon-debug] 09/06/01 14:21:45 [TaskLauncher for task] INFO mapred.TaskTracker : Trying to launch : attempt_200906011420_0001_r_000000_2
[sf-startdaemon-debug] 09/06/01 14:21:45 [TaskLauncher for task] INFO mapred.TaskTracker : In TaskLauncher, current free slots : 2 and trying to launch attempt_200906011420_0001_r_000000_2
[sf-startdaemon-debug] 09/06/01 14:21:45 [Thread-250] INFO mapred.JvmManager : In JvmRunner constructed JVM ID: jvm_200906011420_0001_r_1194438658
[sf-startdaemon-debug] 09/06/01 14:21:45 [Thread-250] INFO mapred.JvmManager : JVM Runner jvm_200906011420_0001_r_1194438658 spawned.
[sf-startdaemon-debug] 2009/06/01 14:21:45:874 BST [INFO ][TerminatorThread] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - Initiating JobTracker termination; serviceThread=Thread[JobTracker,5,RMI Runtime] service=JobTracker instance org.apache.hadoop.mapred.ExtJobTracker@8797fa6 in state LIVE
[sf-startdaemon-debug] 2009/06/01 14:21:45:875 BST [INFO ][TerminatorThread] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - Terminating deployer thread
[sf-startdaemon-debug] 2009/06/01 14:21:45:877 BST [INFO ][TerminatorThread] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - Requesting thread termination
[sf-startdaemon-debug] 2009/06/01 14:21:45:877 BST [INFO ][TerminatorThread] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - waiting for thread to finish
[sf-startdaemon-debug] 2009/06/01 14:21:45:878 BST [INFO ][TerminatorThread] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - Initiating TaskTracker termination; serviceThread=Thread[TaskTracker,5,RMI Runtime] service=tracker_morzine.hpl.hp.com:localhost/127.0.0.1:34641 instance org.apache.hadoop.mapred.ExtTaskTracker@872bb64 in state LIVE. web port=50060 reporting localhost/127.0.0.1:34641
[sf-startdaemon-debug] 2009/06/01 14:21:45:878 BST [INFO ][TerminatorThread] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - Terminating deployer thread
[sf-startdaemon-debug] 2009/06/01 14:21:45:878 BST [INFO ][TerminatorThread] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - Requesting thread termination
[sf-startdaemon-debug] 2009/06/01 14:21:45:878 BST [INFO ][JobTracker] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - Exiting JobTracker worker thread; service is JobTracker instance org.apache.hadoop.mapred.ExtJobTracker@8797fa6 in state LIVE
[sf-startdaemon-debug] 2009/06/01 14:21:45:878 BST [INFO ][TerminatorThread] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - waiting for thread to finish
[sf-startdaemon-debug] 09/06/01 14:21:45 [TaskTracker] INFO mapred.TaskTracker : Interrupted. Closing down.
[sf-startdaemon-debug] 2009/06/01 14:21:45:878 BST [WARN ][TerminatorThread] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - Hadoop Service thread did not terminate within the expected shutdown period. Service is JobTracker instance org.apache.hadoop.mapred.ExtJobTracker@8797fa6 in state LIVE
[sf-startdaemon-debug] 2009/06/01 14:21:45:879 BST [INFO ][TerminatorThread] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - Terminating hadoopService service JobTracker
[sf-startdaemon-debug] 09/06/01 14:21:45 [TerminatorThread] INFO mapred.ExtJobTracker : State change: JobTracker is now CLOSED
[sf-startdaemon-debug] 09/06/01 14:21:45 [TerminatorThread] INFO mapred.JobTracker : Stopping infoServer
[sf-startdaemon-debug] 2009/06/01 14:21:45:886 BST [WARN ][TerminatorThread] HOST morzine.hpl.hp.com:rootProcess:testJobSubmission - Hadoop Service thread did not terminate within the expected shutdown period. Service is tracker_morzine.hpl.hp.com:localhost/127.0.0.1:34641 instance org.apache.hadoop.mapred.ExtTaskTracker@872bb64 in state LIVE. web port=50060 reporting localhost/127.0.0.1:34641
> JobSubmissionTest failing with "Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES"
> ----------------------------------------------------------------------------------
>
> Key: SFOS-1211
> URL: http://jira.smartfrog.org/jira/browse/SFOS-1211
> Project: SmartFrog
> Issue Type: Bug
> Reporter: Steve Loughran
> Assignee: Julio Guijarro
>
> "Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES"
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.smartfrog.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
|