首页 文章

hadoop map任务重启时减少完成80％

提问于
浏览 次
1

我在2.2.0上运行hadoop在一个大约50个节点的集群上，我的工作是64个 Map 任务和20个减少任务 . Map 在大约30分钟内完成，然后所有的reduce任务都在运行，但我发现了一个像这样的奇怪日志：

14/10/08 21:41:04 INFO mapreduce.Job:  map 37% reduce 10%
    14/10/08 21:41:05 INFO mapreduce.Job:  map 38% reduce 11%
    14/10/08 21:41:07 INFO mapreduce.Job:  map 39% reduce 12%
    14/10/08 21:41:08 INFO mapreduce.Job:  map 40% reduce 12%
    14/10/08 21:41:09 INFO mapreduce.Job:  map 41% reduce 12%
    14/10/08 21:41:10 INFO mapreduce.Job:  map 42% reduce 12%
    14/10/08 21:41:12 INFO mapreduce.Job:  map 44% reduce 12%
    14/10/08 21:41:14 INFO mapreduce.Job:  map 45% reduce 12%
    14/10/08 21:41:16 INFO mapreduce.Job:  map 46% reduce 13%
    14/10/08 21:41:18 INFO mapreduce.Job:  map 48% reduce 13%
    14/10/08 21:41:21 INFO mapreduce.Job:  map 49% reduce 13%
    14/10/08 21:41:22 INFO mapreduce.Job:  map 50% reduce 13%
    14/10/08 21:41:25 INFO mapreduce.Job:  map 51% reduce 14%
    14/10/08 21:41:29 INFO mapreduce.Job:  map 52% reduce 15%
    14/10/08 21:41:34 INFO mapreduce.Job:  map 52% reduce 16%
    14/10/08 21:41:38 INFO mapreduce.Job:  map 53% reduce 17%
    14/10/08 21:41:53 INFO mapreduce.Job:  map 53% reduce 18%
    14/10/08 21:42:09 INFO mapreduce.Job:  map 54% reduce 18%
    14/10/08 21:42:18 INFO mapreduce.Job:  map 55% reduce 18%
    14/10/08 21:42:37 INFO mapreduce.Job:  map 56% reduce 18%
    14/10/08 21:42:49 INFO mapreduce.Job:  map 56% reduce 19%
    14/10/08 21:42:53 INFO mapreduce.Job:  map 57% reduce 19%
    14/10/08 21:42:57 INFO mapreduce.Job:  map 58% reduce 19%
    14/10/08 21:42:59 INFO mapreduce.Job:  map 59% reduce 19%
    14/10/08 21:43:00 INFO mapreduce.Job:  map 60% reduce 19%
    14/10/08 21:43:05 INFO mapreduce.Job:  map 61% reduce 19%
    14/10/08 21:43:06 INFO mapreduce.Job:  map 62% reduce 19%
    14/10/08 21:43:09 INFO mapreduce.Job:  map 62% reduce 20%
    14/10/08 21:43:13 INFO mapreduce.Job:  map 63% reduce 20%
    14/10/08 21:43:16 INFO mapreduce.Job:  map 63% reduce 21%
    14/10/08 21:43:19 INFO mapreduce.Job:  map 64% reduce 21%
    14/10/08 21:43:36 INFO mapreduce.Job:  map 65% reduce 21%
    14/10/08 21:43:42 INFO mapreduce.Job:  map 66% reduce 21%
    14/10/08 21:43:44 INFO mapreduce.Job:  map 67% reduce 22%
    14/10/08 21:44:00 INFO mapreduce.Job:  map 68% reduce 22%
    14/10/08 21:44:09 INFO mapreduce.Job:  map 70% reduce 22%
    14/10/08 21:44:12 INFO mapreduce.Job:  map 70% reduce 23%
    14/10/08 21:44:18 INFO mapreduce.Job:  map 71% reduce 23%
    14/10/08 21:44:24 INFO mapreduce.Job:  map 72% reduce 23%
    14/10/08 21:44:28 INFO mapreduce.Job:  map 72% reduce 24%
    14/10/08 21:44:35 INFO mapreduce.Job:  map 73% reduce 24%
    14/10/08 21:44:44 INFO mapreduce.Job:  map 74% reduce 24%
    14/10/08 21:44:50 INFO mapreduce.Job:  map 75% reduce 24%
    14/10/08 21:44:51 INFO mapreduce.Job:  map 76% reduce 24%
    14/10/08 21:44:54 INFO mapreduce.Job:  map 77% reduce 25%
    14/10/08 21:45:00 INFO mapreduce.Job:  map 78% reduce 25%
    14/10/08 21:45:07 INFO mapreduce.Job:  map 78% reduce 26%
    14/10/08 21:45:15 INFO mapreduce.Job:  map 79% reduce 26%
    14/10/08 21:45:17 INFO mapreduce.Job:  map 80% reduce 26%
    14/10/08 21:45:20 INFO mapreduce.Job:  map 81% reduce 26%
    14/10/08 21:45:24 INFO mapreduce.Job:  map 82% reduce 26%
    14/10/08 21:45:27 INFO mapreduce.Job:  map 83% reduce 26%
    14/10/08 21:45:29 INFO mapreduce.Job:  map 83% reduce 27%
    14/10/08 21:45:34 INFO mapreduce.Job:  map 85% reduce 27%
    14/10/08 21:45:37 INFO mapreduce.Job:  map 86% reduce 27%
    14/10/08 21:45:42 INFO mapreduce.Job:  map 86% reduce 28%
    14/10/08 21:45:43 INFO mapreduce.Job:  map 88% reduce 28%
    14/10/08 21:45:45 INFO mapreduce.Job:  map 89% reduce 28%
    14/10/08 21:45:47 INFO mapreduce.Job:  map 89% reduce 29%
    14/10/08 21:45:56 INFO mapreduce.Job:  map 89% reduce 30%
    14/10/08 21:45:58 INFO mapreduce.Job:  map 90% reduce 30%
    14/10/08 21:46:00 INFO mapreduce.Job:  map 91% reduce 30%
    14/10/08 21:46:06 INFO mapreduce.Job:  map 92% reduce 30%
    14/10/08 21:46:10 INFO mapreduce.Job:  map 92% reduce 31%
    14/10/08 21:46:12 INFO mapreduce.Job:  map 93% reduce 31%
    14/10/08 21:46:24 INFO mapreduce.Job:  map 94% reduce 31%
    14/10/08 21:48:02 INFO mapreduce.Job:  map 95% reduce 31%
    14/10/08 21:48:12 INFO mapreduce.Job:  map 95% reduce 32%
    14/10/08 21:48:13 INFO mapreduce.Job:  map 96% reduce 32%
    14/10/08 21:48:22 INFO mapreduce.Job:  map 97% reduce 32%
    14/10/08 21:49:00 INFO mapreduce.Job:  map 98% reduce 32%
    14/10/08 21:49:09 INFO mapreduce.Job:  map 98% reduce 33%
    14/10/08 21:49:19 INFO mapreduce.Job:  map 99% reduce 33%
    14/10/08 21:49:22 INFO mapreduce.Job:  map 100% reduce 33%
    14/10/08 21:49:27 INFO mapreduce.Job:  map 100% reduce 34%
    14/10/08 21:49:28 INFO mapreduce.Job:  map 100% reduce 37%
    14/10/08 21:49:29 INFO mapreduce.Job:  map 100% reduce 40%
    14/10/08 21:49:30 INFO mapreduce.Job:  map 100% reduce 42%
    14/10/08 21:49:32 INFO mapreduce.Job:  map 100% reduce 44%
    14/10/08 21:49:33 INFO mapreduce.Job:  map 100% reduce 46%
    14/10/08 21:49:34 INFO mapreduce.Job:  map 100% reduce 48%
    14/10/08 21:49:35 INFO mapreduce.Job:  map 100% reduce 49%
    14/10/08 21:49:36 INFO mapreduce.Job:  map 100% reduce 54%
    14/10/08 21:49:37 INFO mapreduce.Job:  map 100% reduce 55%
    14/10/08 21:49:39 INFO mapreduce.Job:  map 100% reduce 58%
    14/10/08 21:49:40 INFO mapreduce.Job:  map 100% reduce 59%
    14/10/08 21:49:41 INFO mapreduce.Job:  map 100% reduce 60%
    14/10/08 21:49:42 INFO mapreduce.Job:  map 100% reduce 61%
    14/10/08 21:49:44 INFO mapreduce.Job:  map 100% reduce 62%
    14/10/08 21:49:46 INFO mapreduce.Job:  map 100% reduce 63%
    14/10/08 21:49:48 INFO mapreduce.Job:  map 100% reduce 64%
    14/10/08 21:49:50 INFO mapreduce.Job:  map 100% reduce 65%
    14/10/08 21:49:56 INFO mapreduce.Job:  map 100% reduce 66%
    14/10/08 21:50:03 INFO mapreduce.Job:  map 100% reduce 67%
    14/10/08 21:52:11 INFO mapreduce.Job:  map 100% reduce 68%
    14/10/08 21:56:12 INFO mapreduce.Job:  map 100% reduce 69%
    14/10/08 22:00:35 INFO mapreduce.Job:  map 100% reduce 71%
    14/10/08 22:00:40 INFO mapreduce.Job:  map 100% reduce 72%
    14/10/08 22:00:55 INFO mapreduce.Job:  map 100% reduce 74%
    14/10/08 22:01:09 INFO mapreduce.Job:  map 100% reduce 75%
    14/10/08 22:01:14 INFO mapreduce.Job:  map 100% reduce 77%
    14/10/08 22:02:23 INFO mapreduce.Job:  map 100% reduce 78%
    14/10/08 22:02:37 INFO mapreduce.Job:  map 100% reduce 80%
    14/10/08 22:03:32 INFO mapreduce.Job:  map 100% reduce 82%
    14/10/08 22:04:14 INFO mapreduce.Job:  map 100% reduce 83%
    14/10/08 22:04:43 INFO mapreduce.Job:  map 100% reduce 85%
    14/10/08 22:08:11 INFO mapreduce.Job:  map 100% reduce 86%
    14/10/08 22:08:45 INFO mapreduce.Job:  map 100% reduce 88%
    14/10/08 22:09:53 INFO mapreduce.Job:  map 100% reduce 89%
    14/10/08 22:11:21 INFO mapreduce.Job:  map 100% reduce 91%
    14/10/08 22:11:44 INFO mapreduce.Job:  map 100% reduce 92%
    14/10/08 22:13:23 INFO mapreduce.Job:  map 100% reduce 94%
    14/10/08 22:13:25 INFO mapreduce.Job:  map 100% reduce 95%
    14/10/08 22:15:21 INFO mapreduce.Job:  map 100% reduce 97%
    14/10/08 22:18:14 INFO mapreduce.Job:  map 97% reduce 98%
    14/10/08 22:18:15 INFO mapreduce.Job:  map 97% reduce 100%
    14/10/08 22:19:14 INFO mapreduce.Job:  map 97% reduce 95%
    14/10/08 22:19:15 INFO mapreduce.Job:  map 97% reduce 90%
    14/10/08 22:19:27 INFO mapreduce.Job:  map 97% reduce 91%
    14/10/08 22:19:35 INFO mapreduce.Job:  map 97% reduce 92%
    14/10/08 22:20:36 INFO mapreduce.Job:  map 97% reduce 93%
    14/10/08 22:34:55 INFO mapreduce.Job:  map 94% reduce 93%
    14/10/08 22:35:20 INFO mapreduce.Job:  map 95% reduce 93%
    14/10/08 22:40:46 INFO mapreduce.Job:  map 96% reduce 93%
    14/10/08 22:40:55 INFO mapreduce.Job:  map 97% reduce 93%
    14/10/08 22:40:59 INFO mapreduce.Job:  map 97% reduce 94%
    14/10/08 22:41:12 INFO mapreduce.Job:  map 97% reduce 95%
    14/10/08 22:51:43 INFO mapreduce.Job:  map 98% reduce 95%
    14/10/08 22:52:52 INFO mapreduce.Job:  map 99% reduce 95%
    14/10/08 22:53:01 INFO mapreduce.Job:  map 100% reduce 95%
    14/10/08 22:53:03 INFO mapreduce.Job:  map 100% reduce 96%
    14/10/08 22:53:28 INFO mapreduce.Job:  map 100% reduce 97%
    14/10/08 22:57:19 INFO mapreduce.Job:  map 100% reduce 98%
    14/10/08 23:09:55 INFO mapreduce.Job:  map 100% reduce 100%
    14/10/08 23:10:00 INFO mapreduce.Job: Job job_1412772687841_0019 completed successfully
hadoop

1 回答

  • 0

    如果你的Node在作业之间死亡，就会发生这种情况 .

    例如，任务跟踪器正在执行任务并且其完成50％ . 它通过心跳更新Job Tracker . 然后，如果该节点终止，则进度将丢失，并且必须重新启动作业 . 这可能会导致总体完成百分比降低 .

    回复于

相关问题