首页 文章

(Hadoop):在运行mapreduce作业时,不会执行/调用reduce方法

提问于
浏览
0

我在执行一个mapreduce工作时遇到了问题 . 作为我的map reduce任务的一部分,我正在使用mapreduce连接,其中包括多个map方法和单个reducer方法 .

我的两个map方法都被执行了,但是我的reducer没有从我的驱动程序类执行/调用 .

因此,最终输出仅包含在我的映射阶段收集的数据 .

我在减少阶段使用了错误的输入和输出值吗? Map 和减少阶段之间是否存在输入和输出不匹配?

在这方面帮助我 .

这是我的代码..

public class CompareInputTest extends Configured implements Tool  {

public static class FirstFileInputMapperTest extends Mapper<LongWritable,Text,Text,Text>{


    private Text word = new Text();
    private String keyData,data,sourceTag = "S1$";

    public void map(LongWritable key,Text value,Context context) throws IOException, InterruptedException{

        String[] values = value.toString().split(";");
        keyData = values[1];
        data = values[2];

        context.write(new Text(keyData), new Text(data+sourceTag));


    }
}

public static class SecondFileInputMapperTest extends Mapper<LongWritable,Text,Text,Text>{
    private Text word = new Text();
    private String keyData,data,sourceTag = "S2$";
    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{

        String[] values = value.toString().split(";");
        keyData = values[1];
        data = values[2];


        context.write(new Text(keyData), new Text(data+sourceTag));

    }

              }

public static class CounterReducerTest extends Reducer
 {
    private String status1, status2;

    public void reduce(Text key, Iterable<Text> values, Context context)
       throws IOException, InterruptedException {
        System.out.println("in reducer");

        for(Text value:values)
           {
           String splitVals[] = currValue.split("$");
        System.out.println("in reducer");
       /*
        * identifying the record source that corresponds to a commonkey and
        * parses the values accordingly
       */
      if (splitVals[0].equals("S1")) {
         status1 = splitVals[1] != null ? splitVals[1].trim(): "status1";
      } else if (splitVals[0].equals("S2")) {
          // getting the file2 and using the same to obtain the Message
          status2 = splitVals[2] != null ? splitVals[2].trim(): "status2";
      }
           }

        context.write(key, new Text(status1+"$$$"));
    }






 public static void main(String[] args) throws Exception {


     int res = ToolRunner.run(new Configuration(), new CompareInputTest(),
             args);
System.exit(res);

     }

}

public int run(String[] args) throws Exception {
    Configuration conf = new Configuration();
     Job job = new Job(conf, "count");
     job.setJarByClass(CompareInputTest.class);
     MultipleInputs.addInputPath(job,new Path(args[0]),TextInputFormat.class,FirstFileInputMapperTest.class);
     MultipleInputs.addInputPath(job,new Path(args[1]),TextInputFormat.class,SecondFileInputMapperTest.class);
     job.setReducerClass(CounterReducerTest.class);
     //job.setNumReduceTasks(1);
     job.setMapOutputKeyClass(Text.class);
     job.setMapOutputValueClass(Text.class);
     job.setOutputKeyClass(Text.class);
     job.setOutputValueClass(Text.class);




     FileOutputFormat.setOutputPath(job, new Path(args[2]));



     return (job.waitForCompletion(true) ? 0 : 1);

}

}

1 回答

  • 2

    只需检查reducer类的原型 .

    extends Reducer<KEY, VALUE, KEY,VALUE>
    

    在您的情况下,因为reducer作为输入获取并作为输出Text发出,所以更改定义

    public static class CounterReducerTest extends Reducer
    

    public static class CounterReducerTest extends Reducer<Text,Text,Text,Text>
    

相关问题