首页 文章

输入 Map 中的值不匹配

提问于
浏览
0

我正在使用oozie工作流运行此代码并获取类型不匹配错误:

public static class mapClass extends Mapper<Object, Text, LongWritable, LongWritable> {
        public void map(Object, Text, Context..)
..
         context.write(<LongWritable type> , <LongWritable type> )
}

public static class reduceClass extends Reducer<LongWritable, LongWritable,LongWritable, LongWritable> {
        public void reduce(LongWritable, LongWritable, context)
..
    context.write(<LongWritable type>, <LongWritable type>)
{

}

java.io.IOException:从map中键入不匹配的值:expected org.apache.hadoop.io.LongWritable,recieved org.apache.hadoop.io.Text

我在我的工作流程中使用new-api . 相同的代码没有使用oozie工作正常 .

任何帮助,将不胜感激 . 谢谢 .

-----代码示例---

package org.apache.hadoop;
import java.io.IOException;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;


public class MapperLong extends Mapper<LongWritable, Text, LongWritable, LongWritable> {
    public final static int COL_ZERO = 0;
    public final static int COL_ONE = 1;
    public final static int COL_TWO = 2;
    public final static int COL_THREE = 3;

@Override
    public void map(LongWritable offset, Text line, Context context)
throws IOException, InterruptedException {
        String[] parts = (line.toString()).split(" ");

        LongWritable one = new LongWritable(Integer.parseInt(parts[COL_ONE]));
        LongWritable two = new LongWritable(Integer.parseInt(parts[COL_TWO]));
        context.write(one, two);

    }
}




package org.apache.hadoop;

import java.io.IOException;
import java.util.HashSet;
import java.util.Set;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.mapreduce.Reducer;


public class ReducerLong extends Reducer<LongWritable, LongWritable, LongWritable, LongWritable> {
@Override
    public void reduce(LongWritable colOneKey, Iterable<LongWritable> values,
        Context context) throws IOException, InterruptedException{
        Set<Integer> colTwo = new HashSet<Integer>();

        for (LongWritable val : values) {
            colTwo.add(Integer.valueOf((int)val.get()));
        }
        context.write(colOneKey, new LongWritable(colTwo.size()));
    }
}
}

java.io.IOException:在map中输入值不匹配:期望org.apache.hadoop.io.LongWritable,收到org.apache.hadoop.mapred.MapTask上的org.apache.hadoop.io.Text $ MapOutputBuffer.collect(MapTask .java:876)在org.apache的org.apache.hadoop.mapred.MapTask $ NewOutputCollector.write(MapTask.java:574)org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) . org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)中的hadoop.mapreduce.Mapper.map(Mapper.java:124) 647)atg.apache.hadoop.mapred.MapTask.run(MapTask.java:323)atg.apache.hadoop.mapred.Child $ 4.run(Child.java:270)at java.security.AccessController.doPrivileged( Native方法)位于org.apache.hadoop.mapred.Child的org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)的javax.security.auth.Subject.doAs(Subject.java:396) . 主(Child.java:264)

Input :
34 342 1 1
45 23 0 1
..
..

注意:我将对象类型更改为LongWritable,这没有任何区别 . 在workflow.xml中使用以下属性时抛出上述异常 . 如果没有以下属性,代码将执行生成与输入前缀相同的输出!

<property>
            <name>mapred.output.key.class</name>
            <value>org.apache.hadoop.io.LongWritable</value>
        </property>
        <property>
            <name>mapred.output.value.class</name>
            <value>org.apache.hadoop.io.LongWritable</value>
        </property>

2 回答

  • 1

    好的,我想通了 . 问题出在我定义的oozie工作流程中 .

    I had 
    <name>mapreduce.mapper.class</name>
    and 
    <name>mapreduce.reducer.class</name>
    
    instead of
    <name>mapreduce.map.class</name>
    and 
    <name>mapreduce.reduce.class</name>
    

    [参考https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases MR Api]由于某些原因无法 grab 我的眼睛:-(因为我使用的是非api工作流程的修改版本!

    谢谢大家的时间 .

  • 2

    很可能您使用reducer作为合并器,这意味着它在 Map 的上下文中运行 . 在这里看到一个类似的问题Wrong key class: Text is not IntWritable

相关问题