我正在使用oozie工作流运行此代码并获取类型不匹配错误:
public static class mapClass extends Mapper<Object, Text, LongWritable, LongWritable> {
public void map(Object, Text, Context..)
..
context.write(<LongWritable type> , <LongWritable type> )
}
public static class reduceClass extends Reducer<LongWritable, LongWritable,LongWritable, LongWritable> {
public void reduce(LongWritable, LongWritable, context)
..
context.write(<LongWritable type>, <LongWritable type>)
{
}
java.io.IOException:从map中键入不匹配的值:expected org.apache.hadoop.io.LongWritable,recieved org.apache.hadoop.io.Text
我在我的工作流程中使用new-api . 相同的代码没有使用oozie工作正常 .
任何帮助,将不胜感激 . 谢谢 .
-----代码示例---
package org.apache.hadoop;
import java.io.IOException;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
public class MapperLong extends Mapper<LongWritable, Text, LongWritable, LongWritable> {
public final static int COL_ZERO = 0;
public final static int COL_ONE = 1;
public final static int COL_TWO = 2;
public final static int COL_THREE = 3;
@Override
public void map(LongWritable offset, Text line, Context context)
throws IOException, InterruptedException {
String[] parts = (line.toString()).split(" ");
LongWritable one = new LongWritable(Integer.parseInt(parts[COL_ONE]));
LongWritable two = new LongWritable(Integer.parseInt(parts[COL_TWO]));
context.write(one, two);
}
}
package org.apache.hadoop;
import java.io.IOException;
import java.util.HashSet;
import java.util.Set;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.mapreduce.Reducer;
public class ReducerLong extends Reducer<LongWritable, LongWritable, LongWritable, LongWritable> {
@Override
public void reduce(LongWritable colOneKey, Iterable<LongWritable> values,
Context context) throws IOException, InterruptedException{
Set<Integer> colTwo = new HashSet<Integer>();
for (LongWritable val : values) {
colTwo.add(Integer.valueOf((int)val.get()));
}
context.write(colOneKey, new LongWritable(colTwo.size()));
}
}
}
java.io.IOException:在map中输入值不匹配:期望org.apache.hadoop.io.LongWritable,收到org.apache.hadoop.mapred.MapTask上的org.apache.hadoop.io.Text $ MapOutputBuffer.collect(MapTask .java:876)在org.apache的org.apache.hadoop.mapred.MapTask $ NewOutputCollector.write(MapTask.java:574)org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) . org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)中的hadoop.mapreduce.Mapper.map(Mapper.java:124) 647)atg.apache.hadoop.mapred.MapTask.run(MapTask.java:323)atg.apache.hadoop.mapred.Child $ 4.run(Child.java:270)at java.security.AccessController.doPrivileged( Native方法)位于org.apache.hadoop.mapred.Child的org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)的javax.security.auth.Subject.doAs(Subject.java:396) . 主(Child.java:264)
Input :
34 342 1 1
45 23 0 1
..
..
注意:我将对象类型更改为LongWritable,这没有任何区别 . 在workflow.xml中使用以下属性时抛出上述异常 . 如果没有以下属性,代码将执行生成与输入前缀相同的输出!
<property>
<name>mapred.output.key.class</name>
<value>org.apache.hadoop.io.LongWritable</value>
</property>
<property>
<name>mapred.output.value.class</name>
<value>org.apache.hadoop.io.LongWritable</value>
</property>
2 回答
好的,我想通了 . 问题出在我定义的oozie工作流程中 .
[参考https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases MR Api]由于某些原因无法 grab 我的眼睛:-(因为我使用的是非api工作流程的修改版本!
谢谢大家的时间 .
很可能您使用reducer作为合并器,这意味着它在 Map 的上下文中运行 . 在这里看到一个类似的问题Wrong key class: Text is not IntWritable