加入收藏 | 设为首页 | 会员中心 | 我要投稿 李大同 (https://www.lidatong.com.cn/)- 科技、建站、经验、云计算、5G、大数据,站长网!
当前位置: 首页 > 编程开发 > Java > 正文

错误:java.io.IOException:错误的值类:类org.apache.hadoop.i

发布时间:2020-12-15 04:17:26 所属栏目:Java 来源:网络整理
导读:我有我的mapper和reducer如下.但我得到了一些奇怪的例外. 我无法弄清楚为什么会抛出这种异常. public static class MyMapper implements MapperLongWritable,Text,Info { @Override public void map(LongWritable key,Text value,OutputCollectorText,Info o
我有我的mapper和reducer如下.但我得到了一些奇怪的例外.
我无法弄清楚为什么会抛出这种异常.

public static class MyMapper implements Mapper<LongWritable,Text,Info> {

    @Override
    public void map(LongWritable key,Text value,OutputCollector<Text,Info> output,Reporter reporter)
        throws IOException {
        Text text = new Text("someText")
            //process 
        output.collect(text,infoObjeject);
    }

}

public static class MyReducer implements Reducer<Text,Info,Text> {

    @Override
    public void reduce(Text key,Iterator<Info> values,Text> output,Reporter reporter)
        throws IOException {
        String value = "xyz" //derived in some way
        //process
        output.collect(key,new Text(value)); //exception occurs at this line
    }

}

System.out.println("Starting v14 ");
JobConf conf = new JobConf(RouteBuilderJob.class);
conf.setJobName("xyz");

String jarLocation =ClassUtil.findContainingJar(getClass());

System.out.println("path of jar file = " + jarLocation);

conf.setJarByClass(RouteBuilderJob.class);

conf.setMapOutputKeyClass(Text.class);
conf.setMapOutputValueClass(Info.class);

conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(Text.class);

//am i missing something here???

conf.setMapperClass(RouteBuilderJob.RouteMapper.class);
conf.setCombinerClass(RouteBuilderJob.RouteReducer.class);
conf.setReducerClass(RouteBuilderJob.RouteReducer.class);


conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);

FileInputFormat.setInputPaths(conf,new Path(args[0]));
FileOutputFormat.setOutputPath(conf,new Path(args[1]));

JobClient.runJob(conf);

我得到一个例外:

Error: java.io.IOException: wrong value class: class org.apache.hadoop.io.Text is not class com.xyz.mypackage.Info
at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:199)
at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1307)
at com.xyz.mypackage.job.MyJob$RouteReducer.reduce(MyJob.java:156)
at com.xyz.mypackage.job.MyJob$RouteReducer.reduce(MyJob.java:1)

内部信息对象(实现Writable)使用Text进行序列化

@Override
public void write(DataOutput out) throws IOException {
    Gson gson = new Gson();
    String searlizedStr = gson.toJson(this);
    Text.writeString(out,searlizedStr);
}

@Override
public void readFields(DataInput in) throws IOException {
    String s = Text.readString(in);
    Gson gson = new Gson();
    JsonReader jsonReader = new JsonReader(new StringReader(s));
    jsonReader.setLenient(true);

Info info = gson.fromJson(jsonReader,Info.class);
    //set fields using this.somefield = info.getsomefield() 
}

解决方法

从技术上讲,reduce的输出类型应与输入类型相同.如果使用组合器,合并器的输出被送入减速器,则必须如此.

(编辑:李大同)

【声明】本站内容均来自网络,其相关言论仅代表作者个人观点,不代表本站立场。若无意侵犯到您的权利,请及时与联系站长删除相关内容!

    推荐文章
      热点阅读