错误:java.io.IOException:错误的值类:类org.apache.hadoop.i
发布时间:2020-12-15 04:17:26 所属栏目:Java 来源:网络整理
导读:我有我的mapper和reducer如下.但我得到了一些奇怪的例外. 我无法弄清楚为什么会抛出这种异常. public static class MyMapper implements MapperLongWritable,Text,Info { @Override public void map(LongWritable key,Text value,OutputCollectorText,Info o
我有我的mapper和reducer如下.但我得到了一些奇怪的例外.
我无法弄清楚为什么会抛出这种异常. public static class MyMapper implements Mapper<LongWritable,Text,Info> { @Override public void map(LongWritable key,Text value,OutputCollector<Text,Info> output,Reporter reporter) throws IOException { Text text = new Text("someText") //process output.collect(text,infoObjeject); } } public static class MyReducer implements Reducer<Text,Info,Text> { @Override public void reduce(Text key,Iterator<Info> values,Text> output,Reporter reporter) throws IOException { String value = "xyz" //derived in some way //process output.collect(key,new Text(value)); //exception occurs at this line } } System.out.println("Starting v14 "); JobConf conf = new JobConf(RouteBuilderJob.class); conf.setJobName("xyz"); String jarLocation =ClassUtil.findContainingJar(getClass()); System.out.println("path of jar file = " + jarLocation); conf.setJarByClass(RouteBuilderJob.class); conf.setMapOutputKeyClass(Text.class); conf.setMapOutputValueClass(Info.class); conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(Text.class); //am i missing something here??? conf.setMapperClass(RouteBuilderJob.RouteMapper.class); conf.setCombinerClass(RouteBuilderJob.RouteReducer.class); conf.setReducerClass(RouteBuilderJob.RouteReducer.class); conf.setInputFormat(TextInputFormat.class); conf.setOutputFormat(TextOutputFormat.class); FileInputFormat.setInputPaths(conf,new Path(args[0])); FileOutputFormat.setOutputPath(conf,new Path(args[1])); JobClient.runJob(conf); 我得到一个例外: Error: java.io.IOException: wrong value class: class org.apache.hadoop.io.Text is not class com.xyz.mypackage.Info at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:199) at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1307) at com.xyz.mypackage.job.MyJob$RouteReducer.reduce(MyJob.java:156) at com.xyz.mypackage.job.MyJob$RouteReducer.reduce(MyJob.java:1) 内部信息对象(实现Writable)使用Text进行序列化 @Override public void write(DataOutput out) throws IOException { Gson gson = new Gson(); String searlizedStr = gson.toJson(this); Text.writeString(out,searlizedStr); } @Override public void readFields(DataInput in) throws IOException { String s = Text.readString(in); Gson gson = new Gson(); JsonReader jsonReader = new JsonReader(new StringReader(s)); jsonReader.setLenient(true); Info info = gson.fromJson(jsonReader,Info.class); //set fields using this.somefield = info.getsomefield() } 解决方法
从技术上讲,reduce的输出类型应与输入类型相同.如果使用组合器,合并器的输出被送入减速器,则必须如此.
(编辑:李大同) 【声明】本站内容均来自网络,其相关言论仅代表作者个人观点,不代表本站立场。若无意侵犯到您的权利,请及时与联系站长删除相关内容! |