scala – Spark Streaming Window Operation
发布时间:2020-12-16 18:54:40 所属栏目:安全 来源:网络整理
导读:以下是在30秒的窗口大小和10秒的幻灯片大小上获得单词计数的简单代码. import org.apache.spark.SparkConfimport org.apache.spark.streaming._import org.apache.spark.streaming.StreamingContext._import org.apache.spark.api.java.function._import org
以下是在30秒的窗口大小和10秒的幻灯片大小上获得单词计数的简单代码.
import org.apache.spark.SparkConf import org.apache.spark.streaming._ import org.apache.spark.streaming.StreamingContext._ import org.apache.spark.api.java.function._ import org.apache.spark.streaming.api._ import org.apache.spark.storage.StorageLevel val ssc = new StreamingContext(sc,Seconds(5)) // read from text file val lines0 = ssc.textFileStream("test") val words0 = lines0.flatMap(_.split(" ")) // read from socket val lines1 = ssc.socketTextStream("localhost",9999,StorageLevel.MEMORY_AND_DISK_SER) val words1 = lines1.flatMap(_.split(" ")) val words = words0.union(words1) val wordCounts = words.map((_,1)).reduceByKeyAndWindow(_ + _,Seconds(30),Seconds(10)) wordCounts.print() ssc.checkpoint(".") ssc.start() ssc.awaitTermination() 但是,我从这一行得到错误: val wordCounts = words.map((_,Seconds(10)) .特别是从_ _.错误是 51: error: missing parameter type for expanded function ((x$2,x$3) => x$2.$plus(x$3)) 谁能告诉我问题是什么?谢谢! 解决方法
这非常容易修复,只需明确类型即可.
val wordCounts = words.map((_,1)).reduceByKeyAndWindow((a:Int,b:Int)=> a b,秒(30),秒(10)) 在this answer中解释了scala无法推断出这种情况的类型的原因 (编辑:李大同) 【声明】本站内容均来自网络,其相关言论仅代表作者个人观点,不代表本站立场。若无意侵犯到您的权利,请及时与联系站长删除相关内容! |