java – 为什么地图转换很窄？

发布时间：2020-12-15 04:34:33 所属栏目：Java 来源：网络整理

导读：我知道RDD的窄变换和宽变换之间的区别.我的问题是哪些参数可以证明map,flatmap是窄的？为什么这些转变不广泛？有关RDD学习材料的任何建议表示赞赏. 解决方法 Here is one explanation – Narrow dependency: RDD operations like map,union,filter can opera

我知道RDD的窄变换和宽变换之间的区别.我的问题是哪些参数可以证明map,flatmap是窄的？为什么这些转变不广泛？有关RDD学习材料的任何建议表示赞赏.

解决方法

Here is one explanation

– Narrow dependency: RDD operations like map,union,filter can operate on a single partition and map the data of that partition to resulting single partition. These kind of operations which maps data from one to one partition are referred as Narrow operations. Narrow operations doesn’t required to distribute the data across the partitions.

– Wide dependency: RDD operations like groupByKey,distinct,join may require to map the data across the partitions in new RDD. These kind of operations which maps data from one to many partitions are referred as Wide operations

想到它的另一种方法是这样.子RDD的任何行仅取决于父RDD的1行.由于每个子行都可以指向它所依赖的1父行,因此存在一个狭窄的依赖关系.

（编辑：李大同）

【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!