scala – Spark源代码：如何理解withScope方法

发布时间：2020-12-16 09:55:42 所属栏目：安全来源：网络整理

导读：我无法理解withScope方法的功能(实际上,我真的不知道RDDOperationScope类的含义) 特别是,withScope方法的参数列表中(body：= T)的含义是什么： private[spark] def withScope[T]( sc: SparkContext,name: String,allowNesting: Boolean,ignoreParent: Boolea

我无法理解withScope方法的功能(实际上,我真的不知道RDDOperationScope类的含义)

特别是,withScope方法的参数列表中(body：=> T)的含义是什么：

private[spark] def withScope[T](
  sc: SparkContext,name: String,allowNesting: Boolean,ignoreParent: Boolean)(body: => T): T = {
// Save the old scope to restore it later
val scopeKey = SparkContext.RDD_SCOPE_KEY
val noOverrideKey = SparkContext.RDD_SCOPE_NO_OVERRIDE_KEY
val oldScopeJson = sc.getLocalProperty(scopeKey)
val oldScope = Option(oldScopeJson).map(RDDOperationScope.fromJson)
val oldNoOverride = sc.getLocalProperty(noOverrideKey)
try {
  if (ignoreParent) {
    // Ignore all parent settings and scopes and start afresh with our own root scope
    sc.setLocalProperty(scopeKey,new RDDOperationScope(name).toJson)
  } else if (sc.getLocalProperty(noOverrideKey) == null) {
    // Otherwise,set the scope only if the higher level caller allows us to do so
    sc.setLocalProperty(scopeKey,new RDDOperationScope(name,oldScope).toJson)
  }
  // Optionally disallow the child body to override our scope
  if (!allowNesting) {
    sc.setLocalProperty(noOverrideKey,"true")
  }
  body
} finally {
  // Remember to restore any state that was modified before exiting
  sc.setLocalProperty(scopeKey,oldScopeJson)
  sc.setLocalProperty(noOverrideKey,oldNoOverride)
}
}

您可以使用以下链接找到源代码：
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDDOperationScope.scala

谁能帮我？谢谢,我很长时间都很困惑.

解决方法

以下代码可以帮助您

object TestWithScope {
    def withScope(func: => String) = {
        println("withscope")
        func
    }

    def bar(foo: String) = withScope {
        println("Bar: " + foo)
        "BBBB"
    }

    def main(args: Array[String]): Unit = {
        println(bar("AAAA"));
    }
}

可能的输出

withscope
Bar: AAAA
BBBB

（编辑：李大同）

【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!