加入收藏 | 设为首页 | 会员中心 | 我要投稿 李大同 (https://www.lidatong.com.cn/)- 科技、建站、经验、云计算、5G、大数据,站长网!
当前位置: 首页 > 百科 > 正文

Beam学习笔记(3):Flink Streaming Pipeline Translator

发布时间:2020-12-14 01:37:37 所属栏目:百科 来源:网络整理
导读:FlinkStreamingPipelineTranslator 1 所有的可用Translator: // here you can find all the available translators. static { TRANSLATORS .put (Read .Bounded .class ,new BoundedReadSourceTranslator()) ; TRANSLATORS .put (Read .Unbounded .class ,n

FlinkStreamingPipelineTranslator

1 所有的可用Translator:

// here you can find all the available translators.
  static {
    TRANSLATORS.put(Read.Bounded.class,new BoundedReadSourceTranslator());
    TRANSLATORS.put(Read.Unbounded.class,new UnboundedReadSourceTranslator());
    TRANSLATORS.put(TextIO.Write.Bound.class,new TextIOWriteBoundStreamingTranslator());

    TRANSLATORS.put(ParDo.MultiOutput.class,new ParDoStreamingTranslator());
    TRANSLATORS.put(
        SplittableParDo.ProcessElements.class,new SplittableProcessElementsStreamingTranslator());
    TRANSLATORS.put(
        SplittableParDo.GBKIntoKeyedWorkItems.class,new GBKIntoKeyedWorkItemsTranslator());


    TRANSLATORS.put(Window.Assign.class,new WindowAssignTranslator());
    TRANSLATORS.put(Flatten.PCollections.class,new FlattenPCollectionTranslator());
    TRANSLATORS.put(
        FlinkStreamingViewOverrides.CreateFlinkPCollectionView.class,new CreateViewStreamingTranslator());

    TRANSLATORS.put(Reshuffle.class,new ReshuffleTranslatorStreaming());
    TRANSLATORS.put(GroupByKey.class,new GroupByKeyTranslator());
    TRANSLATORS.put(Combine.PerKey.class,new CombinePerKeyTranslator());
  }

2 applyStreamingTransform:

基本逻辑:将PTransform关联到streamingContext

// create the applied PTransform on the streamingContext streamingContext.setCurrentTransform(node.toAppliedPTransform());
    typedTranslator.translateNode(typedTransform,streamingContext);

12种类型的translator多态实现了StreamTransformTranslator.translateNode(T transform,FlinkStreamingTranslationContext context);方法。

(编辑:李大同)

【声明】本站内容均来自网络,其相关言论仅代表作者个人观点,不代表本站立场。若无意侵犯到您的权利,请及时与联系站长删除相关内容!

    推荐文章
      热点阅读