scala – Spark 2.2非法模式组件：XXX java.lang.IllegalArgumen

发布时间：2020-12-16 09:26:35 所属栏目：安全来源：网络整理

导读：我正在尝试从Spark 2.1升级到2.2.当我尝试将数据帧读取或写入某个位置(CSV或JSON)时,我收到此错误： Illegal pattern component: XXXjava.lang.IllegalArgumentException: Illegal pattern component: XXXat org.apache.commons.lang3.time.FastDatePrinter.

我正在尝试从Spark 2.1升级到2.2.当我尝试将数据帧读取或写入某个位置(CSV或JSON)时,我收到此错误：

Illegal pattern component: XXX
java.lang.IllegalArgumentException: Illegal pattern component: XXX
at org.apache.commons.lang3.time.FastDatePrinter.parsePattern(FastDatePrinter.java:282)
at org.apache.commons.lang3.time.FastDatePrinter.init(FastDatePrinter.java:149)
at org.apache.commons.lang3.time.FastDatePrinter.<init>(FastDatePrinter.java:142)
at org.apache.commons.lang3.time.FastDateFormat.<init>(FastDateFormat.java:384)
at org.apache.commons.lang3.time.FastDateFormat.<init>(FastDateFormat.java:369)
at org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:91)
at org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:88)
at org.apache.commons.lang3.time.FormatCache.getInstance(FormatCache.java:82)
at org.apache.commons.lang3.time.FastDateFormat.getInstance(FastDateFormat.java:165)
at org.apache.spark.sql.catalyst.json.JSONOptions.<init>(JSONOptions.scala:81)
at org.apache.spark.sql.catalyst.json.JSONOptions.<init>(JSONOptions.scala:43)
at org.apache.spark.sql.execution.datasources.json.JsonFileFormat.inferSchema(JsonFileFormat.scala:53)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$7.apply(DataSource.scala:177)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$7.apply(DataSource.scala:177)
at scala.Option.orElse(Option.scala:289)
at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:176)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:366)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:333)
at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:279)

我没有为dateFormat设置默认值,所以我不知道它来自哪里.

spark.createDataFrame(objects.map((o) => MyObject(t.source,t.table,o.partition,o.offset,d)))
    .coalesce(1)
    .write
    .mode(SaveMode.Append)
    .partitionBy("source","table")
    .json(path)

我仍然得到这个错误：

import org.apache.spark.sql.{SaveMode,SparkSession}
val spark = SparkSession.builder.appName("Spark2.2Test").master("local").getOrCreate()
import spark.implicits._
val agesRows = List(Person("alice",35),Person("bob",10),Person("jill",24))
val df = spark.createDataFrame(agesRows).toDF();

df.printSchema
df.show

df.write.mode(SaveMode.Overwrite).csv("my.csv")

这是架构：
????根
?| – name：string(nullable = true)
?| – age：long(nullable = false)

解决方法

我找到了答案.

timestampFormat的默认值是yyyy-MM-dd’T’HH：mm：ss.SSSXXX,这是一个非法参数.在编写数据帧时需要设置它.

修复方法是将其更改为ZZ,其中包括时区.

df.write
.option("timestampFormat","yyyy/MM/dd HH:mm:ss ZZ")
.mode(SaveMode.Overwrite)
.csv("my.csv")

（编辑：李大同）

【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!