Pyspark转换复杂的Dataformat-Java 学习之路

用火花读一块镶木地板文件

df = spark.read.parquet("path_to_file")
df.show(2)

我的df包含

**Output**

+------+-----------------+
| col1 |       col2      |
+------+-----------------+
| "A1" |  {"x":1,"y":2}  |
+------+-----------------+
| "A2" |  {"z":3}        |
+------+-----------------+

我想将数据帧转换为喜欢

+------+------+------+------+
| col1 |  x   |  y   |  z   |
+------+------+------+------+
| "A1" |  1   |  2   | Null |
+------+------+------+------+
| "A2" | Null | Null |  3   |
+------+------+------+------+

初始数据帧的模式显示

DataFrame[col1: string, col2: string]

我在Windows 7机器上使用pyspark 2.3.2

1 回答

这个解决方案并不完全是你所要求的，但也许你并没有这么想 . 因此，请添加评论，如果它不适合您 .

> from pyspark.sql import functions as F
> from pyspark.sql.types import *

> df.show()
+----+-------------+
|col1|         col2|
+----+-------------+
|  A1|{"x":1,"y":2}|
|  A2|      {"z":3}|
+----+-------------+

> df.printSchema()
root
 |-- col1: string (nullable = true)
 |-- col2: string (nullable = true)

> df = df.withColumn(
..  "col2",
..  F.from_json("col2", MapType(StringType(),IntegerType()))
..)

> df.show()
+----+----------------+
|col1|            col2|
+----+----------------+
|  A1|[x -> 1, y -> 2]|
|  A2|        [z -> 3]|
+----+----------------+

> df.select(
..    "col1",
..    "col2.x",
..    "col2.y",
..    "col2.z",
..).show()
+----+----+----+----+
|col1|   x|   y|   z|
+----+----+----+----+
|  A1|   1|   2|null|
|  A2|null|null|   3|
+----+----+----+----+

回复于 2024-04-20T02:40:00+08:00

Pyspark转换复杂的Dataformat

1 回答

相关问题