计算Spark DataFrame上的Spearman相关性

3 回答

• 0

  您只需选择感兴趣的列，提取值并计算统计数据：

  import sqlContext.implicits._
  import org.apache.spark.mllib.stat.Statistics
  
  // Generate some random data
  scala.util.Random.setSeed(1)
  val df = sc.parallelize(g.sample(1000).zip(g.sample(1000))).toDF("x", "y")
  
  
  // Select columns and extract values
  val rddX = df.select($"x").rdd.map(_.getDouble(0))
  val rddY = df.select($"y").rdd.map(_.getDouble(0))
  
  val correlation: Double = Statistics.corr(rddX, rddY, "spearman")
  

  回复于 2024-04-30T06:42:38+08:00
• 1

  你应该可以做这样的事情

  val firstRDD: RDD[Double] = yourDF.select("field1").map(row => row.getDouble(0))
  val secondRDD: RDD[Double] = yourDF.select("field2").map(row => row.getDouble(0))
  val corr = Statistics.corr(firstRDD, secondRDD, "spearman")
  

  回复于 2024-04-30T06:42:38+08:00
• 0

  在scala中，您可以使用以下内容 . 我假设你正确地从rdd创建了数据帧 .

  df.select(corr($"Column1",$"Column2")).show()
  
  +----------------------+
  |corr(Column1, Column2)|
  +----------------------+
  | 0.40962234447942202  |
  +----------------------+
  

  回复于 2024-04-30T06:42:38+08:00