Spark - scala：随机将RDD /拆分RDD分成两个随机部分

2 回答

• 20

  我找到了一种简单快捷的方法来分割数组：

  val Array(f1,f2) = data.randomSplit(Array(0.97, 0.03))
  

  它将使用提供的权重拆分数据 .

  回复于 2024-04-25T11:46:40+08:00
• 6

  你应该使用 randomSplit 方法：

  def randomSplit(weights: Array[Double], seed: Long = Utils.random.nextLong): Array[RDD[T]]
  
  // Randomly splits this RDD with the provided weights.
  // weights for splits, will be normalized if they don't sum to 1
  // returns split RDDs in an array
  

  这是它在火花1.0中的implementation：

  def randomSplit(weights: Array[Double], seed: Long = Utils.random.nextLong): Array[RDD[T]] = {
      val sum = weights.sum
      val normalizedCumWeights = weights.map(_ / sum).scanLeft(0.0d)(_ + _)
      normalizedCumWeights.sliding(2).map { x =>
         new PartitionwiseSampledRDD[T, T](this, new BernoulliSampler[T](x(0), x(1)),seed)
      }.toArray
  }
  

  回复于 2024-04-25T11:46:40+08:00