将列值替换为小于其自身的其他列值的数量

2 回答

• 0

  您可以使用窗口进行排序然后添加行号或另一种方式转换为rdd然后排序最后使用zipWithIndex . 使用窗口：

  from pyspark.sql import functions as F
  from pyspark.sql.window import Window
  
  window = Window \
               .orderBy(F.col('salary'))
  df \
     .withColumn('salary', F.dense_rank().over(window))
  

  回复于 2024-05-17T18:20:58+08:00
• 0

  一种有效的方法是使用如下的窗口函数 .
  按工资订购窗口，并使用当前行之前的所有行 .

  from pyspark.sql import Window
  import pyspark.sql.functions as F
  
  # You study all the rows before the current one. -1 to avoid counting current row
  w = Window.orderBy('salary').rowsBetween(Window.unboundedPreceding,Window.currentRow-1)
  
  # Count salary occurences on the window : salary below current salary
  results = df.withColumn('rank_order',F.count('salary').over(w))
  results.show()
  
  +---+------+----------+
  | id|salary|rank_order|
  +---+------+----------+
  |  2|   500|         0|
  |  3|  1500|         1|
  |  1|  2000|         2|
  +---+------+----------+
  

  回复于 2024-05-17T18:20:58+08:00