4 回答

• 5

  鉴于您正在寻找性能，似乎应该建议一个 data.table 解决方案 .

  有一个函数 rbindlist 这是 same 但比 do.call(rbind, list) 快得多

  library(data.table)
  X <- replicate(50000, data.table(a=rnorm(5), b=1:5), simplify=FALSE)
  system.time(rbindlist.data.table <- rbindlist(X))
  ##  user  system elapsed 
  ##  0.00    0.01    0.02
  

  对于 data.frame 的列表，它也快 very

  Xdf <- replicate(50000, data.frame(a=rnorm(5), b=1:5), simplify=FALSE)
  
  system.time(rbindlist.data.frame <- rbindlist(Xdf))
  ##  user  system elapsed 
  ##  0.03    0.00    0.03
  

  为了比较

  system.time(docall <- do.call(rbind, Xdf))
  ##  user  system elapsed 
  ## 50.72    9.89   60.88
  

  还有一些适当的基准测试

  library(rbenchmark)
  benchmark(rbindlist.data.table = rbindlist(X), 
             rbindlist.data.frame = rbindlist(Xdf),
             docall = do.call(rbind, Xdf),
             replications = 5)
  ##                   test replications elapsed    relative user.self sys.self 
  ## 3               docall            5  276.61 3073.444445    264.08     11.4 
  ## 2 rbindlist.data.frame            5    0.11    1.222222      0.11      0.0 
  ## 1 rbindlist.data.table            5    0.09    1.000000      0.09      0.0
  

  并反对@ JoshuaUlrich的解决方案

  benchmark(use.rbl.dt  = rbl.dt(X), 
            use.rbl.ju  = rbl.ju (Xdf),
            use.rbindlist =rbindlist(X) ,
            replications = 5)
  
  ##              test replications elapsed relative user.self 
  ## 3  use.rbindlist            5    0.10      1.0      0.09
  ## 1     use.rbl.dt            5    0.10      1.0      0.09
  ## 2     use.rbl.ju            5    0.33      3.3      0.31
  

  我不确定你真的需要使用 as.data.frame ，因为 data.table 继承了类 data.frame

  回复于 2024-04-29T22:07:19+08:00
• 17

  rbind.data.frame 做了很多你不需要的检查 . 如果你只做你想做的事情，这应该是一个非常快速的转变 .

  # Use data from Josh O'Brien's post.
  set.seed(21)
  X <- replicate(50000, data.frame(a=rnorm(5), b=1:5), simplify=FALSE)
  system.time({
  Names <- names(X[[1]])  # Get data.frame names from first list element.
  # For each name, extract its values from each data.frame in the list.
  # This provides a list with an element for each name.
  Xb <- lapply(Names, function(x) unlist(lapply(X, `[[`, x)))
  names(Xb) <- Names          # Give Xb the correct names.
  Xb.df <- as.data.frame(Xb)  # Convert Xb to a data.frame.
  })
  #    user  system elapsed 
  #   3.356   0.024   3.388 
  system.time(X1 <- do.call(rbind, X))
  #    user  system elapsed 
  # 169.627   6.680 179.675
  identical(X1,Xb.df)
  # [1] TRUE
  

  受data.table答案的启发，我决定试着让它更快 . 这是我更新的解决方案，尝试保留复选标记 . ;-)

  # My "rbind list" function
  rbl.ju <- function(x) {
    u <- unlist(x, recursive=FALSE)
    n <- names(u)
    un <- unique(n)
    l <- lapply(un, function(N) unlist(u[N==n], FALSE, FALSE))
    names(l) <- un
    d <- as.data.frame(l)
  }
  # simple wrapper to rbindlist that returns a data.frame
  rbl.dt <- function(x) {
    as.data.frame(rbindlist(x))
  }
  
  library(data.table)
  if(packageVersion("data.table") >= '1.8.2') {
    system.time(dt <- rbl.dt(X))  # rbindlist only exists in recent versions
  }
  #    user  system elapsed 
  #    0.02    0.00    0.02
  system.time(ju <- rbl.ju(X))
  #    user  system elapsed 
  #    0.05    0.00    0.05 
  identical(dt,ju)
  # [1] TRUE
  

  回复于 2024-04-29T22:07:19+08:00
• 17

  您观察到所花费的时间随着data.frames的数量呈指数增长，这表明将_2521964分为两个阶段会加快速度 .

  这个简单的实验似乎证实了这是一条非常富有成效的道路：

  ## Make a list of 50,000 data.frames
  X <- replicate(50000, data.frame(a=rnorm(5), b=1:5), simplify=FALSE)
  
  ## First, rbind together all 50,000 data.frames in a single step
  system.time({
      X1 <- do.call(rbind, X)
  })
  #    user  system elapsed 
  # 137.08   57.98  200.08 
  
  
  ## Doing it in two stages cuts the processing time by >95%
  ##   - In Stage 1, 100 groups of 500 data.frames are rbind'ed together
  ##   - In Stage 2, the resultant 100 data.frames are rbind'ed
  system.time({
      X2 <- lapply(1:100, function(i) do.call(rbind, X[((i*500)-499):(i*500)]))
      X3 <- do.call(rbind, X2)
  }) 
  #    user  system elapsed 
  #    6.14    0.05    6.21 
  
  
  ## Checking that the results are the same
  identical(X1, X3)
  # [1] TRUE
  

  回复于 2024-04-29T22:07:19+08:00
• 8

  您有一个data.frames列表，每个都有一行 . 如果可以将每个转换为矢量，我认为这会加快速度 .

  但是，假设它们需要是data.frames，我'll create a function with code borrowed from Dominik'的回答是Can rbind be parallelized in R?

  do.call.rbind <- function (lst) {
    while (length(lst) > 1) {
      idxlst <- seq(from = 1, to = length(lst), by = 2)
      lst <- lapply(idxlst, function(i) {
        if (i == length(lst)) {
          return(lst[[i]])
        }
        return(rbind(lst[[i]], lst[[i + 1]]))
      })
    }
    lst[[1]]
  }
  

  我已经使用这个功能好几个月了，并发现它比 do.call(rbind, ...) 更快，使用更少的内存[免责声明是我几乎只在 xts 对象上使用它]

  每个data.frame拥有的行越多，列表所包含的元素越多，此功能就越有用 .

  如果您有100,000个数字向量的列表， do.call(rbind, ...) 会更好 . 如果你有10亿的长度列表，这将更好 .

  > df <- lapply(1:10000, function(x) data.frame(x = sample(21, 21)))
  > library(rbenchmark)
  > benchmark(a=do.call(rbind, df), b=do.call.rbind(df))
  test replications elapsed relative user.self sys.self user.child sys.child
  1    a          100 327.728 1.755965   248.620   79.099          0         0
  2    b          100 186.637 1.000000   181.874    4.751          0         0
  

  随着您增加列表的长度，相对加速将呈指数级增长 .

  回复于 2024-04-29T22:07:19+08:00