Struggling to use future with profvis

[DavisVaughan]

@HenrikBengtsson try to run the following

library(future)
library(furrr)

# 50,000 instances of `mtcars`
xs <- rep(list(mtcars), 50000)

# Two workers
plan(multisession, workers = 2)

profvis::profvis({
  future_map(xs, identity)
})

or

library(future)
library(future.apply)

# 50,000 instances of `mtcars`
xs <- rep(list(mtcars), 50000)

# Two workers
plan(multisession, workers = 2)

profvis::profvis({
  future_lapply(xs, identity)
})

For me, both of these return something like

profvis: code exited with error: Caught an unexpected error of class FutureError when trying to launch future () on backend of class MultisessionFutureBackend. The reason was: MultisessionFuture () failed to launch future on cluster RichSOCKnode futureverse/furrr#1 (PID 58154 on localhost ‘localhost’). The reason reported was ‘error writing to connection’. Post-mortem diagnostic: A process with this PID exists, which suggests that the localhost worker is still alive. The total size of the 5 globals exported is 89.19 MiB The three largest globals are ‘...furrr_chunk_args’ (89.19 MiB of class ‘list’), ‘...furrr_fn’ (185 bytes of class ‘function’) and ‘...furrr_dots’ (31 bytes of class ‘list’) [future (20fb4acd5a0a2950ebd05810b60d69d9-1); on 20fb4acd5a0a2950ebd05810b60d69d9@davisvaughan-01HQ<58109>]

Any ideas?

[HenrikBengtsson]

Here's a smaller example with just future and a single parallel worker;

library(future)
plan(multisession, workers = I(1)) # a single parallel worker

xs_small <- rep(list(mtcars), times = 1000)
p <- profvis::profvis({ f <- future(xs_small); v <- value(f) })
class(p)
#> [1] "profvis"         "suppress_viewer" "htmlwidget"

xs_large <- rep(list(mtcars), times = 10000)
p <- profvis::profvis({ f <- future(xs_large); v <- value(f) })

#> profvis: code exited with error:
#> Caught an unexpected error of class FutureError when trying to launch
#> future (<unnamed-9>) on backend of class
#> MultisessionFutureBackend. The reason was: MultisessionFuture
#> (<unnamed-9>) failed to launch future on cluster RichSOCKnode futureverse/furrr#1 (PID
#> 365166 on localhost ‘localhost’). The reason reported was ‘error
#> writing to connection’. Post-mortem diagnostic: A process with this
#> PID exists, which suggests that the localhost worker is still
#> alive. The total size of the 1 globals exported is 35.68 MiB There is
#> one global: ‘xs_large’ (35.68 MiB of class ‘list’) [future <unnamed-9>
#> (f4749383b2c8824e3940c2e02d0a27f8-9); on
#> f4749383b2c8824e3940c2e02d0a27f8@hb-x1-2023<364466>]

I'm not sure what's going on, but one guess is that there's some limitation in the underlying parallel PSOCK worker (particularly "The reason reported was ‘error writing to connection’"), because it works with, e.g.

plan(future.callr::callr, workers = I(1))

and

plan(future.mirai::mirai_multisession, workers = I(1))

[DavisVaughan]

@HenrikBengtsson can you transfer this to future then? i dont have write access there so it wont let me transfer to there, but you should be able to.

I feel like maybe this is worth documenting as a gotcha at least?

[HenrikBengtsson]

Here's a reproducible example using only the parallel package;

$ R --vanilla --quiet
> cl <- parallel::makeCluster(1L)
xs_large <- rep(list(mtcars), times = 10000)
p <- profvis::profvis({ y <- parallel::parLapply(cl = cl, X = xs_large, fun = identity) })
profvis: code exited with error:
error writing to connection
> 

At this point, the parent-worker connection is broken:

> parallel::parLapply(cl = cl, X = 0L, fun = identity)
Error in unserialize(node$con) : error reading from connection

I'll transfer this to parallelly changed my find, to future to make sure it gets documented there.

I feel like maybe this is worth documenting as a gotcha at least?

Yeah, maybe it's worth adding a 'Known issues' vignette to future, or possibly to https://future.futureverse.org/articles/future-4-issues.html.

[agila5]

Dear all, I'm writing you this comment since I'm facing the same issue as @DavisVaughan. Based on my understanding, these types of errors represent a limitation of parallel computing in R when working with large objects. Am I correct? I'm sorry, but I'm not sure I understand @HenrikBengtsson's last comment. Thanks for your help if possible!