Allow for functions defined in namespace loaded via load_all to be passed as globals

[yogat3ch]

Continued from #480

It doesn't sound like there are any workaround at present for this kind of issue. Is it feasible as a feature request to strip namespaces from functions passed to globals and create those functions in the environment on the background process?

[yogat3ch]

I guess some functions could have a long chain of dependencies that are built into the namespace that would all have to be re-created in the background process...

[HenrikBengtsson]

Could you please make this post self-contained, so I and others don't have to backtrack it? There are already too many comments to mentally juggle and keep track off across the different issues and discussions. Plus, I'd like to remove those off-topic comments in #480. Hope you understand.

[yogat3ch]

Previous discussion:
We have a request related to this that would really improve the way future interfaces with the golem framework for Shiny apps.
At present, it's not possible to use any functions in code sent to a background process that aren't from an installed package or defined in the code that will run on the background processor. The golem framework uses a package namespace that is loaded with load_all, and thus the namespace is not available in the background process.

I've tried every possible method/workaround to make the functions available on the background process but the only method that works is to define every single function explicitly in the code that will be run on the background process.

Are there any workarounds you are aware of @HenrikBengtsson that we can use until this feature is added?

I feel like we've reached an impasse in trying to go asynchronous with future and are at the brink of writing off the attempt as sunk cost due to this limitation.

We have a request related to this that would really improve the way future interfaces with the golem framework for Shiny apps.
At present, it's not possible to use any functions in code sent to a background process that aren't from an installed package or defined in the code that will run on the background processor. The golem framework uses a package namespace that is loaded with load_all, and thus the namespace is not available in the background process.

I've tried every possible method/workaround to make the functions available on the background process but the only method that works is to define every single function explicitly in the code that will be run on the background process.

Are there any workarounds you are aware of @HenrikBengtsson that we can use until this feature is added?

I feel like we've reached an impasse in trying to go asynchronous with future and are at the brink of writing off the attempt as sunk cost due to this limitation.

Are there any workarounds you are aware of @HenrikBengtsson that we can use until this feature is added?

The workaround would be to set the option or the environment variable manually inside the future where the values are passed via globals, e.g.

myenv <- Sys.getenv("MYENV")
myopt <- getOption("myopt")
f <- future({
  Sys.setenv(MYENV = myenv)
  options(myopt = myopt)
  ...
})

Hi @HenrikBengtsson,
This doesn't work for functions because they retain the namespace that they were loaded in as the function environment, which is not available in the background process.
I'll try to better describe the issue.

We have a function defined in the golem package namespace, my_fun for short, that fetches data from a database. We have two helper sub-functions my_subfun1 & my_subfun2 that are used in my_fun. We want to run my_fun in a background process.
We create an asynchronous version of my_fun that has all the code to instantiate the database object appended to the top of the function body, then pass this to futureCall inside of an R6 method. This all works fine on other functions that call the database.

However, when my_fun is passed, due to the fact that it relies on my_subfun1 & 2, which are defined in the golem package namespace, we get an error that there is no package called <our golem namespace>.
I've tried all the methods highlighted in the discussion to overcome this limitation, but none seem to work.

This is futureCall of my_fun:

private[[p_obj]] <- future::futureCall(function (scenario_ids = 1) 
  {
    self <- list(db = function(host = "db_read_prod_endpoint", 
      user = "db_read_prod_user_id", password = "db_read_prod_pwd", 
      dbname = "crss_db_prod", port = 3310, ...) {
      db_opts <- purrr::map(c(host = host, user = user, 
        password = password), Sys.getenv)
      rlang::exec(pool::dbPool, RMySQL::MySQL(), !!!db_opts, 
        dbname = dbname, port = port, ...)
    })
    self$db <- self$db()
   out = dplyr::tbl(self$db, 'tbl_res_storage') |>
    dplyr::filter(scenario_id %in% scenario_ids) |>
    dplyr::filter(lubridate::month(date) == 12 & lubridate::day(date) == 31) |>
    dplyr::collect() |>
    grouped_median_min_max("storage") |>
    dplyr::mutate(popup_content = paste0("<b>", reservoir_name, "</b><br>",
                                         "Scenario ", scenario_id, "<br>", 
                                         "Median: ", number_formatter_for_labels(median), " AF", "<br>",
                                         "Max: ", number_formatter_for_labels(max), " AF", "<br>",
                                         "Min: ", number_formatter_for_labels(min), " AF", "<br>"
    )
    ) |>
    dplyr::mutate(group = paste("Scenario", scenario_id)) |>
    dplyr::distinct()
    return(out)
  }, seed = 1, args = .args, if (!missing(globals)) 
    globals = globals)

number_formatter_for_labels & grouped_median_min_max as my_subfun1 & 2.

[HenrikBengtsson]

There's a lot of things going on in your example requiring lots of efforts to digest before being able to get to the core of your problem. Is there any chance you can come up with a minimal, self-contained, reproducible example that illustrates what is not working right now.

[yogat3ch]

I'm not sure I know how to do so because the primary issue is a package loaded as a namespace via load_all that's not on the library path.
A reprex will require creating a package and loading it via load_all. I could describe the process necessary to create a reprex, would that work?

[yogat3ch]

Nevermind, I think I can create a reprex with use_this::create_package

[yogat3ch]

Ok, reprex is ready:

usethis::create_package("../crsspUtils", fields = list(
  Imports = c("future"),
  open = FALSE
))
# my_async_fun <- function(x = 1) {
#   future::futureCall(my_fun,
#                      args = list(x = x),
#                      globals = list(my_subfun = my_subfun))
# }
# 
# my_fun <- function(x) {
#   my_subfun(x)
# }
# 
# my_subfun <- function(x) {
#   x * 8
# }

write("my_fun <- function(x) {
  my_subfun(x)
}", "../crsspUtils/R/my_fun.R")

write("my_subfun <- function(x) {
  x * 8
}", "../crsspUtils/R/my_subfun.R")

write("my_async_fun <- function(x = 1) {
  future::futureCall(my_fun,
                     args = list(x = x),
                     globals = list(my_subfun = my_subfun))
}", "../crsspUtils/R/my_async_fun.R")

devtools::load_all("../crsspUtils")
cl <- parallelly::makeClusterPSOCK(
  2,
  port = 5302,
  autoStop = TRUE,
  rscript_libs = .libPaths()
)
future::plan(future::cluster, workers = cl)

future::value(my_async_fun())

Error: there is no package called 'crsspUtils'

SessionInfo

R version 4.1.0 (2021-05-18)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19042)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252 
[2] LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods  
[7] base     

other attached packages:
[1] crsspUtils_0.0.0.9000  crsspierViz_0.0.0.9000
[3] testthat_3.0.4.9000    shiny_1.7.0           
[5] lubridate_1.7.10      

loaded via a namespace (and not attached):
 [1] fs_1.5.0             reactable_0.2.3      usethis_2.0.1       
 [4] fontawesome_0.2.2    devtools_2.4.1.9000  rprojroot_2.0.2     
 [7] tools_4.1.0          tippy_1.0.0          bslib_0.3.0         
[10] utf8_1.2.2           R6_2.5.1             DBI_1.1.1           
[13] colorspace_2.0-2     withr_2.4.2          tidyselect_1.1.1    
[16] prettyunits_1.1.1    processx_3.5.2       RMySQL_0.10.22      
[19] leaflet_2.0.4.1      compiler_4.1.0       cli_3.0.1           
[22] xml2_1.3.2           desc_1.3.0           sass_0.4.0          
[25] scales_1.1.1         bs4Dash_2.0.2        callr_3.7.0         
[28] stringr_1.4.0        digest_0.6.27        pkgconfig_2.0.3     
[31] htmltools_0.5.2      parallelly_1.28.1    sessioninfo_1.1.1   
[34] attempt_0.3.1        dbplyr_2.1.1         fastmap_1.1.0       
[37] htmlwidgets_1.5.4    rlang_0.4.11         rstudioapi_0.13     
[40] jquerylib_0.1.4      generics_0.1.0       jsonlite_1.7.2      
[43] crosstalk_1.1.1      dplyr_1.0.7          config_0.3.1        
[46] magrittr_2.0.1       Rcpp_1.0.7           munsell_0.5.0       
[49] fansi_0.5.0          lifecycle_1.0.0      whisker_0.4         
[52] stringi_1.7.4        yaml_2.2.1           pkgbuild_1.2.0      
[55] blob_1.2.2           parallel_4.1.0       listenv_0.8.0       
[58] promises_1.2.0.1     crayon_1.4.1         echarts4r_0.4.2.9000
[61] knitr_1.33           ps_1.6.0             pillar_1.6.2        
[64] codetools_0.2-18     pkgload_1.2.1        glue_1.4.2          
[67] cicerone_1.0.5.9000  golem_0.3.1.9001     pool_0.1.6          
[70] remotes_2.4.0        renv_0.14.0          vctrs_0.3.8         
[73] httpuv_1.6.1         purrr_0.3.4          tidyr_1.1.3         
[76] future_1.22.1        reactR_0.4.4         assertthat_0.2.1    
[79] cachem_1.0.5         xfun_0.25            mime_0.11           
[82] xtable_1.8-4         roxygen2_7.1.1.9001  UU_0.0.0.9000       
[85] later_1.2.0          dockerfiler_0.1.3    tibble_3.1.5        
[88] memoise_2.0.0        globals_0.14.0       ellipsis_0.3.2      
[91] here_1.0.1