Support for Artifactory

[brianrepko]

There is a similar issue logged for packrat as well - rstudio/packrat#583

Artifactory does not store archived versions in a CRAN-like way - see

• https://www.jfrog.com/jira/browse/RTFACT-17392
• https://www.jfrog.com/jira/browse/RTFACT-21144

The last issue from Artifactory looks like they have changed this in their code but I'm not clear on what version this fix is included in.

I thought about re-implementing this PR from remotes - r-lib/remotes#441 - thinking that will also fix renv but now I don't believe that that is true (since renv only Suggests remotes)

Am I correct in reading the code that install.R eventually calls r_cmd_install on packages?
And the code that works out the correct location of the package is in retreive.R?

Not sure if it makes sense to change renv_retrieve_repos_archive to also try with
repo <- file.path(repo, "src/contrib/Archive", record$Package, record$Version)

or if renv_retrieve_repos_impl should do the test - this is only broken for Archive'd versions...

But am I correct in assuming that a fix to remotes won't do anything for renv?

[brianrepko]

Also not sure if it makes sense to able to update the Repository in renv.lock to be flagged as a "versioned-archives" Repository - then we could build the url properly based on the Repository settings

[kevinushey]

But am I correct in assuming that a fix to remotes won't do anything for renv?

That's correct; renv is independent of remotes. I think the solution for renv would be to try alternate locations here:

renv/R/retrieve.R

Lines 553 to 565 in d10f636

	renv_retrieve_repos_archive <- function(record) {
	name <- sprintf("%s_%s.tar.gz", record$Package, record$Version)
	for (repo in getOption("repos")) {
	repo <- file.path(repo, "src/contrib/Archive", record$Package)
	status <- catch(renv_retrieve_repos_impl(record, "source", name, repo))
	if (identical(status, TRUE))
	return(TRUE)
	}
	return(FALSE)
	}

But we'd need to make this "cheap"; e.g. renv should have a way of quietly detecting whether a file is available at a particular URI. I can try exploring this later.

[kevinushey]

Is there a way to tell if a CRAN repository is an Artifactory repository (e.g. checking the headers or something from a web request)?

[brianrepko]

I just hit both our base URL (the URL that is in the repos option) as well as src/contrib under that and there are three headers that tell you if it is an Artifactory repository.

X-Artifactory-Id
X-Artifactory-Node-Id
Server

Here are the values for ours (sort of)

X-Artifactory-Id: 601b67c93bea52ad53d8fbf7a2d69fed56e4dbe9
X-Artifactory-Node-Id: hostname
Server: Artifactory/6.20.1

The Node-Id header may be only used if setup as a clustered application.
But X-Artifactory-Id should always be there for an Artifactory CRAN repo.
And Server header can be parsed for a version should this only be needed for versions under a given version.

That makes this super cheap - you can then know that a Repository has CRAN structure or Artifactory structure
and based on that just build the repo variable above appropriately.

[kevinushey]

Awesome!

For now, I've implemented something that should work as well -- renv now tries to figure out the structure used for archived packages in the repository, and selects + saves the appropriate one. Would you be able to give the development version of renv a try and let me know if it works for you?

[brianrepko]

I will but not until I return home after the first of the year

[brianrepko]

@kevinushey I'm having trouble getting this tested - some of my initial tests had some issues so I wanted to start completely clean - however I'm on a system that is R 3.6.1 and cannot use usethis 2.0.0 because of the change from git2r to gert (libgit2).

Here is what I'm doing

mkdir -p $ROOT/renvtest/lib/R/3.6
mkdir -p $ROOT/renvtest/cache

edit ~/.Renviron
# set R_USER_LIB to $ROOT/renvtest/lib/R/3.6
# set RENV_PATHS_CACHE to $ROOT/renvtest/cache

edit ~/.Rprofile
# set CRAN mirror and WORK_CRAN (Artifactory)

R
# verify
.libPaths() 
getOption("repos")

# setup library
install.packages('remotes')
remotes::install_github('rstudio/renv')
remotes::install_version(package='usethis',version='1.6.3',upgrade=T)

# create project
usethis::create_project("<$ROOT>/renvtest/project", rstudio=T, open=T)

# init renv
renv::init(bare=T)
* Project '$ROOT/renvtest/project' loaded. [renv 0.12.3-72]
* renv activated -- please restart the R session.
> q('no')

cd project
R
# Bootstrapping renv 0.12.3-72 -----------------------------------------------
* Downloading renv 0.12.3-72 from GitHub ... FAILED
Error in bootstrap(version, libpath) : failed to download renv 0.12.3-72

So I can't seem to init a clean project with a clean library.
I have to use bare=T because otherwise renv tries to upgrade/install usethis 2.0.0

[brianrepko]

My earlier test (yesterday) was with 0.12.3-71 - not sure if I should try to install that instead? renv::upgrade always gives me an rdb corrupt file issue

[brianrepko]

aha - looking at bootstrap.R I needed to set a GITHUB_PAT...

[brianrepko]

Nuts - I need the RENV_PATHS_ROOT changed - not just cache... redoing this whole thing

[brianrepko]

so all worked great on this - I'll upload the "script" that I used to test it tomorrow but no problems at all.
I'm not totally sure if / where you store the originating URL - my only concern on that is when the package version is current, the URL is different than when it is no longer current (src/contrib vs src/contrib/Archive) but I literally don't see where this is kept.

[brianrepko]

here is my "memory" of what I did - again it all looks good to me

mkdir -p $ROOT/renvtest/lib/R/3.6
mkdir -p $ROOT/renvtest/renv

edit ~/.Renviron
# set R_USER_LIB to $ROOT/renvtest/lib/R/3.6
# set RENV_PATHS_ROOT to $ROOT/renvtest/renv
# set GITHUB_PAT

edit ~/.Rprofile
# set CRAN mirror and WORK_CRAN (Artifactory)

cd $ROOT/renvtest
R
  # verify
  .libPaths() 
  getOption("repos")

  # setup library
  install.packages('remotes')
  remotes::install_github('rstudio/renv')
  remotes::install_version(package='usethis',version='1.6.3',upgrade=T)

  # create project
  usethis::create_project("<root>/renvtest/project", rstudio=T, open=T)

  # init renv
  renv::init(bare=T)
  * Project '$ROOT/renvtest/project' loaded. [renv 0.12.3-72]
  * renv activated -- please restart the R session.
  > q('no')

# reload project
cd $ROOT/renvtest/project
R
  # Bootstrapping renv 0.12.3-72 -----------------------------------------------
  * Downloading renv 0.12.3-72 from GitHub ... Done!
  * Installing renv 0.12.3-72 ... Done!
  * Successfully installed and loaded renv 0.12.3-72.
  * Project '$ROOT/renvtest/project' loaded. [renv 0.12.3-72]

  # add the artifactory repo
  local({
    r <- getOption("repos")
    r["WORK_CRAN"] <- "https://example.com/artifactory/CRAN"
    options(repos = r)
  })
  # PackageA - 1.0.0 is old, 1.1.0 is current
  # PackageB - 1.1.0 is old, 1.2.0 is current, depends on PackageA
  renv::install("PackageA@1.0.0")
  #Retrieving 'https://example.com/artifactory/CRAN/src/contrib/Archive/PackageA/1.0.0/PackageA_1.0.0.tar.gz' ...
  #... dependencies ...
  renv::install("PackageB@1.1.0")
  #Retrieving 'https://example.com/artifactory/CRAN/src/contrib/Archive/PackageB/1.1.0/PackageB_1.0.0.tar.gz' ...
  #... dependencies ...

  # create a project-dependencies.R file with
  # library(PackageA)
  # library(PackageB)

  # capture and work with A/B via renv
  renv::status()
  renv::snapshot()
  renv::update("PackageA")
  # 1.0.0 --> 1.1.0?
  # Retrieving 'https://example.com/artifactory/CRAN/src/contrib/PackageA/1.1.0/PackageA_1.1.0.tar.gz' ...
  renv::update("PackageB")
  # 1.1.0 --> 1.2.0?
  # Retrieving 'https://example.com/artifactory/CRAN/src/contrib/PackageB/1.2.0/PackageB_1.2.0.tar.gz' ...
  renv::snapshot()
  q('no')

[brianrepko]

I think we can close this issue - thank you @kevinushey

[kevinushey]

Awesome. Thanks!

For reference, here's where the "magic" happens:

renv/R/retrieve.R

Lines 576 to 621 in 05028b9

	renv_retrieve_repos_archive_path <- function(repo, record) {
	# allow users to provide a custom archive path for a record,
	# in case they're using a repository that happens to archive
	# packages with a different format than regular CRAN network
	# https://github.com/rstudio/renv/issues/602
	override <- getOption("renv.retrieve.repos.archive.path")
	if (is.function(override)) {
	result <- override(repo, record)
	if (!is.null(result))
	return(result)
	}
	# if we already know the format of the repository, use that
	if (exists(repo, envir = `_renv_repos_archive`)) {
	formatter <- get(repo, envir = `_renv_repos_archive`)
	root <- formatter(repo, record)
	return(root)
	}
	# otherwise, try determining the archive paths with a couple
	# custom locations, and cache the version that works for the
	# associated repository
	formatters <- list(
	function(repo, record) {
	with(record, file.path(repo, "src/contrib/Archive", Package))
	},
	function(repo, record) {
	with(record, file.path(repo, "src/contrib/Archive", Package, Version))
	}
	)
	name <- renv_retrieve_repos_archive_name(record, "source")
	for (formatter in formatters) {
	root <- formatter(repo, record)
	url <- file.path(root, name)
	if (renv_download_available(url)) {
	assign(repo, formatter, envir = `_renv_repos_archive`)
	return(root)
	}
	}
	}

Basically, we try both archive paths; if one of those succeeds, then we "save" that as the preferred path to use when attempting to install packages from the archives.