Skip to content

adding collect argument to TblSqlSource execute_query method #199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
asb2111 wants to merge 4 commits into
base: main
Choose a base branch
from
+55 −11
Open

adding collect argument to TblSqlSource execute_query method #199

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
984511a
adding collect argument to TblSqlSource execute_query method
Jan 20, 2026
14e67a3
Changing to default to collect = TRUE
Jan 21, 2026
4e28e8a
Fixing test errors
Jan 21, 2026
79dfb60
Update pkg-r/tests/testthat/test-TblSqlSource.R
asb2111 Jan 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions pkg-r/R/TblSqlSource.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,10 +117,18 @@ TblSqlSource <- R6::R6Class(
#' Execute a SQL query and return results
#'
#' @param query SQL query string to execute
#' @return A data frame containing query results
execute_query = function(query) {
#' @param collect If `TRUE` (default), collects the results into a local data frame
#' using [dplyr::collect()]. If `FALSE`, returns a lazy SQL
#' tibble.
#' @return A data frame (if `collect = TRUE`) or a lazy SQL tibble (if
#' `collect = FALSE`)
execute_query = function(query, collect = TRUE) {
sql_query <- self$prep_query(query)
dplyr::tbl(private$conn, dplyr::sql(sql_query))
result <- dplyr::tbl(private$conn, dplyr::sql(sql_query))
if (collect) {
result <- dplyr::collect(result)
}
result
},

#' @description
Expand Down
9 changes: 7 additions & 2 deletions pkg-r/man/TblSqlSource.Rd
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion pkg-r/tests/testthat/helper-fixtures.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,8 @@ local_tbl_sql_source <- function(

DBI::dbWriteTable(conn, table_name, data, overwrite = TRUE)
tbl <- dplyr::tbl(conn, table_name)
tbl <- tbl_transform(tbl)
tbl <- tbl_transform(tbl) |>
dplyr::compute("test_table")
Copy link
Contributor

@gadenbuie gadenbuie Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this addition required to get the tests to pass? Could you explain why it's needed?

Copy link
Author

@asb2111 asb2111 Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was a CTE table name recursion error occurring, this basically 'reset' the name of the table

Copy link
Contributor

@gadenbuie gadenbuie Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah okay, that's a little concerning... was that error happening for you during testing? I can't repro it personally. Can you give some more details (e.g. sessioninfo::session_info() and which test was causing the error)?

Copy link
Author

@asb2111 asb2111 Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this was actually occurring prior to any of my changes. Here is my session info:`

─ Session info ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.5.0 (2025-04-11)
 os       Red Hat Enterprise Linux 9.5 (Plow)
 system   x86_64, linux-gnu
 ui       RStudio
 language (EN)
 collate  en_US.UTF-8
 ctype    en_US.UTF-8
 tz       America/New_York
 date     2026-01-21
 rstudio  2024.12.0+467.pro1 Kousa Dogwood (server)
 pandoc   2.14.0.3 @ /usr/bin/pandoc
 quarto   1.6.42 @ /data/apps/quarto/1.6.42/bin/quarto

─ Packages ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 package     * version date (UTC) lib source
 blob          1.2.4   2023-03-17 [1] RSPM (R 4.5.0)
 brio          1.1.5   2024-04-24 [1] RSPM (R 4.5.0)
 bslib         0.9.0   2025-01-30 [1] RSPM
 cachem        1.1.0   2024-05-16 [1] RSPM
 callr         3.7.6   2024-03-25 [1] RSPM
 cli           3.6.5   2025-04-23 [1] RSPM (R 4.5.0)
 commonmark    1.9.5   2025-03-17 [1] RSPM (R 4.5.0)
 coro          1.1.0   2024-11-05 [1] CRAN (R 4.5.0)
 crayon        1.5.3   2024-06-20 [1] RSPM
 DBI           1.2.3   2024-06-02 [1] RSPM (R 4.5.0)
 dbplyr        2.5.0   2024-03-19 [1] RSPM
 desc          1.4.3   2023-12-10 [1] RSPM (R 4.5.0)
 devtools      2.4.5   2022-10-11 [1] RSPM (R 4.5.0)
 diffobj       0.3.6   2025-04-21 [1] RSPM (R 4.5.0)
 digest        0.6.37  2024-08-19 [1] RSPM
 dplyr         1.1.4   2023-11-17 [1] RSPM (R 4.5.0)
 duckdb        1.3.1   2025-06-23 [1] RSPM (R 4.5.0)
 ellipsis      0.3.2   2021-04-29 [1] RSPM (R 4.5.0)
 ellmer        0.4.0   2025-11-15 [1] CRAN (R 4.5.0)
 evaluate      1.0.4   2025-06-18 [1] RSPM (R 4.5.0)
 fastmap       1.2.0   2024-05-15 [1] RSPM
 fs            1.6.6   2025-04-12 [1] RSPM
 generics      0.1.4   2025-05-09 [1] RSPM (R 4.5.0)
 glue          1.8.0   2024-09-30 [1] RSPM (R 4.5.0)
 htmltools     0.5.8.1 2024-04-04 [1] RSPM
 htmlwidgets   1.6.4   2023-12-06 [1] RSPM (R 4.5.0)
 httpuv        1.6.16  2025-04-16 [1] RSPM (R 4.5.0)
 httr2         1.2.2   2025-12-08 [1] CRAN (R 4.5.0)
 jquerylib     0.1.4   2021-04-26 [1] RSPM
 jsonlite      2.0.0   2025-03-27 [1] RSPM
 knitr         1.50    2025-03-16 [1] RSPM
 later         1.4.2   2025-04-08 [1] RSPM (R 4.5.0)
 lifecycle     1.0.4   2023-11-07 [1] RSPM (R 4.5.0)
 magrittr      2.0.3   2022-03-30 [1] RSPM (R 4.5.0)
 memoise       2.0.1   2021-11-26 [1] RSPM
 mime          0.13    2025-03-17 [1] RSPM
 miniUI        0.1.2   2025-04-17 [1] RSPM (R 4.5.0)
 pillar        1.10.2  2025-04-05 [1] RSPM (R 4.5.0)
 pkgbuild      1.4.8   2025-05-26 [1] RSPM (R 4.5.0)
 pkgconfig     2.0.3   2019-09-22 [1] RSPM (R 4.5.0)
 pkgload       1.4.0   2024-06-28 [1] RSPM (R 4.5.0)
 processx      3.8.6   2025-02-21 [1] RSPM
 profvis       0.4.0   2024-09-20 [1] RSPM (R 4.5.0)
 promises      1.3.3   2025-05-29 [1] RSPM (R 4.5.0)
 ps            1.9.1   2025-04-12 [1] RSPM
 purrr         1.0.4   2025-02-05 [1] RSPM (R 4.5.0)
 R6            2.6.1   2025-02-15 [1] RSPM (R 4.5.0)
 rappdirs      0.3.3   2021-01-31 [1] RSPM
 Rcpp          1.0.14  2025-01-12 [1] RSPM
 remotes       2.5.0   2024-03-17 [1] RSPM (R 4.5.0)
 rlang         1.1.6   2025-04-11 [1] RSPM (R 4.5.0)
 roxygen2      7.3.3   2025-09-03 [1] CRAN (R 4.5.0)
 rprojroot     2.0.4   2023-11-05 [1] RSPM (R 4.5.0)
 rstudioapi    0.17.1  2024-10-22 [1] RSPM
 S7            0.2.0   2024-11-07 [1] RSPM (R 4.5.0)
 sass          0.4.10  2025-04-11 [1] RSPM
 sessioninfo   1.2.3   2025-02-05 [1] RSPM (R 4.5.0)
 shiny         1.11.0  2025-06-24 [1] RSPM (R 4.5.0)
 stringi       1.8.7   2025-03-27 [1] RSPM (R 4.5.0)
 stringr       1.5.1   2023-11-14 [1] RSPM (R 4.5.0)
 testthat      3.2.3   2025-01-13 [1] RSPM (R 4.5.0)
 tibble        3.3.0   2025-06-08 [1] RSPM (R 4.5.0)
 tidyselect    1.2.1   2024-03-11 [1] RSPM (R 4.5.0)
 urlchecker    1.0.1   2021-11-30 [1] RSPM (R 4.5.0)
 usethis       3.1.0   2024-11-26 [1] RSPM (R 4.5.0)
 utf8          1.2.6   2025-06-08 [1] RSPM (R 4.5.0)
 vctrs         0.6.5   2023-12-01 [1] RSPM (R 4.5.0)
 waldo         0.6.1   2024-11-07 [1] RSPM (R 4.5.0)
 whisker       0.4.1   2022-12-05 [1] RSPM (R 4.5.0)
 withr         3.0.2   2024-10-28 [1] RSPM (R 4.5.0)
 xfun          0.52    2025-04-02 [1] RSPM
 xml2          1.3.8   2025-03-14 [1] RSPM
 xtable        1.8-4   2019-04-21 [1] RSPM (R 4.5.0)

 [1] /home/alexander.breskin/R/4.5.0
 [2] /opt/R/4.5.0/lib/R/library

Here is the specific test failure:

Error (test-TblSqlSource.R:62:5): TblSqlSource with transformed tbl (CTE mode): works with filtered tbl
Error in `db_query_fields.DBIConnection(con, ...)`: Can't query fields.
i Using SQL: SELECT * FROM ( WITH test_table AS ( SELECT test_table.* FROM test_table WHERE ("value" > 20.0) ) SELECT * FROM test_table ) q01 WHERE (0 = 1)
Caused by error in `dbSendQuery()`:
! rapi_prepare: Failed to prepare query SELECT *
FROM (
WITH test_table AS (
SELECT test_table.*
FROM test_table
WHERE ("value" > 20.0)
)
SELECT * FROM test_table
) q01
WHERE (0 = 1)
Error: Binder Error: Circular reference to CTE "test_table", There are two possible solutions. 
1. use WITH RECURSIVE to use recursive CTEs. 
2. If you want to use the TABLE name "test_table" the same as the CTE name, please explicitly add "SCHEMA" before table name. You can try "main.test_table" (main is the duckdb default schema)

I tried the second suggestion, using 'main.test_table' in the query, but that didn't work because it overrode the >20 filter.

Copy link
Contributor

@gadenbuie gadenbuie Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh interesting... querychat has a pattern of wanting the user to provide a table_name, e.g. querychat(mtcars, "my_mtcars"), but for tibbles we try to figure out the name of the underlying table. This failure seems to indicate that something in going wrong and querychat thinks the table names are different, even though they clearly aren't. Or maybe for this particular test we can't figure out the underlying table name and the test needs to use a different table name that isn't test_table.


TblSqlSource$new(tbl, table_name)
}
Expand Down
40 changes: 35 additions & 5 deletions pkg-r/tests/testthat/test-TblSqlSource.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,13 @@ describe("TblSqlSource$new()", {
})
})

it("returns lazy tibble from execute_query()", {
it("returns lazy tibble from execute_query() when collect = FALSE", {
source <- local_tbl_sql_source()

result <- source$execute_query("SELECT * FROM test_table WHERE value > 25")
result <- source$execute_query(
"SELECT * FROM test_table WHERE value > 25",
collect = FALSE
)
expect_s3_class(result, "tbl_sql")
expect_s3_class(result, "tbl_lazy")

Expand All @@ -35,6 +38,31 @@ describe("TblSqlSource$new()", {
expect_equal(collected$value, c(30, 40, 50))
})

it("returns lazy tibble from execute_query() when collect = FALSE", {
source <- local_tbl_sql_source()

result <- source$execute_query(
"SELECT * FROM test_table WHERE value > 25",
collect = FALSE
)
expect_s3_class(result, "tbl_sql")
expect_s3_class(result, "tbl_lazy")
})

it("returns data frame from execute_query() when collect = TRUE", {
source <- local_tbl_sql_source()

result <- source$execute_query(
"SELECT * FROM test_table WHERE value > 25",
collect = TRUE
)
expect_s3_class(result, "data.frame")
expect_false(inherits(result, "tbl_sql"))
expect_false(inherits(result, "tbl_lazy"))
expect_equal(nrow(result), 3)
expect_equal(result$value, c(30, 40, 50))
})

it("returns data frame from test_query()", {
source <- local_tbl_sql_source()

Expand All @@ -59,7 +87,7 @@ describe("TblSqlSource with transformed tbl (CTE mode)", {
)

# CTE should be used since tbl is transformed
result <- source$execute_query("SELECT * FROM test_table")
result <- source$execute_query("SELECT * FROM test_table", collect = FALSE)
collected <- dplyr::collect(result)
expect_equal(nrow(collected), 3)
expect_true(all(collected$value > 20))
Expand Down Expand Up @@ -191,7 +219,8 @@ describe("TblSqlSource edge cases - Category B: Column Naming Issues", {
# SELECT with explicit duplicate column names from JOIN
# DuckDB allows duplicate names but tibble rejects them on collect
result <- source$execute_query(
"SELECT table_a.id, table_b.id FROM table_a JOIN table_b ON table_a.id = table_b.id"
"SELECT table_a.id, table_b.id FROM table_a JOIN table_b ON table_a.id = table_b.id",
collect = FALSE
)
expect_error(
dplyr::collect(result),
Expand Down Expand Up @@ -272,7 +301,8 @@ describe("TblSqlSource edge cases - Category B: Column Naming Issues", {
# SELECT * from JOIN produces duplicate 'id' columns
# tibble rejects duplicate names on collect
result <- source$execute_query(
"SELECT * FROM table_a JOIN table_b ON table_a.id = table_b.id"
"SELECT * FROM table_a JOIN table_b ON table_a.id = table_b.id",
collect = FALSE
)
expect_error(
dplyr::collect(result),
Expand Down