Testing @ WRDS

[andf1]

Hello!

My name is Andrew and I'm a developer at WRDS: I'll be taking a look at your module.

Thanks for drafting this up and sharing with us! Cursorily: this looks fantastic and we very much appreciate the effort! It looks like you have take the basic boilerplate of functionality we provided in our Python wrds module and expanded it usefully with your own ideas and tooling here in this R package -- very cool!

Let me take some time to test your package against a few things on our end to ensure maximum compatibility with our data, various DB schemas / views, and supporting R development infra (on the WRDS Cloud and for our R Studio Server deploy). Once I've had time to test, I'll reply here with any comments or ideas, or next steps.

Thanks again -- be in touch soon.

[iangow]

@statzhero

It seems that this works fine as a drop-in replacement for db <- dbConnect(RPostgres::Postgres()) in the PostgreSQL tabs in my book. Basically, just do db <- wrds_connect() and nothing else changes. (It's better than a simple replacement, because the code in the book relies on users having set up environment variables as described here.)

[statzhero]

Thank you, yes that's exactly right. I would say my two ambitions are: 1) to help people deal with large data (lazy loading, warnings, maybe parquet features down the road) and 2) provide additional "quality-of-life" features (such as filling in missing SIC fields).

[iangow]

Thank you, yes that's exactly right. I would say my two ambitions are: 1) to help people deal with large data (lazy loading, warnings, maybe parquet features down the road) and 2) provide additional "quality-of-life" features (such as filling in missing SIC fields).

I think "quality of life" and reproducibility go together in some ways. People aren't going to adopt reproducible workflows if it isn't (very) easy to do so.

For the book, I think it makes sense to keep things a bit more explicit (e.g., filter(indfmt == "INDL", datafmt == "STD", consol == "C", popsrc == "D") or coalesce(sich, sic)), even if later in life readers will use something that takes care of the details for them.

For quality of life and reproducibility, you should try out the "parquet" approach from the book. Basically, get the WRDS data using db2pq (e.g., wrds_update("funda", "comp") and then db <- dbConnect(duckdb::duckdb()) and funda <- farr::load_parquet(db, "funda", "comp").

My sense is that most researchers have project-specific versions of data (including WRDS data) somewhere on their hard drive (e.g., a data directory in the project folder). Put DATA_DIR="data" in .Renviron in your RStudio project and db2pq (run from inside R at least) and load_parquet() will use that. So this become a very easy way to manage data in a (fairly) reproducible way. (Outside researchers may need to use WRDS versions of the data to reproduce, but it's very easy to do that.)

If you use comp.g_secd at all, you might like this note. TL;DR: Representative query that takes 6 minutes in SAS or 1 minute in WRDS PostgreSQL takes 0.2 seconds with local parquet file.

(In practice, going direct from a local funda.parquet file might produce more reproducible workflows than you'd get from a custom compustat.parquet created from a query against the WRDS PostgreSQL server.)