Use Cargo Workspaces in generated project to reduce compilation times

[AnthonyMichaelTDM]

Currently, running the build scripts takes an unnecessary amount of time because all the scripts are treated like separate binaries but have the same dependencies as the backend, meaning that all the dependencies of the backend are compiled twice if you, for example, run cargo fullstack.

https://doc.rust-lang.org/book/ch14-03-cargo-workspaces.html

[AnthonyMichaelTDM]

For example, running cargo backend will compile every dependency (upwards of 400-500 crates) once, execute the backend.rs binary, which will then compile every dependency again to run the server when it runs cargo watch.

Despite the fact that backend.rs has no dependencies.

[Wulf]

Do you mind timing this on a fresh project with and without this workspace change? Also, could you share the output you get in your terminal? I'm not sure I understand the problem

[AnthonyMichaelTDM]

oh sure, I'll get on that

[AnthonyMichaelTDM]

Methodology notes

• Command used to generate project: cargo run --bin create-rust-app -- create -c -d postgres -b actix-web test-project
• ran cargo clean between each test, not doing so does speed up some compilations because build-artifacts are reused by cargo (but not cargo-watch)
• used time cargo build --bin <binary> to measure runtime
• for the "with workspaces" tests: edited/added manifests (included at end of this comment), moved files around, cleaned up dependencies, split dsync.rs and tsync.rs into binary and libary crates, etc.

Key Point:

• all the binaries that run the backend (backend.rs, fullstack.rs) also have to compile it, and cargo-watch doesn't re-use the compilation artifacts from compiling backend/fullstack (whichever was used), this greatly increases the total time spent compiling things

Results

without workspaces

• backend
  • Crates compiled: 447
  • time information: cargo build --bin backend 1087.23s user 106.52s system 645% cpu 3:05.07 total
• dsync
  • Crates compiled: 447
  • time information: cargo build --bin dsync 1090.79s user 109.00s system 571% cpu 3:30.06 total
• frontend
  • Crates compiled: 447
  • time information: cargo build --bin frontend 1052.26s user 106.69s system 607% cpu 3:10.74 total
• fullstack
  • Crates compiled: 447
  • time information: these are all compiling the same dependencies, so they're all taking around 3 minutes
• tsync
  • Crates compiled: 447
  • time information: these are all compiling the same dependencies, so they're all taking around 3 minutes
• the backend server (test-project)
  • Crates compiled: 447
  • time information: these are all compiling the same dependencies, so they're all taking around 3 minutes

with workspaces

• backend
  • Crates compiled: 1
  • time information: cargo build --bin backend 1.16s user 0.41s system 105% cpu 1.479 total
• dsync
  • Crates compiled: 47 (needs the dsync library)
  • time information: cargo build --bin dsync 105.46s user 11.05s system 465% cpu 25.034 total
• frontend
  • Crates compiled: 404 (needs the create-rust-app libary)
  • time information: cargo build --bin frontend 993.41s user 99.09s system 598% cpu 3:02.55 total
• fullstack
  • Crates compiled: 422 (needs create-rust-app and some other stuff)
  • time information: cargo build --bin fullstack 992.44s user 101.83s system 579% cpu 3:08.87 total
• tsync
  • Crates compiled: 40
  • time information: cargo build --bin tsync 65.34s user 8.26s system 325% cpu 22.612 total
• the backend server (test-project)
  • Crates compiled: 417
  • time information: cargo build --bin test-project 958.04s user 94.59s system 534% cpu 3:16.95 total

Manifests

Click to expand

./Cargo.toml (root manifest)

[workspace]
members = [
  ".cargo/bin/dsync",
  ".cargo/bin/tsync",
  ".cargo/bin/fullstack",
  ".cargo/bin/frontend",
  ".cargo/bin/backend",
  "backend",
]

[workspace.dependencies]
dsync = "0"
tsync = "1"
create-rust-app = { version = "9", default-features = false, features = [
  "plugin_dev",
  "database_postgres",
  "backend_actix-web",
] }

[profile.dev]
debug-assertions = true

./backend/Cargo.toml

[[bin]]
name = "test-project"
path = "src/main.rs"

[dependencies]
actix-files = "0.6.0"
actix-http = "3.0.0"
actix-web = "4.0.1"
futures-util = "0.3.21"
create-rust-app = { workspace = true }
serde_json = "1.0.79"
tsync = { workspace = true }

[dependencies.chrono]
features = ["serde"]
version = "0.4.19"

[dependencies.diesel]
default-features = false
features = ["postgres", "r2d2", "chrono"]
version = "2.0.0-rc.1"

[dependencies.serde]
features = ["derive"]
version = "1.0.133"

[dependencies.tokio]
features = ["full"]
version = "1"

[package]
default-run = "test-project"
edition = "2021"
name = "test-project"
publish = false
version = "0.1.0"

./.cargo/bin/backend/Cargo.toml

[package]
name = "backend"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[[bin]]
name = "backend"
path = "src/backend.rs"
bench = false
test = false

[dependencies]

./.cargo/bin/dsync/Cargo.toml

[package]
name = "dsync"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[[bin]]
name = "dsync"
path = "src/dsync.rs"
bench = false
test = false

[lib]
name = "dsync_lib"
path = "src/lib.rs"
bench = false
test = false

[dependencies]
dsync = {workspace = true}

./.cargo/bin/frontend/Cargo.toml

[package]
name = "frontend"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[[bin]]
name = "frontend"
path = "src/frontend.rs"
bench = false
test = false

[dependencies]
create-rust-app = { workspace = true }

./.cargo/bin/fullstack/Cargo.toml

[package]
name = "fullstack"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[[bin]]
name = "fullstack"
path = "src/fullstack.rs"
bench = false
test = false

[dependencies]
create-rust-app = { workspace = true }
dsync = { path = "../dsync" }
tsync = { path = "../tsync" }

./.cargo/bin/tsync/Cargo.toml

[package]
name = "tsync"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[[bin]]
name = "tsync"
path = "src/tsync.rs"
bench = false
test = false

[lib]
name = "tsync_lib"
path = "src/lib.rs"
bench = false
test = false

[dependencies]
tsync = {workspace = true}

File tree of "using workspaces" test

Click to expand

command used: tree --dirsfirst ./ .cargo

./
├── backend
│   ├── src
│   │   ├── mail
│   │   │   ├── example.rs
│   │   │   └── mod.rs
│   │   ├── models
│   │   │   └── mod.rs
│   │   ├── services
│   │   │   ├── mod.rs
│   │   │   └── todo.rs
│   │   ├── main.rs
│   │   └── schema.rs
│   ├── views
│   │   └── index.html
│   └── Cargo.toml
├── frontend
│   ├── bundles
│   │   └── index.tsx
│   ├── public
│   │   ├── images
│   │   │   ├── favicon.ico
│   │   │   ├── logo192.png
│   │   │   └── logo512.png
│   │   ├── pwa.json
│   │   └── robots.txt
│   ├── src
│   │   ├── containers
│   │   │   ├── Home.tsx
│   │   │   └── Todo.tsx
│   │   ├── hooks
│   │   │   └── useQueryParam.ts
│   │   ├── images
│   │   │   ├── logo2.svg
│   │   │   ├── logo.svg
│   │   │   └── plus.svg
│   │   ├── types
│   │   │   ├── global.d.ts
│   │   │   ├── plugin-dev.d.ts
│   │   │   ├── rust.d.ts
│   │   │   └── untyped-modules.d.ts
│   │   ├── App.css
│   │   ├── App.tsx
│   │   ├── dev.tsx
│   │   ├── reportWebVitals.js
│   │   └── setupDevelopment.tsx
│   ├── tests
│   │   └── example.spec.ts
│   ├── package.json
│   ├── package-lock.json
│   ├── playwright.config.ts
│   ├── tsconfig.json
│   └── vite.config.ts
├── migrations
│   ├── 00000000000000_diesel_initial_setup
│   │   ├── down.sql
│   │   └── up.sql
│   ├── 00000000000001_utc
│   │   ├── down.sql
│   │   └── up.sql
│   └── 00000000000002_todos
│       ├── down.sql
│       └── up.sql
├── build.rs
├── Cargo.lock
├── Cargo.toml
├── diesel.toml
├── README.md
└── tree
.cargo
├── admin
│   └── dist
│       ├── admin.144d5956.css
│       ├── admin.144d5956.css.map
│       ├── admin.67d0b27a.js
│       ├── admin.67d0b27a.js.map
│       └── admin.html
├── bin
│   ├── backend
│   │   ├── src
│   │   │   └── backend.rs
│   │   └── Cargo.toml
│   ├── dsync
│   │   ├── src
│   │   │   ├── dsync.rs
│   │   │   └── lib.rs
│   │   └── Cargo.toml
│   ├── frontend
│   │   ├── src
│   │   │   └── frontend.rs
│   │   └── Cargo.toml
│   ├── fullstack
│   │   ├── src
│   │   │   └── fullstack.rs
│   │   └── Cargo.toml
│   └── tsync
│       ├── src
│       │   ├── lib.rs
│       │   └── tsync.rs
│       └── Cargo.toml
└── config

[Wulf]

@AnthonyMichaelTDM thanks a bundle for the write-up and investigation

[Wulf]

We merged this in, right @AnthonyMichaelTDM ?

[AnthonyMichaelTDM]

yeah, pretty sure

[AnthonyMichaelTDM]

We still don't generate the projects in workspaces, but do support them at the library level