|
42 | 42 |
|
43 | 43 | ## Setting up a development environment |
44 | 44 |
|
45 | | -Start by [forking duckdb-python](https://github.com/duckdb/duckdb-python/fork) into |
46 | | -a personal repository. |
47 | | - |
48 | | -After forking the duckdb-python repo we recommend you clone your fork as follows: |
49 | | -```shell |
50 | | -git clone --recurse-submodules $REPO_URL |
51 | | -git remote add upstream https://github.com/duckdb/duckdb-python.git |
52 | | -git fetch --all |
53 | | -``` |
54 | | - |
55 | | -... or, if you have already cloned your fork: |
56 | | -```shell |
57 | | -git submodule update --init --recursive |
58 | | -git remote add upstream https://github.com/duckdb/duckdb-python.git |
59 | | -git fetch --all |
60 | | -``` |
61 | | - |
62 | | -Two things to be aware of when cloning this repository: |
63 | | -* DuckDB is vendored as a git submodule and needs to be initialized during or after cloning duckdb-python. |
64 | | -* Currently, for DuckDB to determine its version while building, it depends on the local availability of its tags. |
65 | | - |
66 | | -After forking the duckdb-python repo we recommend you clone your fork as follows: |
67 | | - |
68 | | -### Submodule update hook |
69 | | - |
70 | | -If you'll be switching between branches that are have the submodule set to different refs, then make your life |
71 | | -easier and add the git hooks in the .githooks directory to your local config: |
72 | | -```shell |
73 | | -git config --local core.hooksPath .githooks/ |
74 | | -``` |
75 | | - |
76 | | -### Editable installs (general) |
77 | | - |
78 | | -It's good to be aware of the following when performing an editable install: |
79 | | - |
80 | | -- `uv sync` or `uv run [tool]` perform an editable install by default. We have |
81 | | - configured the project so that scikit-build-core will use a persistent build-dir, but since the build itself |
82 | | - happens in an isolated, ephemeral environment, cmake's paths will point to non-existing directories. CMake itself |
83 | | - will be missing. |
84 | | -- You should install all development dependencies, and then build the project without build isolation, in two separate |
85 | | - steps. After this you can happily keep building and running, as long as you don't forget to pass in the |
86 | | - `--no-build-isolation` flag. |
87 | | - |
88 | | -```bash |
89 | | -# install all dev dependencies without building the project (needed once) |
90 | | -uv sync -p 3.11 --no-install-project |
91 | | -# build and install without build isolation |
92 | | -uv sync --no-build-isolation |
93 | | -``` |
94 | | - |
95 | | -### Editable installs (IDEs) |
96 | | - |
97 | | -If you're using an IDE then life is a little simpler. You install build dependencies and the project in the two |
98 | | -steps outlined above, and from that point on you can rely on e.g. CLion's cmake capabilities to do incremental |
99 | | -compilation and editable rebuilds. This will skip scikit-build-core's build backend and all of uv's dependency |
100 | | -management, so for "real" builds you better revert to the CLI. However, this should work fine for coding and debugging. |
101 | | - |
102 | | -## Day to day development |
103 | | - |
104 | | -After setting up the development environment, these are the most common tasks you'll be performing. |
105 | | - |
106 | | -### Tooling |
107 | | -This codebase is developed with the following tools: |
108 | | -- [Astral uv](https://docs.astral.sh/uv/) - for dependency management across all platforms we provide wheels for, |
109 | | - and for Python environment management. It will be hard to work on this codebase without having UV installed. |
110 | | -- [Scikit-build-core](https://scikit-build-core.readthedocs.io/en/latest/index.html) - the build backend for |
111 | | - building the extension. On the background, scikit-build-core uses cmake and ninja for compilation. |
112 | | -- [pybind11](https://pybind11.readthedocs.io/en/stable/index.html) - a bridge between C++ and Python. |
113 | | -- [CMake](https://cmake.org/) - the build system for both DuckDB itself and the DuckDB Python module. |
114 | | -- Cibuildwheel |
115 | | - |
116 | | -### Cleaning |
117 | | -```shell |
118 | | -uv cache clean |
119 | | -rm -rf build .venv uv.lock |
120 | | -``` |
121 | | - |
122 | | -### Running tests |
123 | | - |
124 | | - Run all pytests: |
125 | | -```bash |
126 | | -uv run --no-build-isolation pytest ./tests --verbose |
127 | | -``` |
128 | | - |
129 | | - Exclude the test/slow directory: |
130 | | -```bash |
131 | | -uv run --no-build-isolation pytest ./tests --verbose --ignore=./tests/slow |
132 | | -``` |
133 | | - |
134 | | -### Test coverage |
135 | | - |
136 | | - Run with coverage (during development you probably want to specify which tests to run): |
137 | | -```bash |
138 | | -COVERAGE=1 uv run --no-build-isolation coverage run -m pytest ./tests --verbose |
139 | | -``` |
140 | | - |
141 | | - The `COVERAGE` env var will compile the extension with `--coverage`, allowing us to collect coverage stats of C++ |
142 | | - code as well as Python code. |
143 | | - |
144 | | - Check coverage for Python code: |
145 | | -```bash |
146 | | -uvx coverage html -d htmlcov-python |
147 | | -uvx coverage report --format=markdown |
148 | | -``` |
149 | | - |
150 | | - Check coverage for C++ code (note: this will clutter your project dir with html files, consider saving them in some |
151 | | - other place): |
152 | | -```bash |
153 | | -uvx gcovr \ |
154 | | - --gcov-ignore-errors all \ |
155 | | - --root "$PWD" \ |
156 | | - --filter "${PWD}/src/duckdb_py" \ |
157 | | - --exclude '.*/\.cache/.*' \ |
158 | | - --gcov-exclude '.*/\.cache/.*' \ |
159 | | - --gcov-exclude '.*/external/.*' \ |
160 | | - --gcov-exclude '.*/site-packages/.*' \ |
161 | | - --exclude-unreachable-branches \ |
162 | | - --exclude-throw-branches \ |
163 | | - --html --html-details -o coverage-cpp.html \ |
164 | | - build/coverage/src/duckdb_py \ |
165 | | - --print-summary |
166 | | -``` |
167 | | - |
168 | | -### Typechecking, linting, style, and formatting |
169 | | - |
170 | | -- We're not running any mypy typechecking tests at the moment |
171 | | -- We're not running any Ruff / linting / formatting at the moment |
172 | | -- Follow the [Google Python styleguide](https://google.github.io/styleguide/pyguide.html) |
173 | | -- See the section on [Comments and Docstrings](https://google.github.io/styleguide/pyguide.html#s3.8-comments-and-docstrings) |
174 | | - |
175 | | -### Building wheels and sdists |
176 | | - |
177 | | -To build a wheel and sdist for your system and the default Python version: |
178 | | -```bash |
179 | | -uv build |
180 | | -```` |
181 | | - |
182 | | -To build a wheel for a different Python version: |
183 | | -```bash |
184 | | -# E.g. for Python 3.9 |
185 | | -uv build -p 3.9 |
186 | | -``` |
187 | | - |
188 | | -### Cibuildwheel |
189 | | - |
190 | | -You can run cibuildwheel locally for Linux. E.g. limited to Python 3.9: |
191 | | -```bash |
192 | | -CIBW_BUILD='cp39-*' uvx cibuildwheel --platform linux . |
193 | | -``` |
194 | | - |
195 | | -### Merging changes to pythonpkg from duckdb main |
196 | | - |
197 | | -1. Checkout main |
198 | | -2Identify the merge commits that brought in tags to main: |
199 | | -```bash |
200 | | -git log --graph --oneline --decorate main --simplify-by-decoration |
201 | | -``` |
202 | | - |
203 | | -3. Get the log of commits |
204 | | -```bash |
205 | | -git log --oneline 71c5c07cdd..c9254ecff2 -- tools/pythonpkg/ |
206 | | -``` |
207 | | - |
208 | | -4. Checkout v1.3-ossivalis |
209 | | -5. Get the log of commits |
210 | | -```bash |
211 | | -git log --oneline v1.3.0..v1.3.1 -- tools/pythonpkg/ |
212 | | -``` |
213 | | -git diff --name-status 71c5c07cdd c9254ecff2 -- tools/pythonpkg/ |
214 | | - |
215 | | -```bash |
216 | | -git log --oneline 71c5c07cdd..c9254ecff2 -- tools/pythonpkg/ |
217 | | -git diff --name-status <HASH_A> <HASH_B> -- tools/pythonpkg/ |
218 | | -``` |
219 | | - |
220 | | -## Versioning and Releases |
221 | | - |
222 | | -The DuckDB Python package versioning and release scheme follows that of DuckDB itself. This means that a `X.Y.Z[. |
223 | | -postN]` release of the Python package ships the DuckDB stable release `X.Y.Z`. The optional `.postN` releases ship the same stable release of DuckDB as their predecessors plus Python package-specific fixes and / or features. |
224 | | - |
225 | | -| Types | DuckDB Version | Resulting Python Extension Version | |
226 | | -|------------------------------------------------------------------------|----------------|------------------------------------| |
227 | | -| Stable release: DuckDB stable release | `1.3.1` | `1.3.1` | |
228 | | -| Stable post release: DuckDB stable release + Python fixes and features | `1.3.1` | `1.3.1.postX` | |
229 | | -| Nightly micro: DuckDB next micro nightly + Python next micro nightly | `1.3.2.devM` | `1.3.2.devN` | |
230 | | -| Nightly minor: DuckDB next minor nightly + Python next minor nightly | `1.4.0.devM` | `1.4.0.devN` | |
231 | | - |
232 | | -Note that we do not ship nightly post releases (e.g. we don't ship `1.3.1.post2.dev3`). |
233 | | -
|
234 | | -### Branch and Tag Strategy |
235 | | -
|
236 | | -We cut releases as follows: |
237 | | -
|
238 | | -| Type | Tag | How | |
239 | | -|----------------------|--------------|---------------------------------------------------------------------------------| |
240 | | -| Stable minor release | vX.Y.0 | Adding a tag on `main` | |
241 | | -| Stable micro release | vX.Y.Z | Adding a tag on a minor release branch (e.g. `v1.3-ossivalis`) | |
242 | | -| Stable post release | vX.Y.Z-postN | Adding a tag on a post release branch (e.g. `v1.3.1-post`) | |
243 | | -| Nightly micro | _not tagged_ | Combining HEAD of the _micro_ release branches of DuckDB and the Python package | |
244 | | -| Nightly minor | _not tagged_ | Combining HEAD of the _minor_ release branches of DuckDB and the Python package | |
245 | | -
|
246 | | -### Release Runbooks |
247 | | -
|
248 | | -We cut a new **stable minor release** with the following steps: |
249 | | -1. Create a PR on `main` to pin the DuckDB submodule to the tag of its current release. |
250 | | -1. Iff all tests pass in CI, merge the PR. |
251 | | -1. Manually start the release workflow with the hash of this commit, and the tag name. |
252 | | -1. Iff all goes well, create a new PR to let the submodule track DuckDB main. |
253 | | -
|
254 | | -We cut a new **stable micro release** with the following steps: |
255 | | -1. Create a PR on the minor release branch to pin the DuckDB submodule to the tag of its current release. |
256 | | -1. Iff all tests pass in CI, merge the PR. |
257 | | -1. Manually start the release workflow with the hash of this commit, and the tag name. |
258 | | -1. Iff all goes well, create a new PR to let the submodule track DuckDB's minor release branch. |
259 | | - |
260 | | -We cut a new **stable post release** with the following steps: |
261 | | -1. Create a PR on the post release branch to pin the DuckDB submodule to the tag of its current release. |
262 | | -1. Iff all tests pass in CI, merge the PR. |
263 | | -1. Manually start the release workflow with the hash of this commit, and the tag name. |
264 | | -1. Iff all goes well, create a new PR to let the submodule track DuckDB's minor release branch. |
265 | | -
|
266 | | -### Dynamic Versioning Integration |
267 | | -
|
268 | | -The package uses `setuptools_scm` with `scikit-build` for automatic version determination, and implements a custom |
269 | | -versioning scheme. |
270 | | -
|
271 | | -- **pyproject.toml configuration**: |
272 | | - ```toml |
273 | | - [tool.scikit-build] |
274 | | - metadata.version.provider = "scikit_build_core.metadata.setuptools_scm" |
275 | | - |
276 | | - [tool.setuptools_scm] |
277 | | - version_scheme = "duckdb_packaging._setuptools_scm_version:version_scheme" |
278 | | - ``` |
279 | | -
|
280 | | -- **Environment variables**: |
281 | | - - `MAIN_BRANCH_VERSIONING=0`: Use release branch versioning (patch increments) |
282 | | - - `MAIN_BRANCH_VERSIONING=1`: Use main branch versioning (minor increments) |
283 | | - - `OVERRIDE_GIT_DESCRIBE`: Override version detection |
| 45 | +See the [instructions on duckdb.org](https://duckdb.org/docs/stable/dev/building/python). |
0 commit comments