Skip to content

Commit f9b04b1

Browse files
committed
Devenv instructions linked
1 parent ea3bd59 commit f9b04b1

File tree

1 file changed

+1
-239
lines changed

1 file changed

+1
-239
lines changed

CONTRIBUTING.md

Lines changed: 1 addition & 239 deletions
Original file line numberDiff line numberDiff line change
@@ -42,242 +42,4 @@
4242

4343
## Setting up a development environment
4444

45-
Start by [forking duckdb-python](https://github.com/duckdb/duckdb-python/fork) into
46-
a personal repository.
47-
48-
After forking the duckdb-python repo we recommend you clone your fork as follows:
49-
```shell
50-
git clone --recurse-submodules $REPO_URL
51-
git remote add upstream https://github.com/duckdb/duckdb-python.git
52-
git fetch --all
53-
```
54-
55-
... or, if you have already cloned your fork:
56-
```shell
57-
git submodule update --init --recursive
58-
git remote add upstream https://github.com/duckdb/duckdb-python.git
59-
git fetch --all
60-
```
61-
62-
Two things to be aware of when cloning this repository:
63-
* DuckDB is vendored as a git submodule and needs to be initialized during or after cloning duckdb-python.
64-
* Currently, for DuckDB to determine its version while building, it depends on the local availability of its tags.
65-
66-
After forking the duckdb-python repo we recommend you clone your fork as follows:
67-
68-
### Submodule update hook
69-
70-
If you'll be switching between branches that are have the submodule set to different refs, then make your life
71-
easier and add the git hooks in the .githooks directory to your local config:
72-
```shell
73-
git config --local core.hooksPath .githooks/
74-
```
75-
76-
### Editable installs (general)
77-
78-
It's good to be aware of the following when performing an editable install:
79-
80-
- `uv sync` or `uv run [tool]` perform an editable install by default. We have
81-
configured the project so that scikit-build-core will use a persistent build-dir, but since the build itself
82-
happens in an isolated, ephemeral environment, cmake's paths will point to non-existing directories. CMake itself
83-
will be missing.
84-
- You should install all development dependencies, and then build the project without build isolation, in two separate
85-
steps. After this you can happily keep building and running, as long as you don't forget to pass in the
86-
`--no-build-isolation` flag.
87-
88-
```bash
89-
# install all dev dependencies without building the project (needed once)
90-
uv sync -p 3.11 --no-install-project
91-
# build and install without build isolation
92-
uv sync --no-build-isolation
93-
```
94-
95-
### Editable installs (IDEs)
96-
97-
If you're using an IDE then life is a little simpler. You install build dependencies and the project in the two
98-
steps outlined above, and from that point on you can rely on e.g. CLion's cmake capabilities to do incremental
99-
compilation and editable rebuilds. This will skip scikit-build-core's build backend and all of uv's dependency
100-
management, so for "real" builds you better revert to the CLI. However, this should work fine for coding and debugging.
101-
102-
## Day to day development
103-
104-
After setting up the development environment, these are the most common tasks you'll be performing.
105-
106-
### Tooling
107-
This codebase is developed with the following tools:
108-
- [Astral uv](https://docs.astral.sh/uv/) - for dependency management across all platforms we provide wheels for,
109-
and for Python environment management. It will be hard to work on this codebase without having UV installed.
110-
- [Scikit-build-core](https://scikit-build-core.readthedocs.io/en/latest/index.html) - the build backend for
111-
building the extension. On the background, scikit-build-core uses cmake and ninja for compilation.
112-
- [pybind11](https://pybind11.readthedocs.io/en/stable/index.html) - a bridge between C++ and Python.
113-
- [CMake](https://cmake.org/) - the build system for both DuckDB itself and the DuckDB Python module.
114-
- Cibuildwheel
115-
116-
### Cleaning
117-
```shell
118-
uv cache clean
119-
rm -rf build .venv uv.lock
120-
```
121-
122-
### Running tests
123-
124-
Run all pytests:
125-
```bash
126-
uv run --no-build-isolation pytest ./tests --verbose
127-
```
128-
129-
Exclude the test/slow directory:
130-
```bash
131-
uv run --no-build-isolation pytest ./tests --verbose --ignore=./tests/slow
132-
```
133-
134-
### Test coverage
135-
136-
Run with coverage (during development you probably want to specify which tests to run):
137-
```bash
138-
COVERAGE=1 uv run --no-build-isolation coverage run -m pytest ./tests --verbose
139-
```
140-
141-
The `COVERAGE` env var will compile the extension with `--coverage`, allowing us to collect coverage stats of C++
142-
code as well as Python code.
143-
144-
Check coverage for Python code:
145-
```bash
146-
uvx coverage html -d htmlcov-python
147-
uvx coverage report --format=markdown
148-
```
149-
150-
Check coverage for C++ code (note: this will clutter your project dir with html files, consider saving them in some
151-
other place):
152-
```bash
153-
uvx gcovr \
154-
--gcov-ignore-errors all \
155-
--root "$PWD" \
156-
--filter "${PWD}/src/duckdb_py" \
157-
--exclude '.*/\.cache/.*' \
158-
--gcov-exclude '.*/\.cache/.*' \
159-
--gcov-exclude '.*/external/.*' \
160-
--gcov-exclude '.*/site-packages/.*' \
161-
--exclude-unreachable-branches \
162-
--exclude-throw-branches \
163-
--html --html-details -o coverage-cpp.html \
164-
build/coverage/src/duckdb_py \
165-
--print-summary
166-
```
167-
168-
### Typechecking, linting, style, and formatting
169-
170-
- We're not running any mypy typechecking tests at the moment
171-
- We're not running any Ruff / linting / formatting at the moment
172-
- Follow the [Google Python styleguide](https://google.github.io/styleguide/pyguide.html)
173-
- See the section on [Comments and Docstrings](https://google.github.io/styleguide/pyguide.html#s3.8-comments-and-docstrings)
174-
175-
### Building wheels and sdists
176-
177-
To build a wheel and sdist for your system and the default Python version:
178-
```bash
179-
uv build
180-
````
181-
182-
To build a wheel for a different Python version:
183-
```bash
184-
# E.g. for Python 3.9
185-
uv build -p 3.9
186-
```
187-
188-
### Cibuildwheel
189-
190-
You can run cibuildwheel locally for Linux. E.g. limited to Python 3.9:
191-
```bash
192-
CIBW_BUILD='cp39-*' uvx cibuildwheel --platform linux .
193-
```
194-
195-
### Merging changes to pythonpkg from duckdb main
196-
197-
1. Checkout main
198-
2Identify the merge commits that brought in tags to main:
199-
```bash
200-
git log --graph --oneline --decorate main --simplify-by-decoration
201-
```
202-
203-
3. Get the log of commits
204-
```bash
205-
git log --oneline 71c5c07cdd..c9254ecff2 -- tools/pythonpkg/
206-
```
207-
208-
4. Checkout v1.3-ossivalis
209-
5. Get the log of commits
210-
```bash
211-
git log --oneline v1.3.0..v1.3.1 -- tools/pythonpkg/
212-
```
213-
git diff --name-status 71c5c07cdd c9254ecff2 -- tools/pythonpkg/
214-
215-
```bash
216-
git log --oneline 71c5c07cdd..c9254ecff2 -- tools/pythonpkg/
217-
git diff --name-status <HASH_A> <HASH_B> -- tools/pythonpkg/
218-
```
219-
220-
## Versioning and Releases
221-
222-
The DuckDB Python package versioning and release scheme follows that of DuckDB itself. This means that a `X.Y.Z[.
223-
postN]` release of the Python package ships the DuckDB stable release `X.Y.Z`. The optional `.postN` releases ship the same stable release of DuckDB as their predecessors plus Python package-specific fixes and / or features.
224-
225-
| Types | DuckDB Version | Resulting Python Extension Version |
226-
|------------------------------------------------------------------------|----------------|------------------------------------|
227-
| Stable release: DuckDB stable release | `1.3.1` | `1.3.1` |
228-
| Stable post release: DuckDB stable release + Python fixes and features | `1.3.1` | `1.3.1.postX` |
229-
| Nightly micro: DuckDB next micro nightly + Python next micro nightly | `1.3.2.devM` | `1.3.2.devN` |
230-
| Nightly minor: DuckDB next minor nightly + Python next minor nightly | `1.4.0.devM` | `1.4.0.devN` |
231-
232-
Note that we do not ship nightly post releases (e.g. we don't ship `1.3.1.post2.dev3`).
233-
234-
### Branch and Tag Strategy
235-
236-
We cut releases as follows:
237-
238-
| Type | Tag | How |
239-
|----------------------|--------------|---------------------------------------------------------------------------------|
240-
| Stable minor release | vX.Y.0 | Adding a tag on `main` |
241-
| Stable micro release | vX.Y.Z | Adding a tag on a minor release branch (e.g. `v1.3-ossivalis`) |
242-
| Stable post release | vX.Y.Z-postN | Adding a tag on a post release branch (e.g. `v1.3.1-post`) |
243-
| Nightly micro | _not tagged_ | Combining HEAD of the _micro_ release branches of DuckDB and the Python package |
244-
| Nightly minor | _not tagged_ | Combining HEAD of the _minor_ release branches of DuckDB and the Python package |
245-
246-
### Release Runbooks
247-
248-
We cut a new **stable minor release** with the following steps:
249-
1. Create a PR on `main` to pin the DuckDB submodule to the tag of its current release.
250-
1. Iff all tests pass in CI, merge the PR.
251-
1. Manually start the release workflow with the hash of this commit, and the tag name.
252-
1. Iff all goes well, create a new PR to let the submodule track DuckDB main.
253-
254-
We cut a new **stable micro release** with the following steps:
255-
1. Create a PR on the minor release branch to pin the DuckDB submodule to the tag of its current release.
256-
1. Iff all tests pass in CI, merge the PR.
257-
1. Manually start the release workflow with the hash of this commit, and the tag name.
258-
1. Iff all goes well, create a new PR to let the submodule track DuckDB's minor release branch.
259-
260-
We cut a new **stable post release** with the following steps:
261-
1. Create a PR on the post release branch to pin the DuckDB submodule to the tag of its current release.
262-
1. Iff all tests pass in CI, merge the PR.
263-
1. Manually start the release workflow with the hash of this commit, and the tag name.
264-
1. Iff all goes well, create a new PR to let the submodule track DuckDB's minor release branch.
265-
266-
### Dynamic Versioning Integration
267-
268-
The package uses `setuptools_scm` with `scikit-build` for automatic version determination, and implements a custom
269-
versioning scheme.
270-
271-
- **pyproject.toml configuration**:
272-
```toml
273-
[tool.scikit-build]
274-
metadata.version.provider = "scikit_build_core.metadata.setuptools_scm"
275-
276-
[tool.setuptools_scm]
277-
version_scheme = "duckdb_packaging._setuptools_scm_version:version_scheme"
278-
```
279-
280-
- **Environment variables**:
281-
- `MAIN_BRANCH_VERSIONING=0`: Use release branch versioning (patch increments)
282-
- `MAIN_BRANCH_VERSIONING=1`: Use main branch versioning (minor increments)
283-
- `OVERRIDE_GIT_DESCRIBE`: Override version detection
45+
See the [instructions on duckdb.org](https://duckdb.org/docs/stable/dev/building/python).

0 commit comments

Comments
 (0)