ecospheres-dashboard-backend

A script that synchronizes data.gouv.fr's Ecospheres-related data to a Postgres database. This database is used to build dashboards.

Getting started

Spin off the database

docker compose up

Use the script

Install the required dependencies through requirements.txt or requirements-dev.txt

Export the env var needed for the script to find the database:

export DATABASE_URL=postgresql://postgres:postgres@localhost:5432/dashboard_backend

Bootstrap the database:

python cli.py init-db

Launch the main script:

python cli.py load --env=(demo|prod)

It will download the catalog from data.gouv.fr and update or create the rows in the various tables. Metrics will be computed for the current day (run it multiple days in a row to have historical depth).

Schema changes

Using alembic

Apply pending migrations:

ALEMBIC_ENV=(demo|prod) alembic upgrade head

demo env will use DATABASE_URL env var, prod env will use DATABASE_URL_PROD env var (same as the load script).

Create a new migration (diff code schema and database schema):

ALEMBIC_ENV=(demo|prod) alembic revision --autogenerate -m "message"

Legacy

• 2024-10-08: catalog.harvest_extras has been deprecated, catalog.harvest is now used. Quick migration: ALTER TABLE catalog DROP COLUMN IF EXISTS harvest_extras;
• 2024-11-02: the migration to SQLAlchemy #19 introduces migrations support.

Linting

Linting, formatting and import sorting are done automatically by Ruff launched by a pre-commit hook. So, before contributing to the repository, it is necessary to initialize the pre-commit hooks:

pre-commit install

Once this is done, code formatting and linting, as well as import sorting, will be automatically checked before each commit.

If you cannot use pre-commit, it is necessary to format, lint, and sort imports with Ruff before committing:

ruff check --fix .
ruff format .

WARNING: running ruff on the codebase will lint and format all of it, whereas using pre-commit will only be done on the staged files

Dokku

Published on http://ecospheres-catalog-scripts.app.france.sh (dummy page).

Manages the ecospheres-catalog database, also used by ecospheres-catalog-dokku (Metabase) as a secondary database.

dokku config:set --no-restart POSTGRES_DATABASE_SCHEME=postgresql
dokku postgres:link ecospheres-catalog ecospheres-catalog-scripts