Release v0.16.0

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.15.0
+current_version = 0.16.0
 parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)((?P<release>(a|na))+(?P<build>\d+))?
 serialize = 
 	{major}.{minor}.{patch}{release}{build}

.github/workflows/ci-develop.yml

@@ -13,6 +13,7 @@ jobs:
     runs-on: ${{ matrix.os }}
     if: ${{ !github.event.pull_request.draft }}
     strategy:
+      max-parallel: 1
       matrix:
         os: [macos-latest, ubuntu-latest, windows-latest]
         python-version: ['3.10', '3.11', '3.12']

.github/workflows/ci-production.yml

@@ -13,6 +13,7 @@ jobs:
     runs-on: ${{ matrix.os }}
     if: ${{ !github.event.pull_request.draft }}
     strategy:
+      max-parallel: 1
       matrix:
         os: [macos-latest, ubuntu-latest, windows-latest]
         python-version: ['3.10', '3.11', '3.12']
@@ -32,7 +33,7 @@ jobs:
     - name: create package
       run: python -m build --sdist
     - name: import open-mastr
-      run: python -m pip install ./dist/open_mastr-0.15.0.tar.gz
+      run: python -m pip install ./dist/open_mastr-0.16.0.tar.gz
     - name: Create credentials file
       env:
         MASTR_TOKEN: ${{ secrets.MASTR_TOKEN }}

.github/workflows/test-pypi-publish.yml

@@ -13,8 +13,6 @@ jobs:
     environment: pypi-publish
     steps:
     - uses: actions/checkout@v4
-      with:
-        ref: release
     - name: Set up Python 3.10
       uses: actions/setup-python@v3
       with:

CHANGELOG.md

@@ -6,9 +6,25 @@ For each version important additions, changes and removals are listed here.
 The format is inspired from [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and the versioning aims to respect [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
-## [v0.XX.X] unreleased - 202X-XX-XX
+## [v0.16.0] Partial downloads with open-MaStR PartialPumpkinPull - 2025-11-26
 ### Added
+- Add partial bulk download
+  [#652](https://github.com/OpenEnergyPlatform/open-MaStR/pull/652)
 ### Changed
+- Updates the system_catalog dict with missing Einheittyp values
+  [#653](https://github.com/OpenEnergyPlatform/open-MaStR/pull/653)
+- Fix package publication workflow
+  [#636](https://github.com/OpenEnergyPlatform/open-MaStR/pull/636)
+- Change print statement about data cleansing
+  [#650](https://github.com/OpenEnergyPlatform/open-MaStR/pull/650)
+- Improve logging
+  [#666](https://github.com/OpenEnergyPlatform/open-MaStR/pull/666)
+- Several improvements in bulk download: Support retaining old zip bulk files;
+  Prevent zip file deletion on full download; Add technology checks to full
+  bulk download
+  [#667](https://github.com/OpenEnergyPlatform/open-MaStR/pull/667)
+- Limit number of parallel CI jobs
+  [#669](https://github.com/OpenEnergyPlatform/open-MaStR/pull/669)
 ### Removed
 
 
@@ -35,7 +51,7 @@ and the versioning aims to respect [Semantic Versioning](http://semver.org/spec/
   [#621](https://github.com/OpenEnergyPlatform/open-MaStR/pull/621)
 ### Removed
 - Moved old code artefacts from `scripts` folder to paper specific
-  [repository](https://github.com/FlorianK13/verify-marktstammdaten) 
+  [repository](https://github.com/FlorianK13/verify-marktstammdaten)
   [#561](https://github.com/OpenEnergyPlatform/open-MaStR/pull/561)
 - Remove old dependencies and broken README links
   [#619](https://github.com/OpenEnergyPlatform/open-MaStR/pull/619)

CITATION.cff

@@ -34,10 +34,14 @@ authors:
     given-names: "Alexandra-Andreea"
     alias: "@AlexandraImbrisca"
     affiliation: "Technical University of Munich"
+  - family-names: 'Krämer'
+    given-names: "Kevin"
+    alias: "pt-kkraemer"
+    affiliation: "ProjectTogether gGmbH"  
 title: "open-MaStR"
 type: software
 license: AGPL-3.0
-version: 0.15.0
+version: 0.16.0
 doi: 
-date-released: 2025-04-19
+date-released: 2025-11-26
 url: "https://github.com/OpenEnergyPlatform/open-MaStR/"

README.rst

@@ -108,7 +108,6 @@ These projects already use open-mastr:
 - `Wasserstoffatlas <https://wasserstoffatlas.de/>`_
 - `EE-Status App <https://ee-status.de/>`_
 - `Digiplan Anhalt <https://digiplan.rl-institut.de/>`_
-- `Data Quality Assessment of the MaStR <https://marktstammdaten.kotthoff.dev/>`_
 - `EmPowerPlan <https://epp.rl-institut.de/>`_
 - `Goal100 Monitor <https://goal100.org/monitor>`_
 
@@ -119,7 +118,7 @@ changes in a `Pull Request <https://github.com/OpenEnergyPlatform/open-MaStR/pul
 External Resources
 ===================
 Besides open-mastr, some other resources exist that ease the process of working with the Marktstammdatenregister:
-- If you are interested in browsing the MaStR online, check out the github organisation `Marktstammdatenregister.dev <https://github.com/marktstammdatenregister-dev>`_.
+
 - The `bundesAPI/Marktstammdaten-API <https://github.com/bundesAPI/marktstammdaten-api>`_ is another implementation to access data via an official API.
 
 Collaboration
@@ -146,7 +145,7 @@ Data
 
 
 .. |badge_license| image:: https://img.shields.io/github/license/OpenEnergyPlatform/open-MaStR
-    :target: LICENSE.txt
+    :target: LICENSE.md
     :alt: License
 
 .. |badge_rtd| image:: https://readthedocs.org/projects/open-mastr/badge/?style=flat

docs/advanced.md

@@ -63,7 +63,7 @@ The project home directory is structured as follows (files and folders below `da
         File names are defined here.
      * `logging.yml` <br>
         Logging configuration. For changing the log level to increase or decrease details of log
-        messages, edit the level of the handlers.
+        messages, edit the level of the handlers. See below for details on logging.
 * **data**
      * `dataversion-<date>` <br>
         Contains exported data as csv files from method [`to_csv`][open_mastr.Mastr.to_csv]
@@ -83,6 +83,19 @@ The project home directory is structured as follows (files and folders below `da
 For the download via the API, logs are stored in a single file in `/$HOME/<user>/.open-MaStR/logs/open_mastr.log`.
 New logging messages are appended. It is recommended to delete the log file from time to time because of its required disk space.
 
+By default, the log level is set to `INFO`. You can increase or decrease the verbosity by either changing `logging.yml` (see above)
+or adjusting it manually in your code. E.g. to enable `DEBUG` messages in `open_mastr.log` you can use the following snippet:
+
+```python
+
+  import logging
+  from open_mastr import Mastr
+
+  # Increase to DEBUG to show more details in open_mastr.log
+  # Must be called after importing open_mastr to have the open-MaStR logger imported
+  logging.getLogger("open-MaStR").setLevel(logging.DEBUG)
+```
+
 
 ### Data
 
@@ -148,8 +161,11 @@ If needed, the tables in the database can be obtained as csv files. Those files
 
 === "Disadvantages"
     * No single tables or entries can be downloaded
-    * Download takes long time
+    * Download takes long time (you can use the partial download though, see [Getting Started](getting_started.md#bulk-download))
 
+**Note**: By default, existing zip files in `$HOME/.open-MaStR/data/xml_download` are deleted when a new file is
+downloaded. You can change this behavior by setting `keep_old_downloads`=True in
+[`Mastr.download()`][open_mastr.Mastr.download].
 
 ## SOAP API download
 

docs/getting_started.md

@@ -35,7 +35,16 @@ db = Mastr()
 db.download()
 ```
 
-When a `Mastr` object is initialized, a sqlite database is created in `$HOME/.open-MaStR/data/sqlite`. With the function `Mastr.download()`, the **whole MaStR is downloaded** in the zipped xml file format. It is then read into the sqlite database and simple data cleansing functions are started.
+When a `Mastr` object is initialized, a sqlite database is created in `$HOME/.open-MaStR/data/sqlite`. With the function [`Mastr.download()`][open_mastr.Mastr.download], the **whole MaStR is downloaded** in the zipped xml file format. It is then read into the sqlite database and simple data cleansing functions are started.
+
+If you are interested in a specific part of the dataset, you can specify this by using the `data` parameter:
+
+```python
+from open_mastr import Mastr
+
+db = Mastr()
+db.download(data=["wind","hydro"])
+```
 
 More detailed information can be found in the section [bulk download](advanced.md#bulk-download).
 

environment.yml

@@ -3,4 +3,4 @@ channels:
     - conda-forge
     - defaults
 dependencies:
-    - python=3.10
+    - python=3.11

open_mastr/mastr.py

@@ -2,7 +2,10 @@
 from sqlalchemy import inspect, create_engine
 
 # import xml dependencies
-from open_mastr.xml_download.utils_download_bulk import download_xml_Mastr
+from open_mastr.xml_download.utils_download_bulk import (
+    download_xml_Mastr,
+    delete_xml_files_not_from_given_date,
+)
 from open_mastr.xml_download.utils_write_to_database import (
     write_mastr_xml_to_database,
 )
@@ -23,6 +26,10 @@
     create_db_query,
     db_query_to_csv,
     reverse_fill_basic_units,
+    delete_zip_file_if_corrupted,
+    create_database_engine,
+    rename_table,
+    create_translated_database_engine,
 )
 from open_mastr.utils.config import (
     create_data_dir,
@@ -33,13 +40,6 @@
 )
 import open_mastr.utils.orm as orm
 
-# import initialize_database dependencies
-from open_mastr.utils.helpers import (
-    create_database_engine,
-    rename_table,
-    create_translated_database_engine,
-)
-
 # constants
 from open_mastr.utils.constants import TECHNOLOGIES, ADDITIONAL_TABLES
 
@@ -92,7 +92,10 @@ def __init__(self, engine="sqlite", connect_to_translated_db=False) -> None:
         else:
             self.engine = create_database_engine(engine, self._sqlite_folder_path)
 
-        print(
+        log.info(
+            "\n==================================================\n"
+            "--------->      open-MaStR started      <---------\n"
+            "==================================================\n"
             f"Data will be written to the following database: {self.engine.url}\n"
             "If you run into problems, try to "
             "delete the database and update the package by running "
@@ -107,6 +110,7 @@ def download(
         data=None,
         date=None,
         bulk_cleansing=True,
+        keep_old_downloads: bool = False,
         api_processes=None,
         api_limit=50,
         api_chunksize=1000,
@@ -126,8 +130,8 @@ def download(
             from marktstammdatenregister.de,
             (see :ref:`Configuration <Configuration>`). Default to 'bulk'.
         data : str or list or None, optional
-            Determines which types of data are written to the database. If None, all data is
-            used. If it is a list, possible entries are listed below with respect to the download method. Missing categories are
+            Determines which data is partially downloaded from the bulk download and written to the database. If None, all data is downloaded and written to the database.
+            If it is a list, possible entries are listed below with respect to the download method. Missing categories are
             being developed. If only one data is of interest, this can be given as a string. Default to None, where all data is included.
 
             | Data                  | Bulk | API  |
@@ -157,7 +161,7 @@ def download(
             |-----------------------|------|------|
             | "today"                | latest files are downloaded from marktstammdatenregister.de  | -  |
             | "20230101"      | If file from this date exists locally, it is used. Otherwise it throws an error (You can only receive todays data from the server)  | -   |
-            | "existing"               | Use latest downloaded zipped xml files, throws an error if the bulk download folder is empty  | -  |
+            | "existing"               | Deprecated since 0.16, see [#616](https://github.com/OpenEnergyPlatform/open-MaStR/issues/616#issuecomment-3089377062)  | -  |
             | "latest"               | -  | Retrieve data that is newer than the newest data already in the table  |
             | datetime.datetime(2020, 11, 27)      | -  | Retrieve data that is newer than this time stamp   |
             | None      | set date="today"  | set date="latest"   |
@@ -168,6 +172,8 @@ def download(
             In its original format, many entries in the MaStR are encoded with IDs. Columns like
             `state` or `fueltype` do not contain entries such as "Hessen" or "Braunkohle", but instead
             only contain IDs. Cleansing replaces these IDs with their corresponding original entries.
+        keep_old_downloads: bool
+            If set to True, prior downloaded MaStR zip files will be kept.
         api_processes : int or None or "max", optional
             Number of parallel processes used to download additional data.
             Defaults to `None`. If set to "max", the maximum number of possible processes
@@ -233,12 +239,20 @@ def download(
                 xml_folder_path,
                 f"Gesamtdatenexport_{bulk_download_date}.zip",
             )
-            download_xml_Mastr(zipped_xml_file_path, date, xml_folder_path)
 
-            print(
-                f"\nWould you like to speed up the bulk download?\n"
-                f"Try our new parallelized processing by setting os.environ['USE_RECOMMENDED_NUMBER_OF_PROCESSES'] = True "
-                f"or configure your own number of processes via os.environ['NUMBER_OF_PROCESSES'] = your_number\n"
+            delete_zip_file_if_corrupted(zipped_xml_file_path)
+            if not keep_old_downloads:
+                delete_xml_files_not_from_given_date(
+                    zipped_xml_file_path,
+                    xml_folder_path,
+                )
+
+            download_xml_Mastr(zipped_xml_file_path, date, data, xml_folder_path)
+
+            log.info(
+                "\nWould you like to speed up the creation of your MaStR database?\n"
+                "Try our new parallelized processing by setting os.environ['USE_RECOMMENDED_NUMBER_OF_PROCESSES'] = True "
+                "or configure your own number of processes via os.environ['NUMBER_OF_PROCESSES'] = your_number\n"
             )
 
             write_mastr_xml_to_database(
@@ -255,8 +269,8 @@ def download(
             # Set api_processes to None in order to avoid the malfunctioning usage
             if api_processes:
                 api_processes = None
-                print(
-                    "Warning: The implementation of parallel processes "
+                log.warning(
+                    "The implementation of parallel processes "
                     "is currently under construction. Please let "
                     "the argument api_processes at the default value None."
                 )
@@ -425,9 +439,11 @@ def translate(self) -> None:
             try:
                 os.remove(new_path)
             except Exception as e:
-                print(f"An error occurred: {e}")
+                log.error(
+                    f"An error occurred while removing old translated database: {e}"
+                )
 
-            print("Replacing previous version of the translated database...")
+            log.info("Replacing previous version of the translated database...")
 
         for table in inspector.get_table_names():
             rename_table(table, inspector.get_columns(table), self.engine)
@@ -436,9 +452,9 @@ def translate(self) -> None:
 
         try:
             os.rename(old_path, new_path)
-            print(f"Database '{old_path}' changed to '{new_path}'")
+            log.info(f"Database '{old_path}' changed to '{new_path}'")
         except Exception as e:
-            print(f"An error occurred: {e}")
+            log.error(f"An error occurred while renaming database: {e}")
 
         self.engine = create_engine(f"sqlite:///{new_path}")
         self.is_translated = True

open_mastr/soap_api/metadata/description.py

@@ -1,10 +1,13 @@
 from io import BytesIO
+import logging
 import re
 from urllib.request import urlopen
 from zipfile import ZipFile
 import xmltodict
 from collections import OrderedDict
 
+log = logging.getLogger(__name__)
+
 
 class DataDescription(object):
     """
@@ -150,9 +153,11 @@ def functions_data_documentation(self):
                             fcn["sequence"]["element"]["@type"].split(":")[1]
                         ]["sequence"]["element"]
                 else:
-                    print(type(fcn["sequence"]))
-                    print(fcn["sequence"])
-                    raise ValueError
+                    log.error(f"Unexpected sequence type: {type(fcn['sequence'])}")
+                    log.error(f"Sequence content: {fcn['sequence']}")
+                    raise ValueError(
+                        f"Unexpected sequence structure in function metadata"
+                    )
 
                 # Add data for inherited columns from base types
                 if "@base" in fcn:

open_mastr/utils/config/logging.yml

@@ -4,6 +4,8 @@ disable_existing_loggers: False
 formatters:
   standard:
     format: "%(asctime)s [%(levelname)s] %(message)s"
+  debug:
+    format: "%(asctime)s [%(levelname)s] %(name)s:%(funcName)s:%(lineno)d - %(message)s"
 
 handlers:
   console:
@@ -12,14 +14,13 @@ handlers:
     class: "logging.StreamHandler"
     stream: "ext://sys.stdout"
   file:
-    class: "logging.FileHandler"
     level: "DEBUG"
-    formatter: "standard"
+    formatter: "debug"
+    class: "logging.FileHandler"
     mode: "a"
 
-root:
-    level: "DEBUG"
-
 loggers:
   open-MaStR:
+    level: "INFO"
     handlers: ["console", "file"]
+    propagate: no