Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
ce3e658
Delete method=API from mastr.py #622
FlorianK13 Apr 24, 2025
bce5810
Delete deprecated mirror class #622
FlorianK13 Apr 24, 2025
fe466df
Delete MaStRDownload Class #622
FlorianK13 Apr 24, 2025
257dde0
Delete API from main script #622
FlorianK13 Apr 24, 2025
39af5cf
Delete empty file #622
FlorianK13 Apr 24, 2025
7a6a10a
Change docstring of Mastr class #622
FlorianK13 Apr 24, 2025
d5dbfc0
Delete unused test file #622
FlorianK13 Apr 24, 2025
a1490f9
Delete API parts from test #622
FlorianK13 Apr 24, 2025
af78c23
Commenting out the metadata functions #622
FlorianK13 Apr 24, 2025
b05431e
Repair broken test #622
FlorianK13 Apr 24, 2025
9bae59e
Reintroduce test file #622
FlorianK13 Apr 24, 2025
ee4c3f3
Fix parameter testing #622
FlorianK13 Apr 24, 2025
97c8962
Fix invalid parameter test #622
FlorianK13 Apr 24, 2025
88f9674
Add small comment to function #622
FlorianK13 Apr 24, 2025
43f2008
Delete broken test #622
FlorianK13 Apr 29, 2025
2270ff1
Delete outdated return values #622
FlorianK13 Apr 29, 2025
2e0e912
Delete unused test requirements file #622
FlorianK13 Apr 29, 2025
224030c
Delete API Mirror scripts #622
FlorianK13 Apr 29, 2025
416e6f8
Delete outdated orm.py #622
FlorianK13 Apr 29, 2025
bd8aa54
Delete Soap API download from docs #622
FlorianK13 Apr 29, 2025
1c738d7
Repair broken link in docs #622
FlorianK13 Apr 29, 2025
83ed65a
Delete Soap API download references from docs #622
FlorianK13 Apr 29, 2025
8014719
Delete unmaintained release-notes #622
FlorianK13 Apr 29, 2025
376b349
Delete outdated `sphinx` file #622
FlorianK13 Apr 29, 2025
d0cab9b
Update Changelog #622
FlorianK13 Apr 29, 2025
99d7652
Delete API specific tables #622
FlorianK13 Aug 19, 2025
2821d82
Merge branch 'develop' into 622-deprecate-api-download
FlorianK13 Aug 19, 2025
c4b2911
Delete outdated parameter that survived the merge #622
FlorianK13 Aug 19, 2025
9dd939c
Repair docstrings #622
FlorianK13 Aug 19, 2025
4a21f83
Make xml download work in tests #660
FlorianK13 Aug 26, 2025
8e47564
Hopefully repair test #622
FlorianK13 Aug 27, 2025
a90d488
Dispose engine after usage #622
FlorianK13 Aug 27, 2025
da77a15
Create tranlated db directly in function #622
FlorianK13 Aug 27, 2025
3bfb683
Deactivate _translate testing #622
FlorianK13 Aug 27, 2025
808ef94
Re-add class BasicUnit to orm #622
nesnoj Oct 20, 2025
c78024c
Merge branch 'develop' into 622-deprecate-api-download
FlorianK13 Oct 27, 2025
4be6399
Revert changes for delayed testing strategy #622
FlorianK13 Oct 27, 2025
3819953
Revert changes at test_mastr.py #622
FlorianK13 Oct 27, 2025
ba2291d
Merge branch 'develop' into 622-deprecate-api-download
FlorianK13 Dec 5, 2025
b689f61
Update docs/advanced.md #622
FlorianK13 Dec 8, 2025
82624cd
Update docs/getting_started.md #622
FlorianK13 Dec 8, 2025
bef07f0
Update open_mastr/mastr.py #622
FlorianK13 Dec 8, 2025
e1216b6
Update open_mastr/mastr.py #622
FlorianK13 Dec 8, 2025
5e786db
Update open_mastr/mastr.py #622
FlorianK13 Dec 8, 2025
bb46dc5
Update open_mastr/mastr.py #622
FlorianK13 Dec 8, 2025
337e3bb
Delete unused function to check credentials #622
FlorianK13 Dec 8, 2025
6dd79dc
Update docs/advanced.md #622
FlorianK13 Dec 8, 2025
b533a05
Remove API specific constants #622
FlorianK13 Dec 8, 2025
99c8e33
Run black #622
FlorianK13 Dec 8, 2025
87e2c41
Merge branch 'develop' into 622-deprecate-api-download
FlorianK13 Jan 13, 2026
acabbda
Delete unused import #622
FlorianK13 Jan 13, 2026
329add6
Use correct date parameter #622
FlorianK13 Jan 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ and the versioning aims to respect [Semantic Versioning](http://semver.org/spec/
- Limit number of parallel CI jobs
[#669](https://github.com/OpenEnergyPlatform/open-MaStR/pull/669)
### Removed
- Deprecate and remove Soap API Download and Mirror code
[#635](https://github.com/OpenEnergyPlatform/open-MaStR/pull/635)


## [v0.15.0] Turbo parsing with open-MaStR EasterEggspress - 2025-04-19
Expand Down
40 changes: 6 additions & 34 deletions docs/advanced.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,14 +155,6 @@ Moreover, the datatypes of different entries are set in the data cleansing proce

If needed, the tables in the database can be obtained as csv files. Those files are created by first merging corresponding tables (e.g all tables that contain information about solar) and then dumping those tables to `.csv` files with the [`to_csv`][open_mastr.Mastr.to_csv] method.

=== "Advantages"
* No registration for an API key is needed
* Download of the whole dataset is possible

=== "Disadvantages"
* No single tables or entries can be downloaded
* Download takes long time (you can use the partial download though, see [Getting Started](getting_started.md#bulk-download))

**Note**: By default, existing zip files in `$HOME/.open-MaStR/data/xml_download` are deleted when a new file is
downloaded. You can change this behavior by setting `keep_old_downloads`=True in
[`Mastr.download()`][open_mastr.Mastr.download].
Expand All @@ -179,12 +171,8 @@ via its API a [registration](https://www.marktstammdatenregister.de/MaStRHilfe/f
To download data from the MaStR API using the `open-MaStR`, the credentials (MaStR user and token) need to be provided in a certain way. Three options exist:

1. **Credentials file:**
Both, user and token, are stored in plain text in the credentials file.
For storing the credentials in the credentials file (plus optionally using keyring for the token) simply instantiate
[`MaStRDownload`][open_mastr.soap_api.download.MaStRDownload] once and you get asked for a user name and a token. The
information you insert will be used to create the credentials file.

It is also possible to create the credentials file by hand, using this format:
Both, user and token, are stored in plain text in the credentials file. The file is located at
'~/.open-MaStR/config/credentials.cfg'. Fill in your user and token like this:

```
[MaStR]
Expand All @@ -194,8 +182,6 @@ To download data from the MaStR API using the `open-MaStR`, the credentials (MaS

The `token` should be written in one line, without line breaks.

The credentials file needs to be stored at: `$HOME/.open-MaStR/config/credentials.cfg`

2. **Credentials file + keyring:**
The user is stored in the credentials file, while the token is stored encrypted in the [keyring](https://pypi.org/project/keyring/).

Expand Down Expand Up @@ -419,33 +405,19 @@ For API calls, models and optional parameters refer to the
identifier (`EinheitMastrNummer`) and their power plant type (`Einheitentyp`). You can then sort them by power
plant type and use the power plant type specific API query to retrieve information about it. <br>

Cumbersome? <br>
Luckily, `open-MaStR` has you covered and provides methods to just query for all units of a power
plant type.




### MaStRDownload

The class `MaStRDownload` builds upon methods provided in the class `MaStRAPI`. <br>

It provides methods to download power plant unit types and additional information
for each unit type, such as extended unit data, permit data, chp-specific data, location data
or eeg-specific data. <br>

The class handles the querying logic and knows which additional data for each unit type is available
and which SOAP service has to be used to query it.
!!! warning "MaStRDownload has been removed"
In versions > `v0.16.0` the `MaStRDownload` class cannot be used anymore.


### MaStRMirror

The class `MaStRMirror` builds upon methods provided in the class `MaStRDownload`. <br>

The aim of the class has been to mirror the Marktstammdatenregister database and keep it up-to-date.
Historically, `open-mastr` has been developed before the owner of the dataset, BNetzA, offered the `bulk` download.
The class can still be used for use-cases where only the most recent changes to a local database are of interest.
For downloading the entire MaStR database we recommend the bulk download functionalities by specifying `donwload(method="bulk")`.
!!! warning "MaStRMirror has been removed"
In versions > `v0.16.0` the `MaStRMirror` class cannot be used anymore.



64 changes: 0 additions & 64 deletions docs/conf.py

This file was deleted.

31 changes: 0 additions & 31 deletions docs/data-release-notes.md

This file was deleted.

8 changes: 4 additions & 4 deletions docs/dataset.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Most unit information is openly accessible and is published under an open data l

For units with a net capacity of less than 30 kW, some location information is restricted from publication. This includes street names, house numbers, parcel designations, and exact coordinates of units. The most detailed location information accessible for all units is the postal code or the municipality.

In our paper titled [Analyzing Data Reliability in Germany's Energy System: A Validation of Unit Locations of the Marktstammdatenregister](https://arxiv.org/abs/2304.10581), we provide further insights into the content and quality of the dataset.
In our paper titled [Monitoring Germany's Core Energy System Dataset: A Data Quality Analysis of the Marktstammdatenregister](https://doi.org/10.1145/3717413.3717421), we provide further insights into the content and quality of the dataset.

## Content

Expand All @@ -17,12 +17,12 @@ The German Federal Network Agency regularly updates the dataset and adds new tab

## Difference between `bulk` and `API` dataset

As you may have noticed, we distinguish between `bulk` and `API` datasets. The `bulk` dataset refers to the data obtained from the zipped XML files downloaded from [here](https://www.marktstammdatenregister.de/MaStR/Datendownload) using the [`Mastr.download`][open_mastr.Mastr.download] function. The `API` data is obtained by requesting information via the SOAP-API and the [`soap_api.download.MaStRDownload`][open_mastr.soap_api.download.MaStRDownload] module.
As you may have noticed, we distinguish between `bulk` and `API` datasets. The `bulk` dataset refers to the data obtained from the zipped XML files downloaded from [here](https://www.marktstammdatenregister.de/MaStR/Datendownload) using the [`Mastr.download`][open_mastr.Mastr.download] function. The `API` data is obtained by requesting information via the SOAP-API and the [`soap_api.download.MaStRAPI`][open_mastr.soap_api.download.MaStRAPI] module.

??? question "Why is the table structure in the open-mastr database as it is?"

The structure of the database is historically determined by the data retrieved via API. (open-mastr existed before the XML-dump was provided).
<br> See [MaStR data model](#Mastr-data-model)
<br> See [MaStR data model](#mastr-data-model)


## Tables in the database
Expand Down Expand Up @@ -86,7 +86,7 @@ After downloading the MaStR, you will find a database with a large number of tab


### MaStR data model
A useful overview of the MaStR data model can be found [here (in german)](https://www.marktstammdatenregister.de/MaStRHilfe/files/webdienst/Objektmodell%20-%20Fachliche%20Ansicht%20V1.2.0.pdf). A translated version using the names from the tables you can find in your local database is presented here:
A useful overview of the MaStR data model can be found at the MaStR [help page](https://www.marktstammdatenregister.de/MaStRHilfe/subpages/faq.html). A translated version using the names from the tables you can find in your local database is presented here:

=== "translated image (english)"
![Data model of the MaStR](images/DetailAnlagen_english.PNG)
Expand Down
12 changes: 2 additions & 10 deletions docs/getting_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,17 +50,9 @@ More detailed information can be found in the section [bulk download](advanced.m

API download
-----------------------------------
When using `download(method="API")`, the data is retrieved from the MaStR API. For using the MaStR API, credentials
are needed (see [SOAP API download](advanced.md#soap-api-download)).
!!! warning "API download has been removed"
In versions > `v0.16.0` the `download(method="API")` cannot be used anymore. The default behaviour changes `method="API"` to `method="bulk"`. You can still use the basic SOAP API functionalities from the [`MaStRAPI`][open_mastr.soap_api.download.MaStRAPI] class.

```python
from open_mastr import Mastr

db = Mastr()
db.download(method='API')
```

The default settings will save retrieved data into the sqlite database. The function can be used to mirror the open-MaStR database using the SOAP API. Note that the data provided by the bulk download and the SOAP API is similar, but not exactly the same.

## Accessing the database

Expand Down
2 changes: 0 additions & 2 deletions docs/reference/advanced.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
# Advanced functions to use the MaStR SOAP-API

::: open_mastr.soap_api.download.MaStRAPI
::: open_mastr.soap_api.download.MaStRDownload
::: open_mastr.soap_api.mirror.MaStRMirror
41 changes: 0 additions & 41 deletions main.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
open-MaStR - Main file

Bulk: Download XML-Dump and fill in local SQLite database.
API: Download latest entries using the SOAP-API.

SPDX-License-Identifier: AGPL-3.0-or-later
"""
Expand Down Expand Up @@ -45,34 +44,6 @@
"permit",
]

# API download
# for parameter explanation see: https://open-mastr.readthedocs.io/en/latest/advanced/#soap-api-download

api_date = "latest"
api_chunksize = 10
api_limit = 10
api_processes = None

data_api = [
"biomass",
"combustion",
"gsgk",
"hydro",
"nuclear",
"solar",
"storage",
"wind",
]

api_data_types = ["unit_data", "eeg_data", "kwk_data", "permit_data"]

api_location_types = [
"location_elec_generation",
"location_elec_consumption",
"location_gas_generation",
"location_gas_consumption",
]

if __name__ == "__main__":
# instantiate Mastr class
db = Mastr()
Expand All @@ -81,18 +52,6 @@
# bulk download
db.download(method="bulk", data=data_bulk, date=bulk_date, bulk_cleansing=True)

# API download
db.download(
method="API",
data=data_api,
date=api_date,
api_processes=api_processes,
api_limit=api_limit,
api_chunksize=api_chunksize,
api_data_types=api_data_types,
api_location_types=api_location_types,
)

## export to csv
"""
Technology-related tables are exported as joined, whereas additional tables
Expand Down
Loading