Extract individual files from bulk instead of actually downloading the bulk

[pt-kkraemer]

Hi everyone :)

Description of the issue

Most of the time, I do not need all the tables from the bulk download for updates. But I also can't be bothered to use the API to look for recent changes.

Ideas of solution

Implementing a new download method partial-bulk that only downloads the data specified in Mastr.download(data=None).

Workflow checklist

• I am aware of the workflow in CONTRIBUTING.md

• Command line tool of unzip_http proves that single tables from the bulk can be extracted without full download
  

• Proof of concept with a quick and dirty implementation: only downloading tables concerning wind takes 14 seconds. Again, I used unzip_http for implementation of the partial download.

Do you think that something like a partial-bulk method would be a good feature?

[nesnoj]

Hey @pt-kkraemer !

Thanks for the proposal, sounds like a nice useful feature to me! 🙌
Could you imagine to implement this?
We’ll just need to discuss the impact of adding new dependencies, especially in terms of long-term maintenance and compatibility.

[pt-kkraemer]

Yes I am currently implementing it more thoroughly.

Some problems already appear, for example with unzip_http I download and save unzipped xml-Files, which I then currently zip again so that the format fits write_mastr_xml_to_database. I imagine something like downloading the unzipped xml directly to a zipped directory is a better way of doing this, but I still have to figure that out.

[FlorianK13]

Hi @pt-kkraemer, this actually sounds nice and I was not aware that something like this exists. Right now I have the feeling that this would need some major restructuring of the code, ie some "only care about these tables" from the beginning of the download to parsing etc. If you are already implementing this, could you shortly sketch how you plan to integrate this in the current software? Maybe some short process flow?

[pt-kkraemer]

I just commited a minimum viable product here.

The changes I implemented are:

1. I am using the data argument in db.download() to specify which xml-Files to download.
2. Creating a download_xml_Mastr_partial function which includes the data argument.
3. This function implements the single file download using the BULK_INCLUDES_TABLE_MAP to translate the data argument to xml-File names.
4. I download the specified xml-Files to a folder which follows the current naming convention.
5. I zip the folder and delete the unzipped folder (this is not very sensible but works right now). This way I can reuse utils_write_to_database.py.
6. Added a new method == "partial bulk" to mastr.py which selects download_xml_Mastr_partial.
7. I updated several validations in helpers.py.

Next steps would be:

• Explaining "partial bulk" in docstrings
• Checking if other xml-Files than "Katalogwerte.xml" are needed for further data cleaning.
• Updating the sqlite-DB with tables that are not in it on the same day currently only works by deleting the xml-downloads, otherwise the software finds an already existing file download and skips the download altogether.

[FlorianK13]

I'm thinking about a deeper integration into the code. When I run the following as a user:

db=Mastr()
db.download(method="bulk", data="wind")

As a User, I would not expect that all data is downloaded and only wind is parsed. So maybe the partial download should be directly included in method="bulk".

• db.download(method="bulk", data="wind") results in downloading only wind and parsing it
• db.download(method="bulk", data=["wind", "solar"]) results in downloading wind and solar and parsing it
• db.download() results in downloading and parsing everything

[FlorianK13]

flowchart TB
a1("download()") --> a2(Download whole zipped folder)
a2 --> a3(Parse whole Folder)
b1("download(data='wind')") --> b2(Download only zipped wind files)
b2 --> b3(Parse wind files)

Loading

• Is date="existing" still useful in this setup?
• What happens when running download(data="wind") and then download(date="existing", data="solar") `

[FlorianK13]

Updating the sqlite-DB with tables that are not in it on the same day currently only works by deleting the xml-downloads, otherwise the software finds an already existing file download and skips the download altogether.

Day One

db.download(data="solar")

Day Two

db.download(data="wind")

should result in an sqlite database with entries in solar from day one and entries in wind from day 2. Or did you mean sth else?

[pt-kkraemer]

I meant something like this:

Day One 9 am

db.download(data="solar")

Day One 11 am

db.download(data="wind")

The second part would not work with a partial download currently because right now we check if a download from today exists and then the download is not started.

When we want to implement a partial download we'd have to check if the xml file we want to download is already in the zipped folder or not, not if a zipped folder from today exists.

[pt-kkraemer]

Quick update:

I implemented new changes here.

Changes consist of:

1. deleting the "partial bulk" download method name
2. using data specifier to decide if a full download or partial download should be carried out
3. create a function check_download_completeness that checks if, when carrying out two or more partial downloads after one another, the corresponding xml-files already exist in the zipped folder
4. integrating check_download_completeness into download_xml_Mastr_partial and adding a print statement that states which xml-files are missing from the saved zipped folder and then only downloading the missing xml-files

A resulting test run where I first download the permits and then the wind tables looks like this:

I have not yet checked what happens with date = 'existing'

[pt-kkraemer]

Quick update:

As of right now, unzip_http downloads the zipped files and extracts them. I just opened a PR with them to include a function which downloads parts of the remote zip file and adds it to a local zip file instead of saving the remote zip file part uncompressed. This is in my opinion necessary because saving for example uncompressed solar files from MaStR would take up too much disk space (in addition to the sqlite).

See here

[FlorianK13]

I think some function like this could be helpful to handle date='existing'

get_zip_metadata()
-> {date="20250401", "data"=["wind", "solar"]}

This could be either extracted from the zip file or we save this information to a metadata file when downloading the zips?

[pt-kkraemer]

That sounds good. Even when downloading the bulk, such a file could be useful:

No zip found:

get_zip_metadata()
-> {date= None, data=None}

Complete zip found:

get_zip_metadata()
-> {date="20250429", data=["all"]}

I am not sure about the None or "all" value for data. Maybe "all" could just be replaced with all the keys from BULK_INCLUDE_TABLES_MAP.

[FlorianK13]

Could we use the data = transform_data_parameter(data, **kwargs) that is created in download() and write this data to the metadata file?
I think this should be fine, having in mind the simplifications that will happen in #635

[pt-kkraemer]

Update on my PR at unzip_http:
My PR for a partly zipped instead of partly unzipped download function was unfortunately denied (see here).
I nevertheless think such a function is necessary, this means we would have to write something like a class which inherits the unzip_http.RemoteZipFile class and include the new functionality in there, what do you think?

Concerning the transform_data_parameter function: it is missing date information right? We would have to use transform_data_parameter and transform_date_parameter. Or we create a function that uses the outputs from those two transform_ functions and writes it into a metadata file.