Description of the issue
We're trying to load the MaStR data for solar plants only into an existing MySQL database. Since we want to retain control of our database state and have changes be done via alembic migrations, we would like to create the tables before instantiating the Mastr object.
We created the tables necessary for the solar download (solar_extended, solar_eeg) ourselves, but of course Mastr tries to create the others. Since we (unfortunately) use MySQL, which unfortunately requires an explicit length for varchar fields (i.e. String(length=some_len) instead of bare String), instantiating Mastr fails with:
sqlalchemy.exc.CompileError: (in table 'basic_units', column 'EinheitMastrNummer'): VARCHAR requires a length on dialect mysql
Ideas of solution
Our issue has two aspects:
- MySQL is not supported.
- One cannot prevent
Mastr from trying to create tables.
One easy fix for us would be making it possible to skip the table creation step. This is in general a good idea in my opinion to give the user more control over their database if desired. In some environments, this is also necessary because runtime database users aren’t granted permission to use any DDL statements for security reasons. (One can exploit things like function creation in Postgres to get access to the filesystem, arbitrary commands, etc.)
Another step would be supporting MySQL. This requires filling out all the length attributes of the String fields, which is definitely annoying work and may not be very easy to do in a future-proof way.
A third step would be instructing Mastr to only create those tables that are necessary for the intended download.
Workflow checklist
Description of the issue
We're trying to load the MaStR data for solar plants only into an existing MySQL database. Since we want to retain control of our database state and have changes be done via alembic migrations, we would like to create the tables before instantiating the
Mastrobject.We created the tables necessary for the solar download (
solar_extended,solar_eeg) ourselves, but of courseMastrtries to create the others. Since we (unfortunately) use MySQL, which unfortunately requires an explicit length for varchar fields (i.e.String(length=some_len)instead of bareString), instantiatingMastrfails with:Ideas of solution
Our issue has two aspects:
Mastrfrom trying to create tables.One easy fix for us would be making it possible to skip the table creation step. This is in general a good idea in my opinion to give the user more control over their database if desired. In some environments, this is also necessary because runtime database users aren’t granted permission to use any DDL statements for security reasons. (One can exploit things like function creation in Postgres to get access to the filesystem, arbitrary commands, etc.)
Another step would be supporting MySQL. This requires filling out all the length attributes of the
Stringfields, which is definitely annoying work and may not be very easy to do in a future-proof way.A third step would be instructing
Mastrto only create those tables that are necessary for the intended download.Workflow checklist