Merge pull request #773 from ie3-institute/jb/#716-refactor-csv-datasources

Refactor and harmonize CSV and SQL data sources

Author: jo-bao

CHANGELOG.md

@@ -15,9 +15,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Fixed wrong rated power unit hint [#804](https://github.com/ie3-institute/PowerSystemDataModel/issues/804)
 - Fixed wrong hash code generation of ConnectorResult [#817](https://github.com/ie3-institute/PowerSystemDataModel/issues/817) 
 
-
 ### Changed
 - Removing deprecated classes and methods [#540](https://github.com/ie3-institute/PowerSystemDataModel/issues/540)
+- Refactor CSV data sources [#716](https://github.com/ie3-institute/PowerSystemDataModel/issues/716)
 
 
 ## [3.0.0] - 2023-02-16
@@ -39,26 +39,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 - Reduced code smells [#492](https://github.com/ie3-institute/PowerSystemDataModel/issues/492)
-    - Protected constructors for abstract classes
-    - Use pattern matching
-    - Remove unused imports
-    - Use enhanced switch statements
-    - Replace lambdas with method references
-    - Use `Stream#toList`
-    - Adapt visibility for JUnit 5
+  - Protected constructors for abstract classes
+  - Use pattern matching
+  - Remove unused imports
+  - Use enhanced switch statements
+  - Replace lambdas with method references
+  - Use `Stream#toList`
+  - Adapt visibility for JUnit 5
 - More code smell fixing [#633](https://github.com/ie3-institute/PowerSystemDataModel/issues/633)
-    - Use `List#of`
-    - Use direct assignment with switch/case structures
-    - Turn some classes into records
-    - Making abstract classes' constructor protected
-    - Improving some RegExs
-    - Replacing `filter(Optional::isPresent).map(Optional::get)` on streams with `flatMap(Optional::stream)`
-    - instanceof variable declarations
-    - Removing unnecessary parentheses
-    - Miscellaneous code smells
+  - Use `List#of`
+  - Use direct assignment with switch/case structures
+  - Turn some classes into records
+  - Making abstract classes' constructor protected
+  - Improving some RegExs
+  - Replacing `filter(Optional::isPresent).map(Optional::get)` on streams with `flatMap(Optional::stream)`
+  - instanceof variable declarations
+  - Removing unnecessary parentheses
+  - Miscellaneous code smells
 - Fix JavaDoc creation
-    - Create JavaDoc with java 17 instead of java 8
-    - Let JavDoc pass, if there are warnings **ATTENTION:** Should be removed, when JavaDoc is fixed! (cf. Issue [#494](https://github.com/ie3-institute/PowerSystemDataModel/issues/494))
+  - Create JavaDoc with java 17 instead of java 8
+  - Let JavDoc pass, if there are warnings **ATTENTION:** Should be removed, when JavaDoc is fixed! (cf. Issue [#494](https://github.com/ie3-institute/PowerSystemDataModel/issues/494))
 - `BufferedCsvWriter` writes columns in the order, that the headline elements are defined [#434](https://github.com/ie3-institute/PowerSystemDataModel/issues/393)
 - Cleaned up `IndividualTimeSeriesMetaInformation`-related methods in `CsvFileConnector` [#544](https://github.com/ie3-institute/PowerSystemDataModel/issues/544)
 - Fixed spotlessApply handling for `.groovy` files [#637](https://github.com/ie3-institute/PowerSystemDataModel/issues/637)
@@ -79,15 +79,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - `edu.ie3.datamodel.io.connectors.CsvFileConnector.CsvIndividualTimeSeriesMetaInformation`
   - and related methods
 - BREAKING: Comprehensive harmonization around weather sources [#267](https://github.com/ie3-institute/PowerSystemDataModel/issues/267)
-    - Adapted the expected column scheme
-      - General weather model
-        - `coordinate` to `coordinateid`
-      - DWD COSMO model
-        - `diffuseirradiation` to `diffuseirradiance`
-        - `directirradiation` to `directirradiance`
-      - ICON model:
-        - `"datum"` to `"time"`
-    - Force user to provide time stamp pattern to `CouchbaseWeatherSource` to ensure harmonized querying
+  - Adapted the expected column scheme
+    - General weather model
+      - `coordinate` to `coordinateid`
+    - DWD COSMO model
+      - `diffuseirradiation` to `diffuseirradiance`
+      - `directirradiation` to `directirradiance`
+    - ICON model:
+      - `"datum"` to `"time"`
+  - Force user to provide time stamp pattern to `CouchbaseWeatherSource` to ensure harmonized querying
 - BREAKING: Updating PowerSystemUtils dependency to 2.0-SNAPSHOT [#595](https://github.com/ie3-institute/PowerSystemDataModel/issues/595)
 - BREAKING: Generified the `LoadInput` attribute `standardLoadProfile` to `loadProfile` as it should also address the newly added `TemperatureDependantLoadProfile`s [#601](https://github.com/ie3-institute/PowerSystemDataModel/issues/601)
 - Adapted to new double converters in PSU [#705](https://github.com/ie3-institute/PowerSystemDataModel/issues/705)

docs/readthedocs/_static/figures/uml/DataSourceClassDiagram.png

@@ -0,0 +1,40 @@
+# I/O
+
+The PowerSystemDataModel library additionally offers I/O-capabilities.
+In the long run, it is our aim to provide many different source and sink technologies.
+Therefore, the I/O-package is structured as highly modular.
+
+```{toctree}
+---
+maxdepth: 2
+---
+csvfiles
+sql
+influxdb
+ValidationUtils.md
+```
+
+## Data sink structure
+
+[![Class diagram of data sink classes](../_static/figures/uml/DataSinkClassDiagram.png)](../_static/figures/uml/DataSinkClassDiagram.png)
+
+## Data source structure
+
+The sources are divided in three blocks:
+1. InputEntities and ResultEntities
+2. TimeSeries related sources
+3. Weather and Coordinate sources
+
+[![Class diagram of entity sources](../_static/figures/uml/EntitySourceClassDiagram.png)](../_static/figures/uml/EntitySourceClassDiagram.png)
+
+[![Class diagram of time series sources](../_static/figures/uml/TimeSeriesSourceClassDiagram.png)](../_static/figures/uml/TimeSeriesSourceClassDiagram.png)
+
+[![Class diagram of weather and coordinate sources](../_static/figures/uml/WeatherCoordinateSourceClassDiagram.png)](../_static/figures/uml/WeatherCoordinateSourceClassDiagram.png)
+
+The function to read the sources are implemented in the DataSource classes.
+
+[![Class diagram of data sources](../_static/figures/uml/FunctionalDataSourceClassDiagram.png)](../_static/figures/uml/FunctionalDataSourceClassDiagram.png)
+
+## Data deployment
+
+[![Diagram of input data deployment](../_static/figures/uml/InputDataDeployment.png)](../_static/figures/uml/InputDataDeployment.png)

docs/readthedocs/_static/figures/uml/EntitySourceClassDiagram.png

@@ -121,6 +121,7 @@ The class `DefaultInputHierarchy` offers some helpful methods to validate and cr
 hierarchy.
 
 ## De-serialization (loading models)
+
 Having an instance of [Grid Container](/models/input/grid/gridcontainer) is most of the time the target whenever you load your
 grid. It consists of the three main blocks:
 
@@ -131,9 +132,6 @@ grid. It consists of the three main blocks:
 Those blocks are also reflected in the structure of data source interface definitions.
 There is one source for each of the containers, respectively.
 
-![Class diagram of data sources](../_static/figures/uml/DataSourceClassDiagram.png)
-Class diagram of data sources
-
 As a full data set has references among the models (e.g. a line model points to its' nodes it connects), there is a
 hierarchical structure, in which models have to be loaded.
 Therefore, the different sources have also references among themselves.