Merge branch 'dev' into vb/#791_delete_initFiles

Author: t-ober

CHANGELOG.md

@@ -15,9 +15,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Fixed wrong rated power unit hint [#804](https://github.com/ie3-institute/PowerSystemDataModel/issues/804)
 - Fixed wrong hash code generation of ConnectorResult [#817](https://github.com/ie3-institute/PowerSystemDataModel/issues/817) 
 
-
 ### Changed
 - Removing deprecated classes and methods [#540](https://github.com/ie3-institute/PowerSystemDataModel/issues/540)
+- Refactor CSV data sources [#716](https://github.com/ie3-institute/PowerSystemDataModel/issues/716)
 
 
 ## [3.0.0] - 2023-02-16
@@ -39,26 +39,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 - Reduced code smells [#492](https://github.com/ie3-institute/PowerSystemDataModel/issues/492)
-    - Protected constructors for abstract classes
-    - Use pattern matching
-    - Remove unused imports
-    - Use enhanced switch statements
-    - Replace lambdas with method references
-    - Use `Stream#toList`
-    - Adapt visibility for JUnit 5
+  - Protected constructors for abstract classes
+  - Use pattern matching
+  - Remove unused imports
+  - Use enhanced switch statements
+  - Replace lambdas with method references
+  - Use `Stream#toList`
+  - Adapt visibility for JUnit 5
 - More code smell fixing [#633](https://github.com/ie3-institute/PowerSystemDataModel/issues/633)
-    - Use `List#of`
-    - Use direct assignment with switch/case structures
-    - Turn some classes into records
-    - Making abstract classes' constructor protected
-    - Improving some RegExs
-    - Replacing `filter(Optional::isPresent).map(Optional::get)` on streams with `flatMap(Optional::stream)`
-    - instanceof variable declarations
-    - Removing unnecessary parentheses
-    - Miscellaneous code smells
+  - Use `List#of`
+  - Use direct assignment with switch/case structures
+  - Turn some classes into records
+  - Making abstract classes' constructor protected
+  - Improving some RegExs
+  - Replacing `filter(Optional::isPresent).map(Optional::get)` on streams with `flatMap(Optional::stream)`
+  - instanceof variable declarations
+  - Removing unnecessary parentheses
+  - Miscellaneous code smells
 - Fix JavaDoc creation
-    - Create JavaDoc with java 17 instead of java 8
-    - Let JavDoc pass, if there are warnings **ATTENTION:** Should be removed, when JavaDoc is fixed! (cf. Issue [#494](https://github.com/ie3-institute/PowerSystemDataModel/issues/494))
+  - Create JavaDoc with java 17 instead of java 8
+  - Let JavDoc pass, if there are warnings **ATTENTION:** Should be removed, when JavaDoc is fixed! (cf. Issue [#494](https://github.com/ie3-institute/PowerSystemDataModel/issues/494))
 - `BufferedCsvWriter` writes columns in the order, that the headline elements are defined [#434](https://github.com/ie3-institute/PowerSystemDataModel/issues/393)
 - Cleaned up `IndividualTimeSeriesMetaInformation`-related methods in `CsvFileConnector` [#544](https://github.com/ie3-institute/PowerSystemDataModel/issues/544)
 - Fixed spotlessApply handling for `.groovy` files [#637](https://github.com/ie3-institute/PowerSystemDataModel/issues/637)
@@ -79,15 +79,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - `edu.ie3.datamodel.io.connectors.CsvFileConnector.CsvIndividualTimeSeriesMetaInformation`
   - and related methods
 - BREAKING: Comprehensive harmonization around weather sources [#267](https://github.com/ie3-institute/PowerSystemDataModel/issues/267)
-    - Adapted the expected column scheme
-      - General weather model
-        - `coordinate` to `coordinateid`
-      - DWD COSMO model
-        - `diffuseirradiation` to `diffuseirradiance`
-        - `directirradiation` to `directirradiance`
-      - ICON model:
-        - `"datum"` to `"time"`
-    - Force user to provide time stamp pattern to `CouchbaseWeatherSource` to ensure harmonized querying
+  - Adapted the expected column scheme
+    - General weather model
+      - `coordinate` to `coordinateid`
+    - DWD COSMO model
+      - `diffuseirradiation` to `diffuseirradiance`
+      - `directirradiation` to `directirradiance`
+    - ICON model:
+      - `"datum"` to `"time"`
+  - Force user to provide time stamp pattern to `CouchbaseWeatherSource` to ensure harmonized querying
 - BREAKING: Updating PowerSystemUtils dependency to 2.0-SNAPSHOT [#595](https://github.com/ie3-institute/PowerSystemDataModel/issues/595)
 - BREAKING: Generified the `LoadInput` attribute `standardLoadProfile` to `loadProfile` as it should also address the newly added `TemperatureDependantLoadProfile`s [#601](https://github.com/ie3-institute/PowerSystemDataModel/issues/601)
 - Adapted to new double converters in PSU [#705](https://github.com/ie3-institute/PowerSystemDataModel/issues/705)

build.gradle

@@ -9,7 +9,7 @@ plugins {
   id 'de.undercouch.download' version '5.4.0'
   id 'kr.motd.sphinx' version '2.10.1' // documentation generation
   id 'jacoco' // java code coverage plugin
-  id "org.sonarqube" version "4.0.0.2929" // sonarqube
+  id "org.sonarqube" version "4.2.1.3168" // sonarqube
   id 'net.thauvin.erik.gradle.semver' version '1.0.4' // semantic versioning
 }
 
@@ -18,7 +18,7 @@ ext {
   javaVersion = JavaVersion.VERSION_17
   groovyVersion = "4.0"
   groovyBinaryVersion = "4.0.12"
-  testcontainersVersion = '1.18.1'
+  testcontainersVersion = '1.18.3'
 
   scriptsLocation = 'gradle' + File.separator + 'scripts' + File.separator //location of script plugins
 }
@@ -70,7 +70,7 @@ dependencies {
   testImplementation 'org.junit.jupiter:junit-jupiter:5.9.3'
   testImplementation "org.spockframework:spock-core:2.3-groovy-$groovyVersion"
   testImplementation 'org.objenesis:objenesis:3.3' // Mock creation with constructor parameters
-  testImplementation 'net.bytebuddy:byte-buddy:1.14.4' // Mocks of classes
+  testImplementation 'net.bytebuddy:byte-buddy:1.14.5' // Mocks of classes
 
   // testcontainers (docker framework for testing)
   testImplementation "org.testcontainers:testcontainers:$testcontainersVersion"
@@ -87,10 +87,10 @@ dependencies {
 
   // Databases
   implementation 'org.influxdb:influxdb-java:2.23'
-  implementation 'com.couchbase.client:java-client:3.4.6'
+  implementation 'com.couchbase.client:java-client:3.4.7'
   runtimeOnly 'org.postgresql:postgresql:42.6.0' // postgresql jdbc driver required during runtime
 
-  implementation 'commons-io:commons-io:2.12.0' // I/O functionalities
+  implementation 'commons-io:commons-io:2.13.0' // I/O functionalities
   implementation 'org.apache.commons:commons-compress:1.23.0' // I/O functionalities
   implementation 'org.apache.commons:commons-lang3:3.12.0'
 }

docs/readthedocs/_static/figures/uml/DataSourceClassDiagram.png

@@ -0,0 +1,40 @@
+# I/O
+
+The PowerSystemDataModel library additionally offers I/O-capabilities.
+In the long run, it is our aim to provide many different source and sink technologies.
+Therefore, the I/O-package is structured as highly modular.
+
+```{toctree}
+---
+maxdepth: 2
+---
+csvfiles
+sql
+influxdb
+ValidationUtils.md
+```
+
+## Data sink structure
+
+[![Class diagram of data sink classes](../_static/figures/uml/DataSinkClassDiagram.png)](../_static/figures/uml/DataSinkClassDiagram.png)
+
+## Data source structure
+
+The sources are divided in three blocks:
+1. InputEntities and ResultEntities
+2. TimeSeries related sources
+3. Weather and Coordinate sources
+
+[![Class diagram of entity sources](../_static/figures/uml/EntitySourceClassDiagram.png)](../_static/figures/uml/EntitySourceClassDiagram.png)
+
+[![Class diagram of time series sources](../_static/figures/uml/TimeSeriesSourceClassDiagram.png)](../_static/figures/uml/TimeSeriesSourceClassDiagram.png)
+
+[![Class diagram of weather and coordinate sources](../_static/figures/uml/WeatherCoordinateSourceClassDiagram.png)](../_static/figures/uml/WeatherCoordinateSourceClassDiagram.png)
+
+The function to read the sources are implemented in the DataSource classes.
+
+[![Class diagram of data sources](../_static/figures/uml/FunctionalDataSourceClassDiagram.png)](../_static/figures/uml/FunctionalDataSourceClassDiagram.png)
+
+## Data deployment
+
+[![Diagram of input data deployment](../_static/figures/uml/InputDataDeployment.png)](../_static/figures/uml/InputDataDeployment.png)

docs/readthedocs/_static/figures/uml/EntitySourceClassDiagram.png

@@ -121,6 +121,7 @@ The class `DefaultInputHierarchy` offers some helpful methods to validate and cr
 hierarchy.
 
 ## De-serialization (loading models)
+
 Having an instance of [Grid Container](/models/input/grid/gridcontainer) is most of the time the target whenever you load your
 grid. It consists of the three main blocks:
 
@@ -131,9 +132,6 @@ grid. It consists of the three main blocks:
 Those blocks are also reflected in the structure of data source interface definitions.
 There is one source for each of the containers, respectively.
 
-![Class diagram of data sources](../_static/figures/uml/DataSourceClassDiagram.png)
-Class diagram of data sources
-
 As a full data set has references among the models (e.g. a line model points to its' nodes it connects), there is a
 hierarchical structure, in which models have to be loaded.
 Therefore, the different sources have also references among themselves.