Merge branch 'dev' into pp/#825-enhance-documentation-for-csv-timeseries

# Conflicts:
#	CHANGELOG.md

Author: pierrepetersmeier

.gitignore

@@ -242,4 +242,9 @@ docs/readthedocs/_build
 **/*.synctex.gz
 **/*.synctex(busy)
 
-docs/javadoc
+### Metals ###
+.metals/
+.bloop/
+project/**/metals.sbt
+
+docs/javadoc

build.gradle

@@ -18,7 +18,7 @@ ext {
   javaVersion = JavaVersion.VERSION_17
   groovyVersion = "4.0"
   groovyBinaryVersion = "4.0.18"
-  testcontainersVersion = '1.19.4'
+  testcontainersVersion = '1.19.6'
 
   scriptsLocation = 'gradle' + File.separator + 'scripts' + File.separator //location of script plugins
 }
@@ -53,7 +53,7 @@ repositories {
 dependencies {
 
   //	 ie³ power system utils
-  implementation 'com.github.ie3-institute:PowerSystemUtils:2.0'
+  implementation 'com.github.ie3-institute:PowerSystemUtils:2.2'
 
   implementation 'tech.units:indriya:2.2'
 
@@ -70,10 +70,10 @@ dependencies {
   // testing
   testImplementation "org.apache.groovy:groovy:$groovyBinaryVersion"
 
-  testImplementation 'org.junit.jupiter:junit-jupiter:5.10.1'
+  testImplementation 'org.junit.jupiter:junit-jupiter:5.10.2'
   testImplementation "org.spockframework:spock-core:2.3-groovy-$groovyVersion"
   testImplementation 'org.objenesis:objenesis:3.3' // Mock creation with constructor parameters
-  testImplementation 'net.bytebuddy:byte-buddy:1.14.11' // Mocks of classes
+  testImplementation 'net.bytebuddy:byte-buddy:1.14.12' // Mocks of classes
 
   // testcontainers (docker framework for testing)
   testImplementation "org.testcontainers:testcontainers:$testcontainersVersion"
@@ -83,19 +83,19 @@ dependencies {
   testImplementation "org.testcontainers:couchbase:$testcontainersVersion"
 
   // logging
-  implementation platform('org.apache.logging.log4j:log4j-bom:2.22.1')
+  implementation platform('org.apache.logging.log4j:log4j-bom:2.23.0')
   implementation 'org.apache.logging.log4j:log4j-api' // log4j
   implementation 'org.apache.logging.log4j:log4j-core' // log4j
   implementation 'org.apache.logging.log4j:log4j-slf4j-impl' // log4j -> slf4j
 
   // Databases
   implementation 'org.influxdb:influxdb-java:2.24'
-  implementation 'com.couchbase.client:java-client:3.5.2'
-  runtimeOnly 'org.postgresql:postgresql:42.7.1' // postgresql jdbc driver required during runtime
+  implementation 'com.couchbase.client:java-client:3.5.3'
+  runtimeOnly 'org.postgresql:postgresql:42.7.2' // postgresql jdbc driver required during runtime
 
   implementation 'commons-io:commons-io:2.15.1' // I/O functionalities
-  implementation 'org.apache.commons:commons-compress:1.25.0' // I/O functionalities
-  implementation 'org.apache.commons:commons-lang3:3.14.0'
+  implementation 'commons-codec:commons-codec:1.16.1' // needed by commons-compress
+  implementation 'org.apache.commons:commons-compress:1.26.0' // I/O functionalities
 }
 
 tasks.withType(JavaCompile) {

docs/readthedocs/gettingstarted.md

@@ -1,5 +1,4 @@
 # Getting started
-
 Welcome, dear fellow of bottom up power system modelling!
 This section is meant to give you some help getting hands on our project.
 If you feel, something is missing, please contact us!
@@ -45,3 +44,32 @@ and add the dependency:
   <version>3.0-SNAPSHOT</version>
 </dependency>
 ```
+
+## Important changes
+
+With the release of `PowerSystemDataModel` version `5.0` the support for the old csv file format will be fully removed.
+It was already marked as `deprecated` back in version `1.1.0`. For those who are still using a model in the old csv format
+the following guide will provide a fast and easy way to convert old format into the new one.
+
+- Since the support is removed in version `5.0`, the `PowerSystemDataModel` version `3.x` or `4.x` must be 
+  used to read the old format. The `PSDM` will automatically write the output model in the new csv format.
+
+
+``` java
+    /* Parameterization */
+    String gridName = "gridWithOldFormat";
+    String csvSep = ",";
+    Path folderPath = Path.of("PATH_TO_THE_FOLDER");
+    boolean isHierarchic = false;
+    Path output = Path.of("PATH_OF_THE_OUTPUT_FOLDER");
+    FileNamingStrategy namingStrategy = new FileNamingStrategy();
+    
+    /* Reading the old format */
+    JointGridContainer container = CsvJointGridContainerSource.read(gridName, csvSep, folderPath, isHierarchic);
+    
+    /* Writing in the new format */
+    CsvFileSink sink = new CsvFileSink(output, namingStrategy, csvSep);
+    sink.persistJointGrid(container);
+```
+
+

docs/readthedocs/io/ValidationUtils.md

@@ -2,15 +2,31 @@
 This page gives an overview about the ValidationUtils in the *PowerSystemDataModel*.
 
 ## What are the ValidationUtils?
-The methods in ValidationUtils and subclasses can be used to check that objects are valid, meaning their parameters have valid values and they are correctly connected.
+The methods in ValidationUtils and subclasses can be used to check that objects are valid. The checks can be separated into two types:
+1. General validation
+2. Uniqueness validation
+
 
 ## What is checked?
-- The check methods include checks that assigned values are valid, e.g. lines are not allowed to have negative lengths or the rated power factor of any unit must be between 0 and 1.
-- Furthermore, several connections are checked, e.g. that lines only connect nodes of the same voltage level or that the voltage levels indicated for the transformer sides match the voltage levels of the nodes they are connected to.
+The general validation checks:
+- if assigned values are valid, e.g. lines are not allowed to have negative lengths or the rated power factor of any unit must be between 0 and 1
+- furthermore, several connections are checked, e.g. that lines only connect nodes of the same voltage level or that the voltage levels indicated for the transformer sides match the voltage levels of the nodes they are connected to.
+
+The uniqueness validation checks if a collection of given objects are unique in either:
+- a specific field
+- or in a combination of multiple fields
+
+For example `AssetInputs` needs to be unique in their `uuid` and in their `id` while
+`weather values` needs to be unique in the combination of their `time` and their `coordinate`.
+
+
+## How to use?
+- To use the general validation the user should call the method `ValidationUtils.check(Object)`.
+- To use the uniqueness validation the user should call a method of `UniquenessValidationUtils` to check the uniqueness.
+
+## How does the general validation work?
 
-## How does it work?
-- The method `ValidationUtils.check(Object)` is the only method that should be called by the user.
-- This check method identifies the object class and forwards it to a specific check method for the given object
+- The method `ValidationUtils.check(Object)` identifies the object class and forwards it to a specific check method for the given object
 - The overall structure of the ValidationUtils methods follows a cascading scheme, orientated along the class tree
 - Example: A `LineInput lineInput` should be checked
     1. `ValidationUtils.check(lineInput)` is called
@@ -49,15 +65,14 @@ The ValidationUtils include validation checks for...
         - PvInput
         - StorageInput
         - WecInput
-        - (missing: EvcsInput)
+        - EvcsInput (also checks ChargingPointType)
     - SystemParticipantTypeInput
         - BmTypeInput
         - ChpTypeInput
         - EvTypeInput
         - HpTypeInput
         - StorageTypeInput
         - WecTypeInput
-        - (missing: EvcsTypeInput/ChargingPointType)
 - ThermalUnitValidationUtils
     - ThermalUnitInput
         - ThermalSinkInput

docs/readthedocs/io/csvfiles.md

@@ -99,7 +99,7 @@ You may extend / alter the naming with pre- or suffix by calling `new EntityPers
 Let's spend a few more words on the individual time series:
 Those files are meant to carry different types of content - one might give information about wholesale market prices,
 the other is a record of power values provided by a real system.
-To be able to understand, what's inside of the file, the *columnScheme* part of the file name gives insight of it's
+To be able to understand, what's inside of the file, the *columnScheme* part of the file name gives insight of its
 content.
 
 For example, you have an IndividualTimeSeries CSV file for energy prices, then you use the key `c` from the table below

docs/readthedocs/models/input/em.md

@@ -0,0 +1,50 @@
+(em_model)=
+
+# Energy Management Unit
+
+A model of an Energy Management Unit that controls the power of connected system participants. 
+Participants are connected to an EM each via their `em` field.
+
+## Attributes, Units and Remarks
+
+```{eval-rst}
+.. list-table::
+   :widths: 33 33 33
+   :header-rows: 0
+
+
+   * - Attribute
+     - Unit
+     - Remarks
+
+   * - uuid
+     - --
+     - 
+
+   * - id
+     - --
+     - Human readable identifier
+
+   * - operator
+     - --
+     - 
+
+   * - operationTime
+     - --
+     - Timely restriction of operation
+
+   * - controlStrategy
+     - --
+     - String representation (e.g. name) of a control strategy
+
+   * - parentEm
+     - --
+     - | Reference to a superior Energy Management Unit that is controlling this EM.
+       | Field can be empty or missing, if this EM itself is not controlled.
+
+```
+
+## Caveats
+
+Nothing - at least not known.
+If you found something, please contact us!

docs/readthedocs/models/input/participant/bm.md

@@ -107,6 +107,12 @@ Model of a biomass power plant.
      - € / MWh
      - Fixed feed in tariff
 
+   * - em
+     - --
+     - | UUID reference to an :ref:`Energy Management Unit<em_model>` that is controlling
+       | this system participant. Field can be empty or missing, if this participant
+       | is not controlled.
+
 ```
 
 ## Caveats

docs/readthedocs/models/input/participant/chp.md

@@ -114,6 +114,12 @@ Combined heat and power plant.
      - | Whether to adapt output based on (volatile)
        | market price or not
 
+   * - em
+     - --
+     - | UUID reference to an :ref:`Energy Management Unit<em_model>` that is controlling
+       | this system participant. Field can be empty or missing, if this participant
+       | is not controlled.
+
 ```
 
 ## Caveats

docs/readthedocs/models/input/participant/ev.md

@@ -20,7 +20,7 @@ Model of an electric vehicle, that is occasionally connected to the grid via an
 
    * - uuid
      - --
-     -
+     - 
 
    * - id
      - --
@@ -45,12 +45,16 @@ Model of an electric vehicle, that is occasionally connected to the grid via an
 
    * - sRated
      - kVA
-     - Rated apparent power
+     - Rated apparent power for AC
+     
+   * - sRatedDC
+     - kW
+     - power for DC
 
    * - cosPhiRated
      - --
      - Rated power factor
-
+     
 ```
 
 ### Entity Model
@@ -67,27 +71,37 @@ Model of an electric vehicle, that is occasionally connected to the grid via an
 
    * - uuid
      - --
-     -
+     - 
 
    * - id
      - --
      - Human readable identifier
 
    * - operator
      - --
-     -
+     - 
 
    * - operationTime
      - --
      - Timely restriction of operation
 
    * - node
      - --
-     -
+     - 
+     
+   * - qCharacteristics
+     - --
+     - :ref:`Reactive power characteristic<participant-general-q-characteristic>` to follow
 
    * - type
      - --
-     -
+     - 
+
+   * - em
+     - --
+     - | UUID reference to an :ref:`Energy Management Unit<em_model>` that is controlling
+       | this system participant. Field can be empty or missing, if this participant
+       | is not controlled.
 
 ```
 

docs/readthedocs/models/input/participant/evcs.md

@@ -59,6 +59,12 @@ station and has some limitations outlined below.
      - --
      - 	:ref:`Charging station location types<location-types>` 
 
+   * - em
+     - --
+     - | UUID reference to an :ref:`Energy Management Unit<em_model>` that is controlling
+       | this system participant. Field can be empty or missing, if this participant
+       | is not controlled.
+
 ```
 
 ### Type Model