Merge pull request #127 from pnnl/develop

Develop

Author: d3j331

README.md

@@ -27,3 +27,4 @@ Change log:
 - v1.2.1  Fixed python shell files, fixed complex python to use helics complex.
 - v1.2.2  Fixed the installation for Ubuntu 22.04.
 - v1.3.0  Refactor the TESP PyPI api. Upgrade all models(GridLAB-D, EnergyPlus, NS3) to work with HELICS 3.4. Add modifier.py for GridLAB-D models
+- v1.3.2  Updated model and modifier for GridLAB-D models, added readme for GLM modifier and Store examples

doc/Demonstrations_and_Examples.rst

@@ -30,6 +30,8 @@ TESP Capability Demonstrations
     ./demonstrations/houses.rst
     ./demonstrations/gld_player_recorder.rst
     ./demonstrations/gld_modifier.rst
+    ./demonstrations/datastore.rst
+    
 
 TESP uses TCP/IP port 5570 for communication and requires Python 3. Simulations can start many processes, and take minutes or hours to complete. At this time, instructions are given only for the Linux package or Docker version of TESP, which is installable. See below for advice on running TESP in native Mac OS X or Windows.
 

doc/demonstrations/gld_modifier.rst

@@ -41,29 +41,32 @@ With the modifier, we can read the GridLAB-D model into the GLMModifier data str
 GridLAB-D split their functionality into various modules and for this example, we're going to be adding houses to the model which means we need to make sure the "residential" module gets added to the model file.::
 
     glmMod.add_module('residential', [])
+    
+The GLMModifier has an attribute that holds the entire GridLAB-D model, "glm". Purely to make our live a few characters easier when using the GLMModifier, we can assign this to another variable of our choice and strip out the need constantly pre-pend many of our commands with "GLMModifier".::
 
-GLMModifier makes it easy to get the names of all of the objects of a given class and in this case, to add houses, we need to look for GridLAB-D's "triplex_meters" to attach the houses to.::
-
-    tp_meter_names = glmMod.get_object_names('triplex_meter')
+	glm = GLMMod.glm
 
-If we wanted the objects themselves we would instead call::
+GLMModifier makes it easy to get the names of all of the objects of a given class and in this case, to add houses, we need to look for GridLAB-D's "triplex_meters" to attach the houses to.::
 
-    tp_meter_objs = glmMod.get_object('triplex_meter').instance
+    tp_meter_objs = glm.triplex_meter
+    tp_meter_names = list(tp_meter_objs.keys())
 
-``tp_meter_objs`` is then a Python dictionary with the keys being the object names and the value being an Python dictionary of the object parameters and values.
+``tp_meter_objs`` is a Python dictionary with the keys being the object names and the value being an Python dictionary of the object parameters and values. To make a list of the names of the meters, we just need to ask for the keys of the dictionary as a list.
 
 Adding Objects
 --------------
 
 ``tp_meter_names`` is a list of the names of the GridLAB-D ``triplex_meter`` objects as strings. Using those names we can build up a Python dictionary that defines the parameters of another ``triplex_meter`` object we're going to add to the model. The dictionary is called "meter_params" and has three members all defined by the data from an existing specific ``triplex_meter`` in the model.::
 
+    new_name = tp_meter_names[house_num]
+    billing_meter_name = f"{new_name}_billing"
     meter_params = {
-              'parent': tp_meter_names[house_num],
-              'phases': glmMod.get_object('triplex_meter').instance[f'{tp_meter_names[house_num]}']['phases'],
-              'nominal_voltage': glmMod.get_object('triplex_meter').instance[tp_meter_names[house_num]]['nominal_voltage']
+        "parent": new_name,
+        "phases": glm.triplex_meter[f"{new_name}"]["phases"],
+        "nominal_voltage": glm.triplex_meter[f"{new_name}"]["nominal_voltage"],
     }
 
-The ``phases`` and ``nominal_voltage`` show a common practice with the GLMModifier: ask GLMModifier for all objects of a certain type in the model and then ask for a specific one from that list by name. This shows up as a ``.get_objects("name of GridLAB-D class").instance("object name")``. ``.instance()`` returns a Python dictionary with all the GridLAB-D parameters as members. Alternatively, there is a specific API that does the same thing: ``glmMod.get_object_named_instance("name of GridLAB-D class", "object name")``
+The ``phases`` and ``nominal_voltage`` are easily defined using GLMModifier as they are just members of a dictionary that defines a specific triplex meter. 
 
 Once the Python dictionary with the GridLAB-D object parameters are defined, it can simply be added to the model.::
 
@@ -73,23 +76,23 @@ Once the Python dictionary with the GridLAB-D object parameters are defined, it
 
 Adding and Modifying Existing Object Parameter Values
 -----------------------------------------------------
-Further down in the example, there's an example of how to modify an existing object. In this case, the ``.add_object()`` method returns the the GridLAB-D object (effectively a Python dictionary). Its also possible to get the same object using the ``.get_objects().instance()``. Using either method, once you have the GridLAB-D object, its easy to modify any of its properties such as::
+Further down in the example, there's a portion of code showing of how to modify an existing object. In this case, we use the fact that ``.add_object()`` method returns the the GridLAB-D object (effectively a Python dictionary) once it is added to the model. Once you have the GridLAB-D object, its easy to modify any of its properties such as::
 
     house_obj['floor_area'] = 2469
 
 This exact syntax is also valid for adding a parameter that is undefined to an existing GridLAB-D object.
 
 Deleting Existing Object Parameter Values
 -----------------------------------------
-To delete a GridLAB-D object parameter value there is a dedicated API call::
+To delete a GridLAB-D object parameter value, you can just set to to `None`::
 
-    glmMod.del_object_attr('house', house_name, 'Rroof')
+    house_to_edit["Rroof"] = None
 
-Note that GridLAB-D requires some parameters to be defined to run its simulations. Removing the parameter will remove it from the GridLAB-D model file that gets created (.glm) but may effectively force GridLAB-D to use its internal default value.
+Note that GridLAB-D requires some parameters to be defined to run its simulations. Removing the parameter will remove it from the GridLAB-D model file that gets created (.glm) but may effectively force GridLAB-D to use its internal default value. That is, clearing the parameter value in this way is not the same as setting it to an undefined value.
 
 Deleting Existing Objects
 -------------------------
-Its possible to delete an object and all its parameter values::
+Its possible to delete an object and all its parameter values from the GridLAB-D model::
 
     glmMod.del_object('house', house_to_delete)
 

examples/capabilities/datastore/te30_usestore.py

@@ -6,37 +6,111 @@
 """
 
 
-import tesp_support.api.store as fle
+import tesp_support.api.store as store
+import os
+import pprint
+import matplotlib.pyplot as plt
+import pandas as pd
+
+# Setting up pretty printing, mostly for debugging.
+pp = pprint.PrettyPrinter(indent=4)
 
 
 def process_results(case_name):
     """
     Opens up datastore (.zip) and metadata (.json) to process results
-    
+
     Assumes te30_store.zip and te30_store.json have been copied from ../te30 folder to
     the same folder as this script (examples/capabilities/datastore).
     """
-    # Load in metadata to see what contents are in the TE30 store
-    # fle.unzip(case_name)
-
-    # example
-    my_store = fle.Store(case_name)
-    # this is a cvs file
-    my_file = my_store.get_schema('weather')
-    data = my_file.get_series_data('weather', '2013-07-01 00:00', '2013-07-02 00:00')
-    tseries = [data]
-    print(tseries)
-
-    # List all the files in the store for inspection
-    for item in my_store.get_schema():
-        print(item)
-		
-    # Arbitrarily, let's look at the real power load at a specific billing meter
-	# This data is coming from GridLAB-D (based on the TE30 documentation)
-	
-	# metadata table has the name of each parameter and "value" is the unit.
-	# Same order as what's in the index tables
+    start_date_1 = "2013-07-01 00:00"
+    end_date_1 = "2013-07-02 00:00"
+
+    # Depending on how your unzip tools work, unzipping the data store may
+    # create another folder and put all the zipped files into it. If so,
+    # we need to change our working directory to that folder.
+    if os.path.exists(case_name):
+        os.chdir(case_name)
+
+    # Define the data store
+    te30_store = store.Store(case_name)
+
+    # List all the files in the store for inspection; particularly useful if
+    # you're new to the dataset
+    print(f"Schemas in data store:")
+    for item in te30_store.get_schema():
+        print(f"\t{item}")
+
+    # Get weather data (CSV)
+    # "weather" is the name of the data store schema for "weather.csv
+    weather_schema = te30_store.get_schema("weather")
+
+    # Inspect the available tables and data
+    print(f"Weather tables {pp.pformat(weather_schema.tables)}")
+    print(f"Weather columns {pp.pformat(weather_schema.columns)}")
+    # The "solar_flux" column is what we're looking for.
+
+    # For better or worse, the single table of data inside weather.csv is
+    # also named "weather".
+    weather_data = weather_schema.get_series_data("weather", start_date_1, end_date_1)
+
+    # Checking data type for timestamp and convert if necessary
+    weather_time = weather_data["timestamp"]
+    if isinstance(weather_time.iloc[0], str):
+        weather_time = pd.to_datetime(weather_time, format="%Y-%m-%d %H:%M:%S PDT")
+    # And convert the data as strings to numeric values
+    if isinstance(weather_data["solar_flux"].iloc[0], str):
+        solar_data = pd.to_numeric(weather_data["solar_flux"])
+
+    # As a convenience, make a new dataframe with only the data I need
+    weather_data = pd.concat([weather_time.T, solar_data], axis=1)
+    weather_data = weather_data.set_index("timestamp")
+
+    # Get rooftop solar production data (HDF5)
+    inverter_schema = te30_store.get_schema("inverter_TE_ChallengeH_metrics")
+
+    print(f"Inverter tables list  {pp.pformat(inverter_schema.tables)}")
+    print(f"Inverter columns dictionary {pp.pformat(inverter_schema.columns)}")
+    # For silly reasons, GridLAB-D stores each day of data in its own table
+    # called "index1", "index2", etc.
+
+    # The schema is just a dictionary so if we want to look at columns for just a
+    # table it's pretty easy. (In this case the columns are identical for each day so
+    # it's not that exciting.)
+    print(f"What is this? {pp.pformat(inverter_schema.tables)}")
+    print(
+        f"Inverter columns list for table of data for first simulated day ('index1') "
+        f"{pp.pformat(inverter_schema.columns['index1'])}"
+    )
+
+    inverter_data = inverter_schema.get_series_data("index1", start_date_1, end_date_1)
+    # Just going to be looking at data from a single house
+    houseA11_inv = inverter_data.loc[(inverter_data["name"] == b"inv_F1_house_A11")]
+    inverter_time = houseA11_inv["date"]
+    # If the date is typed as a string instead of a datetime object, we need to
+    # convert it to a datetime object to allow for indexing.
+    if isinstance(inverter_time.iloc[0], str):
+        inverter_time = inverter_time.str[:-4]
+        inverter_time = pd.to_datetime(inverter_time, format="%Y-%m-%d %H:%M:%S")
+    # Making a new DataFrame for convenience
+    inverter_data = pd.concat([inverter_time.T, houseA11_inv["real_power_avg"]], axis=1)
+    inverter_data = inverter_data.set_index("date")
+
+    # Plot the resulting data
+    fig = plt.figure()
+    ax1 = fig.add_subplot()
+    ax1.set_xlabel("Time")
+    ax1.set_ylabel("Inverter Power (W)")
+    ax1.plot(inverter_data["real_power_avg"], label="Inverter Power", color="blue")
+    ax2 = ax1.twinx()
+    ax2.set_ylabel("Solar Flux (W/ft^2)")
+    ax2.plot(weather_data["solar_flux"], label="Solar Flux", color="orange")
+    h1, l1 = ax1.get_legend_handles_labels()
+    h2, l2 = ax2.get_legend_handles_labels()
+    ax1.legend(h1 + h2, l1 + l2, loc=2)
+    fig.autofmt_xdate(rotation=45)
+    plt.show()
 
 
 if __name__ == "__main__":
-    process_results("te30_store")
+    process_results("te30_store")