Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,5 @@
**.vscode/
**.pytest_cache/
**.venv/
**src/
**src/
**.source/_build/
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file added source/_build/doctrees/developer/set-up.doctree
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file added source/_build/doctrees/environment.pickle
Binary file not shown.
Binary file added source/_build/doctrees/index.doctree
Binary file not shown.
Binary file added source/_build/doctrees/quick-start.doctree
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file added source/_build/doctrees/references/models.doctree
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file added source/_build/doctrees/user/config-file.doctree
Binary file not shown.
Binary file added source/_build/doctrees/user/connections.doctree
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file added source/_build/doctrees/user/simulations.doctree
Binary file not shown.
4 changes: 4 additions & 0 deletions source/_build/html/.buildinfo
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Sphinx build info version 1
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
config: b30deb3da3929e74d3ef0e6a3e2b9a2e
tags: 645f666f9bcd5a90fca523b33c5a78b7
106 changes: 106 additions & 0 deletions source/_build/html/_sources/deprecated-cluster-setup.md.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Cluster Pi Setup

The Illuminator can be deployed to a cluster of Raspberry Pi's. It requires one Raspberry Pi acting as a **server** or 'master' and several **clients** Raspberry Pi's.
Raspberry Pi's must be connected and configured as a local network, and the
*server* must be configured to have permissions to access and control the *clients* through the Secure Shell Protocol (SSH).

During simulation, the *server* engage with the *clients* to run the simulations defined in the [simulation configuration file](user/config-file.md), and
information is exchanged between Rasberry Pi's using network sockets.
The **server** provides a Dashboard to visualize the results, and saves them to a `.csv` files for later analysis.


<div align="center">
<img align="center" src="_static/img/Structure.jpg" width="500">
</div>

## Hardware Requirements
- A Raspberry Pi to use as a *server*.
- One or more Raspebrry Pi's to use as *clients*.
- A networkw switch to connect all Rasberry Pi's as a local network.

## Set up

Conduct the following steps on each Raspberry Pi to deploy the illuminator in a cluster. These instructions require to install *Illuminator* from source.

:::{warning}
The steps were defined before the release of version `3.0.0` and therefore issues might arise when using the latest Illumiator version.
:::

1. [Install Raspberry pi OS using Raspberry Pi imager.](https://www.raspberrypi.com/software/)
2. Set an static IP address for the Raspberry Pi. Use the following command on the terminal to open the `dhcpcd.conf` file:

```shell
sudo nano /etc/dhcpcd.conf
```

In the `dhcpcd.conf` file, find the information to change the IP address to static as following:

```shell
interface etho
static ip_address=192.168.0.1/24 # change the IP address as you want
```

Give all users execute permission to all the documents in `runshfile/` in order to make sure the *server* can access the *client* model.

```shell
chmod -R a+X *dir*
```

Finally, reboot the Raspberry Pi using `sudo reboot` on the terminal.
3. [Configure SSH connections so that the *server* can connect to the *clients* without a password.](https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys-2)

4. Install the Illuminator Python package from source, and the addional dependencies:

```shell
# or, if from source code
pip install Illuminator/
```

```shell
# aditional dependencies
pip install tk python-csv python-math scipy wandb itertools
```
5. Use the following command on the *server's* terminal to check the connections for each of the *clients:*

```shell
# notice that the followng assumes that each client has a
# user named 'illuminator'
ssh illuminator@<ip> #<ip> represent the client's IP address set in step 2
```
6. Run the `build_runshfile.py` file in the configuration directory on the *server*, this will generate a `run.sh` script. Pass the appropiate `config.yaml` file containing the configuration for the simulation scenario:

```shell
python3 build_runshfile.py <config.yaml>
```

The `runs.sh` file contains a list of commands that will start the models required by a simulation defined in the `config.yaml`, such as:

```shell
# Example
lxterminal -e ssh illuminator@192.168.0.1 './Desktop/illuminatorclient/configuration/runshfile/runBattery.sh 192.168.0.1 5123 /home/illuminator/Desktop/Final_illuminator'&
lxterminal -e ssh illuminator@192.168.0.2 './Desktop/illuminatorclient/configuration/runshfile/runBattery.sh 192.168.0.2 5123 /home/illuminator/Desktop/Final_illuminator'&
```

:::{important}
**Explanation**

`lxterminal` starts a terminal on a remote machine (a client). So `lxterminal -e ssh illuminator@192.168.0.1` would use SSH to login to machine `192.168.0.1` with the user `illuminator` which has no password (this should be improved).

Three values are passed to the `ssh` command (the part between single quoates): `'./Desktop/illuminatorclient/configuration/runshfile/runWind.sh 192.168.0.1 5123 /home/illuminator/Desktop/Final_illuminator'&`.
This starts the script `./Desktop/illuminatorclient/configuration/runshfile/runWind.sh` on the remote machine with the following parameters:
* IP address: `192.168.0.1`
* Port: `5123`
* Path of mosaik file: `/home/illuminator/Desktop/Final_illuminator`

The `& `at the end starts the process in the background, so that the `run.sh` script does not wait for the command to finish but executes the next command immediately.

For example, the `runWind.sh` looks like this:

```shell
#! /bin/bash
cd $3/Wind
python wind_mosaik.py $1:$2 –remote
```
There you see the three parameters in action.
:::

79 changes: 79 additions & 0 deletions source/_build/html/_sources/developer/dev-cluster-setup.md.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# Illuminator Cluster Set Up

The Illuminator cluster setup is an optional execution of Illuminator scenarios in different Raspberry Pis. This offers a more hands-on approach to running Illuminator cases with the option to have one or multiple models of a scenario running on different Raspberry Pi devices. The Illuminator Cluster is also used for the [Illuminator Demonstator Kit](https://github.com/Illuminator-team/Illuminator-Demonstrator), a physical setup in which flows and interactions between components of a specific scenario are visualised for educational purposes.

## Master-Worker Raspberry Pis
For the cluster setup the Raspberry Pis need to be connected to the same network. The master Raspberry pi holds the yaml file of the scenario to be run. It connects to the worker Raspberry Pis through ssh and "tasks" them with running specific models of the scenario.

![alt text](https://raw.githubusercontent.com/Illuminator-team/Illuminator-Demonstrator/refs/heads/main/figs/image-1.png?token=GHSAT0AAAAAAC4762D6BQXODDXAGXTGH4AY2BDHTVQ)
*Example setup of a Raspberry Pi cluster running the Illuminator*


All Raspberry Pis can run multiple models, specifically they can run one model per available port.

## Raspberry Pi Setup

The following 2 steps need to be done to all Raspberry Pis, in the future the Install Illuminator step will also be done through the master to make deployment smoother.

### Flash SD card

Install a Raspberry Pi imager and flash the SD card with Raspberry Pi OS.
In the config file of the imager add the following configurations:

| **Parameter** | value |
|:------------:|:-------------:|
| **Username** | your_username |
| **Password** | your_password |
| **ssh** | Enabled |

Your defined username and password will be needed to connect to the worker Raspberry Pis through your master. Note that the username field is case sensitive.

### Install Illuminator

Clone the github repository by running `git clone https://github.com/Illuminator-team/Illuminator.git`. Enter the Illuminator directory and switch to the cluster branch by running `git checkout cluster`.

Note that the Illuminator directory needs to have a capital I.

## Master Pi Setup
The following steps are all conducted through the Master Raspberry Pi. First we need to ensure our connection to the worker Raspberry Pis for remote execution of our scripts.

### IP Address Setup

The IPs of all Raspberry Pis need to be static for us to know where to connect. Navigate to the `Illuminator/cluster_setup` directory and run `static_ip.py`. The script will automatically make the device's IP static.

To make the worker Raspberry Pis'IPs static navigate to `Illuminator` dir and edit and run `setup.py`. This script generates keys for ssh connections without needing a password. Make sure your username, password and IPs reflect those of the worker Pis you wish to set up.

### Simulation Preparation

Normally the Illuminator scenarios are defined as yaml files. Similarly, those very same files are used in the cluster setup with some additional steps which allow us to run specific models in the worker Raspberry Pis.

Within your yaml file define the additional fields
```
connect:
ip: 145.94.213.75 # IP of the worker Pi we want to run this model
port: 5123 # port of the worker Pi that will run this model
```
To prepare the scenario you wish to run in the cluster run `Illuminator cluster your_scenario.yaml`.

### Simulation Preparation

Execute `run.sh` and run `Illuminator scenario run your_scenario.yaml`.

## Warnings

- Slaves might need to be updated after adding a model.
- Slaves run .sh files to actually start the host (available in Illuminator/configuration/runshfile), if changed/added they need to be made executable through chmod +x <file.sh>.

This section of the documentation is a work in progress, please consult the related guide of the Illuminator Demonstrator Kit repository found [here](https://github.com/Illuminator-team/Illuminator-Demonstrator).

<!-- We are currently working on automating some of the steps to deploy the **Illuminator** on a Raspberry Pi cluster.
The main idea of this line of development is to simplify the number of steps required to install and enable simulations, after the networking of the cluster is completed. The following are some of the goals we will like to achieve:

1. **Installation:** it shall be possible to install the 'Illuminator' on each *client* and *server* with a single command on the terminal. For example: `pip install illuminator`. This will remove the burden of copying the source code to to each client and server imposed by the current implementation. Achieving this will make goal (2) more feasible.
2. **Model accessibility:** it shall be possible for the *server* to run models in any *client* by using the `lxterminal` command, with a command such as:
`lxterminal -e ssh illuminator@192.168.0.1 'illuminator cluster run <model-name> --remote'`. Where the `--remote` flag must tell *Mosaik* that the simulation will run in a distributed environment (cluster). In the current implementation, one has to specified the path to the Python file containing the model (usually a directoy on the client). However, that is harder to maintain because new versions of the Illuminator rely on the source code and not on Python wheels or TARs. Therefore, the focus here should be on making sure that once the Illuminator is installed to the Python path, then all models are accessible at the OS level and they can be run in *remote* model.
3. **Business logic for cluster scenarios:** shall develop the business logic to use the information in (`connect`) in the [scenario configuration file](../user/config-file.md) to start simulators on the clienst; such that, for example, calling `illuminator cluster <scenario.yaml>` reads on which client a model should be started, and start the relevant simulators on that client and runs the simulation. -->

:::{note}
The ideas above can be used by contributors to participate in the development of the *Illuminator*. Feedback and solutions are always welcome. Please contact the [Illuminator Development Team](mailto:illuminator@tudelft.nl) if you would like to contribute.
:::
7 changes: 7 additions & 0 deletions source/_build/html/_sources/developer/dev-dashboard.md.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Dashboard

We are looking for an light-weigth opensource source solution for implementing a dasboard for the *Illuminator*. The main purspose of the dashboard is to display the simulations result during runtime, therefore a solution must provide visualisations in real-time. We haven't explore the posibilites, but [Graphana](https://grafana.com/) can be a candidate.

:::{note}
The ideas above can be use by contributors to participate in the development of *Illuminator*. Better ideas and solutions always welcome. Please contact the [Illuminator Development Team](mailto:illuminator@tudelft.nl) if you want to contribute.
:::
168 changes: 168 additions & 0 deletions source/_build/html/_sources/developer/developer-docstrings.md.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
# Custom Model Development

The Illuminator supports and endorses the creation of custom models for your simulation needs. Model creation can be done with basic prior programming knowledge and a basic understanding of inheritance and object oriented programming.

All new models must satisfy the below criteria, this is to ensure a smooth integration and contribution to the Illuminator. Any model created & submitted must fulfill the following conventions to be added to the Illuminator package.

## Good Coding Practices

The points listed in this subsection are general good coding practices already utilised in the Illuminator. We encourage you to adopt these habits for this and any project you are a part of. Such practices directly improve the readability and maintainability of the project.

```python
# Bad Example

def func(x, y):
return x + y

temp = func(10, 20)
print(temp)

# Good Example

def add_numbers(first_number, second_number):
return first_number + second_number

result = add_numbers(10, 20)
print(result)
```

### Meaningful Function & Variable Names

Functions and variables should have clean and meaningful names that clearly indicate their purpose and functionality.

A descriptive name provides context, reduces ambiguity, and eliminates the need for excessive comments to explain its use. Adhering to consistent naming conventions also ensures uniformity across the project and enhances collaboration.

### Unnecessary Computations

Avoid redundant computations, excessive loops, and duplicate code segments to enhance efficiency and reduce points of error.

Evaluate whether computations can be precomputed, reused, or consolidated to minimize resource consumption. By refactoring repetitive patterns into shared methods, you not only simplify maintenance but also prevent errors caused by inconsistent updates to duplicated logic.

:::{note}
Feedback and solutions are always welcome. Please contact the [Illuminator Development Team](mailto:illuminator@tudelft.nl) if you would like to contribute.
:::

### Hard-Coded bits & Spaghetti Code

Hard-coded values and spaghetti code undermine the flexibility and scalability of a project and make debugging and future development exceedingly difficult. Prioritize clean, versatille design.

### Libraries, Dependencies & Versions

You are welcome to use any library or package in your custom models, try however to not import unused modules or entire libraries when only specific functionalities are required.

```python

# Bad Example:

import pandas # Entire library is imported but only DataFrame is used

df = pandas.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
print(df)

# Good Example:

from pandas import DataFrame # Only importing what is needed

df = DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
print(df)

```

Additionally, please ensure you update and specify exact version numbers in dependency manifests (requirements.txt) to ensure that builds are reproducible and unaffected by breaking changes in newer releases.

## Project Documentation

Documentation and code comprehensibility are crucial for an open-source project like the Illuminator. Only through them can contributors and users understand, maintain, and improve the codebase effectively.

### Comments

Focus on adding comments where the code's purpose or logic is not immediately obvious to the reader. Avoid redundant comments that merely restate the code or clutter the file.

### Docstrings

Docstrings are crucial to custom models because each model's documentation is generated through them. Follow the current style for functions and classes.

The docstrings should have a clearly separated sections for other parts of the code (if any). These may include, but is not limited to: Parameters, Attributes, Returns, Raises.

Each separated section should start with the Title of the section, followed by a row of dashes such as in the following section:

```python
def multiply(a, b):
"""
Computes the multiplication of two numbers and returns its value.

...
Parameters
----------
a : int
The first integer of the multiplication formula
b : int
The second integer of the multiplication formula

Returns
-------
int
The two values multiplied.
"""
return a * b
```

When appropriate, we should also ensure to include the name and/or type of variables for the any of the aforementioned sections.

Lastly, type hints are also a useful addition to any code. They can be used to "hint" to other developers what is expected as input and/or output when a function is used.

```python
def sum(a:int, b:int) -> float:
```
As seen from the example above we can immediately conclude that for this function to work it will need an integer `a` and `b`, with the return value being of type float

Combined, the docstrings may look something like this:

```python
def sum(a:int, b:int) -> float:
"""
Computes the sum of `a` and `b` and returns the outcome

...

Parameters
----------
a : int
The first integer of the sum formula
b : int
The second integer of the sum formula

Returns
-------
result : float
The sum of a + b
"""
result = a + b
return result
```
Docstrings are essential to the Illuminator since each model's documentation is automatically generated through them. It is a model developers responsibility to ensure thorough documentation of their implemented model.

### Consistency & Linting

Always adhere to the conventions and style used in the existing code, including naming patterns, indentation, spacing, and structure. For example, if camelCase is used for variable names, avoid switching to snake_case.

Tools like linters and formatters can help enforce these conventions, this project utilises [pylint](https://pypi.org/project/pylint/) across its codebase. This package also helps maintain aforementioned good coding practices. Any submitted code must pass pylint's format check.

## Model Testing

This section lists the testing methods you must implement along with your model to verify it operates properly within the rest of the Illuminator framework.

### Scenario Implementation

Creating a YAML scenario file for your custom model is essential for verifying that it integrates and runs properly within the simulation framework. This file essentially acts as a lightweight end-to-end test, providing an easy way to confirm that the model behaves as expected.

### Unit Tests

Unit tests are a critical component of ensuring the internal functionality of your custom model. These tests validate individual methods and components to confirm that they produce the expected outputs for a variety of input cases, including edge scenarios. Further instructions can be found in the [Creating/Writing Tests](https://illuminator-team.github.io/Illuminator/developer/writing-tests.html) section.

<!-- ## Missing data in older docstrings
There is still a lot of missing data for older docstrings, which has not been completed due to missing domain knowledge.
In order to contribute to those, one should simply search for any file which contains `???` in its docstrings and change it to whatever is appropriate.
In most cases, the description is missing since we could still acquire datatypes of attributes/parameters based on context clues or debugging/testing. However some bits of code are unused which means that it is also missing the type hints/docstring object types.

There exists an excel sheet within the `docs` folder called `Illuminator Model Classification.xlsx` which contains a semi-filled list of variables and their descriptions which can be used to help finish the incomplete docstrings. -->
Loading